Jul 14 13:25:05.507922 ip-10-0-140-78 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 14 13:25:05.507933 ip-10-0-140-78 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 14 13:25:05.507940 ip-10-0-140-78 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 14 13:25:05.508158 ip-10-0-140-78 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 14 13:25:15.665361 ip-10-0-140-78 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 14 13:25:15.665381 ip-10-0-140-78 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 9ede8f38fd1943e885f44d41cea7adad --
Jul 14 13:27:14.570278 ip-10-0-140-78 systemd[1]: Starting Kubernetes Kubelet...
Jul 14 13:27:14.952562 ip-10-0-140-78 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 13:27:14.952562 ip-10-0-140-78 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 14 13:27:14.952562 ip-10-0-140-78 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 13:27:14.952562 ip-10-0-140-78 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 14 13:27:14.952562 ip-10-0-140-78 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 13:27:14.952562 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.914352    2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 14 13:27:14.952562 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918277    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 13:27:14.952562 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918289    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 13:27:14.952562 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918293    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 13:27:14.952562 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918297    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 13:27:14.952562 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918300    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 13:27:14.952562 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918302    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 13:27:14.952562 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918305    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 13:27:14.953454 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918308    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 13:27:14.953454 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918311    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 13:27:14.953454 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918313    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 13:27:14.953454 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918317    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 13:27:14.953454 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918321    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 13:27:14.953454 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918324    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 13:27:14.953454 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918327    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 13:27:14.953454 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918337    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 13:27:14.953454 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918340    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 13:27:14.953454 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918342    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 13:27:14.953454 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918345    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 13:27:14.953454 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918347    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 13:27:14.953454 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918351    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 13:27:14.953454 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918354    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 13:27:14.953454 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918356    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 13:27:14.953454 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918359    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 13:27:14.953454 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918362    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 13:27:14.953454 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918364    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 13:27:14.953454 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918366    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 13:27:14.953454 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918369    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 13:27:14.954180 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918372    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 13:27:14.954180 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918375    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 13:27:14.954180 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918377    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 13:27:14.954180 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918380    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 13:27:14.954180 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918382    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 13:27:14.954180 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918385    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 13:27:14.954180 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918388    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 13:27:14.954180 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918390    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 13:27:14.954180 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918393    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 13:27:14.954180 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918396    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 13:27:14.954180 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918400    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 13:27:14.954180 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918404    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 13:27:14.954180 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918407    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 13:27:14.954180 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918409    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 13:27:14.954180 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918412    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 13:27:14.954180 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918415    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 13:27:14.954180 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918417    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 13:27:14.954180 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918420    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 13:27:14.954180 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918423    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 13:27:14.955095 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918427    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 13:27:14.955095 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918430    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 13:27:14.955095 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918433    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 13:27:14.955095 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918436    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 13:27:14.955095 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918439    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 13:27:14.955095 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918442    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 13:27:14.955095 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918445    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 13:27:14.955095 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918448    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 13:27:14.955095 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918450    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 13:27:14.955095 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918452    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 13:27:14.955095 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918455    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 13:27:14.955095 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918457    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 13:27:14.955095 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918460    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 13:27:14.955095 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918462    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 13:27:14.955095 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918465    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 13:27:14.955095 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918467    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 13:27:14.955095 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918470    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 13:27:14.955095 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918472    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 13:27:14.955095 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918476    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 13:27:14.957854 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918478    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 13:27:14.957854 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918481    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 13:27:14.957854 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918484    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 13:27:14.957854 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918486    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 13:27:14.957854 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918489    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 13:27:14.957854 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918491    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 13:27:14.957854 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918494    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 13:27:14.957854 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918497    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 13:27:14.957854 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918499    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 13:27:14.957854 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918503    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 13:27:14.957854 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918506    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 13:27:14.957854 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918508    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 13:27:14.957854 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918513    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 13:27:14.957854 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918517    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 13:27:14.957854 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918520    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 13:27:14.957854 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918522    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 13:27:14.957854 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918525    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 13:27:14.957854 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918527    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 13:27:14.957854 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918530    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 13:27:14.957854 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918532    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 13:27:14.959236 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918535    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 13:27:14.959236 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918537    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 13:27:14.959236 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918540    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 13:27:14.959236 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918542    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 13:27:14.959236 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918544    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 13:27:14.959236 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918547    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 13:27:14.959236 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918550    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 13:27:14.959236 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918552    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 13:27:14.959236 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918555    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 13:27:14.959236 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918557    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 13:27:14.959236 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918560    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 13:27:14.959236 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918562    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 13:27:14.959236 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918566    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 13:27:14.959236 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.918569    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 13:27:14.959236 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919274    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 13:27:14.959236 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919281    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 13:27:14.959236 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919284    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 13:27:14.959236 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919287    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 13:27:14.959236 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919290    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 13:27:14.959236 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919292    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 13:27:14.960437 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919295    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 13:27:14.960437 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919297    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 13:27:14.960437 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919299    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 13:27:14.960437 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919302    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 13:27:14.960437 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919304    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 13:27:14.960437 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919307    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 13:27:14.960437 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919309    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 13:27:14.960437 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919312    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 13:27:14.960437 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919314    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 13:27:14.960437 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919317    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 13:27:14.960437 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919320    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 13:27:14.960437 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919324    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 13:27:14.960437 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919326    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 13:27:14.960437 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919330    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 13:27:14.960437 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919334    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 13:27:14.960437 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919337    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 13:27:14.960437 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919340    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 13:27:14.960437 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919342    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 13:27:14.960437 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919345    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 13:27:14.962835 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919350    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 13:27:14.962835 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919353    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 13:27:14.962835 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919356    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 13:27:14.962835 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919359    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 13:27:14.962835 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919385    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 13:27:14.962835 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919389    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 13:27:14.962835 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919395    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 13:27:14.962835 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919398    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 13:27:14.962835 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919401    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 13:27:14.962835 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919404    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 13:27:14.962835 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919407    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 13:27:14.962835 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919410    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 13:27:14.962835 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919412    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 13:27:14.962835 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919415    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 13:27:14.962835 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919417    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 13:27:14.962835 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919420    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 13:27:14.962835 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919422    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 13:27:14.962835 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919426    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 13:27:14.962835 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919429    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 13:27:14.965252 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919431    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 13:27:14.965252 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919434    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 13:27:14.965252 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919439    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 13:27:14.965252 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919442    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 13:27:14.965252 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919444    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 13:27:14.965252 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919447    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 13:27:14.965252 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919450    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 13:27:14.965252 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919453    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 13:27:14.965252 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919455    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 13:27:14.965252 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919458    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 13:27:14.965252 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919460    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 13:27:14.965252 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919463    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 13:27:14.965252 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919465    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 13:27:14.965252 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919467    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 13:27:14.965252 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919469    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 13:27:14.965252 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919472    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 13:27:14.965252 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919475    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 13:27:14.965252 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919477    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 13:27:14.965252 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919479    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 13:27:14.965252 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919482    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 13:27:14.967611 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919490    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 13:27:14.967611 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919493    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 13:27:14.967611 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919495    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 13:27:14.967611 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919497    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 13:27:14.967611 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919500    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 13:27:14.967611 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919502    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 13:27:14.967611 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919505    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 13:27:14.967611 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919507    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 13:27:14.967611 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919510    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 13:27:14.967611 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919512    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 13:27:14.967611 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919515    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 13:27:14.967611 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919519    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 13:27:14.967611 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919523    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 13:27:14.967611 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919525    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 13:27:14.967611 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919530    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 13:27:14.967611 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919533    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 13:27:14.967611 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919535    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 13:27:14.967611 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919537    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 13:27:14.967611 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919540    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 13:27:14.967611 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919542    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 13:27:14.970451 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919545    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 13:27:14.970451 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919547    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 13:27:14.970451 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919550    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 13:27:14.970451 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919552    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 13:27:14.970451 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919554    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 13:27:14.970451 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919557    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 13:27:14.970451 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919560    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 13:27:14.970451 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919562    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 13:27:14.970451 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919564    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 13:27:14.970451 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919567    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 13:27:14.970451 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919570    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 13:27:14.970451 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919572    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 13:27:14.970451 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919575    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 13:27:14.970451 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919585    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 13:27:14.970451 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.919587    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 13:27:14.970451 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920065    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 13:27:14.970451 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920072    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 13:27:14.970451 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920075    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 13:27:14.970451 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920078    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 13:27:14.970451 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920081    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 13:27:14.971789 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920084    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 13:27:14.971789 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920086    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 13:27:14.971789 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920089    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 13:27:14.971789 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920091    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 13:27:14.971789 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920094    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 13:27:14.971789 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920097    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 13:27:14.971789 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920101    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 13:27:14.971789 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920104    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 13:27:14.971789 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920107    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 13:27:14.971789 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920109    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 13:27:14.971789 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920112    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 13:27:14.971789 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920114    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 13:27:14.971789 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920117    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 13:27:14.971789 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920119    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 13:27:14.971789 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920122    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 13:27:14.971789 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920124    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 13:27:14.971789 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920127    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 13:27:14.971789 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920129    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 13:27:14.971789 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920132    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 13:27:14.971789 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920134    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 13:27:14.972855 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920138    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 13:27:14.972855 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920141    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 13:27:14.972855 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920145    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 13:27:14.972855 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920147    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 13:27:14.972855 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920150    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 13:27:14.972855 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920160    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 13:27:14.972855 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920163    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 13:27:14.972855 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920166    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 13:27:14.972855 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920169    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 13:27:14.972855 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920172    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 13:27:14.972855 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920175    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 13:27:14.972855 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920177    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 13:27:14.972855 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920180    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 13:27:14.972855 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920183    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 13:27:14.972855 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920186    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 13:27:14.972855 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920188    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 13:27:14.972855 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920191    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 13:27:14.972855 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920194    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 13:27:14.972855 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920198    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 13:27:14.972855 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920200    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 13:27:14.973790 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920203    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 13:27:14.973790 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920205    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 13:27:14.973790 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920208    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 13:27:14.973790 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920210    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 13:27:14.973790 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920212    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 13:27:14.973790 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920215    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 13:27:14.973790 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920217    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 13:27:14.973790 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920220    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 13:27:14.973790 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920222    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 13:27:14.973790 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920225    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 13:27:14.973790 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920227    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 13:27:14.973790 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920230    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 13:27:14.973790 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920232    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 13:27:14.973790 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920237    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 13:27:14.973790 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920239    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 13:27:14.973790 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920242    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 13:27:14.973790 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920244    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 13:27:14.973790 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920247    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 13:27:14.973790 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920255    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 13:27:14.973790 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920257    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 13:27:14.974782 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920260    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 13:27:14.974782 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920262    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 13:27:14.974782 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920265    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 13:27:14.974782 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920267    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 13:27:14.974782 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920269    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 13:27:14.974782 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920272    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 13:27:14.974782 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920274    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 13:27:14.974782 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920277    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 13:27:14.974782 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920279    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 13:27:14.974782 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920281    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 13:27:14.974782 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920285    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 13:27:14.974782 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920288    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 13:27:14.974782 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920290    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 13:27:14.974782 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920292    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 13:27:14.974782 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920297    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 13:27:14.974782 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920300    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 13:27:14.974782 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920303    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 13:27:14.974782 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920307    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 13:27:14.974782 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920310    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 13:27:14.975382 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920313    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 13:27:14.975382 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920315    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 13:27:14.975382 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920318    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 13:27:14.975382 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920320    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 13:27:14.975382 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920322    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 13:27:14.975382 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920325    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 13:27:14.975382 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920329    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 13:27:14.975382 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920332    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 13:27:14.975382 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920334    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 13:27:14.975382 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920336    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 13:27:14.975382 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920339    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 13:27:14.975382 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920341    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 13:27:14.975382 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920349    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 13:27:14.975382 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920352    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 13:27:14.975382 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.920354    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 13:27:14.975382 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920443    2569 flags.go:64] FLAG: --address="0.0.0.0"
Jul 14 13:27:14.975382 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920459    2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 14 13:27:14.975382 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920468    2569 flags.go:64] FLAG: --anonymous-auth="true"
Jul 14 13:27:14.975382 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920475    2569 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 14 13:27:14.975382 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920479    2569 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 14 13:27:14.975382 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920482    2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 14 13:27:14.975966 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920486    2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 14 13:27:14.975966 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920490    2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 14 13:27:14.975966 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920496    2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 14 13:27:14.975966 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920498    2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 14 13:27:14.975966 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920503    2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 14 13:27:14.975966 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920506    2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 14 13:27:14.975966 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920509    2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 14 13:27:14.975966 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920512    2569 flags.go:64] FLAG: --cgroup-root=""
Jul 14 13:27:14.975966 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920514    2569 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 14 13:27:14.975966 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920517    2569 flags.go:64] FLAG: --client-ca-file=""
Jul 14 13:27:14.975966 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920520    2569 flags.go:64] FLAG: --cloud-provider="external"
Jul 14 13:27:14.975966 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920523    2569 flags.go:64] FLAG: --cluster-dns="[]"
Jul 14 13:27:14.975966 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920526    2569 flags.go:64] FLAG: --cluster-domain=""
Jul 14 13:27:14.975966 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920529    2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 14 13:27:14.975966 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920532    2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 14 13:27:14.975966 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920535    2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 14 13:27:14.975966 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920538    2569 flags.go:64] FLAG: --container-log-max-files="5"
Jul 14 13:27:14.975966 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920543    2569 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 14 13:27:14.975966 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920546    2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 14 13:27:14.975966 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920549    2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 14 13:27:14.975966 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920552    2569 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 14 13:27:14.975966 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920555    2569 flags.go:64] FLAG: --contention-profiling="false"
Jul 14 13:27:14.975966 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920557    2569 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 14 13:27:14.975966 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920560    2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 14 13:27:14.976886 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920564    2569 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 14 13:27:14.976886 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920566    2569 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 14 13:27:14.976886 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920570    2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 14 13:27:14.976886 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920573    2569 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 14 13:27:14.976886 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920576    2569 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 14 13:27:14.976886 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920579    2569 flags.go:64] FLAG: --enable-load-reader="false"
Jul 14 13:27:14.976886 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920582    2569 flags.go:64] FLAG: --enable-server="true"
Jul 14 13:27:14.976886 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920585    2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 14 13:27:14.976886 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920589    2569 flags.go:64] FLAG: --event-burst="100"
Jul 14 13:27:14.976886 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920592    2569 flags.go:64] FLAG: --event-qps="50"
Jul 14 13:27:14.976886 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920596    2569 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 14 13:27:14.976886 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920599    2569 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 14 13:27:14.976886 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920602    2569 flags.go:64] FLAG: --eviction-hard=""
Jul 14 13:27:14.976886 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920606    2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 14 13:27:14.976886 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920608    2569 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 14 13:27:14.976886 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920611    2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 14 13:27:14.976886 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920614    2569 flags.go:64] FLAG: --eviction-soft=""
Jul 14 13:27:14.976886 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920617    2569 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 14 13:27:14.976886 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920620    2569 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 14 13:27:14.976886 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920623    2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 14 13:27:14.976886 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920626    2569 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 14 13:27:14.976886 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920629    2569 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 14 13:27:14.976886 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920631    2569 flags.go:64] FLAG: --fail-swap-on="true"
Jul 14 13:27:14.976886 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920634    2569 flags.go:64] FLAG: --feature-gates=""
Jul 14 13:27:14.976886 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920638    2569 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 14 13:27:14.977813 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920642    2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 14 13:27:14.977813 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920645    2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 14 13:27:14.977813 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920648    2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 14 13:27:14.977813 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920650    2569 flags.go:64] FLAG: --healthz-port="10248"
Jul 14 13:27:14.977813 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920653    2569 flags.go:64] FLAG: --help="false"
Jul 14 13:27:14.977813 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920656    2569 flags.go:64] FLAG: --hostname-override="ip-10-0-140-78.ec2.internal"
Jul 14 13:27:14.977813 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920659    2569 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 14 13:27:14.977813 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920662    2569 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 14 13:27:14.977813 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920665    2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 14 13:27:14.977813 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920668    2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 14 13:27:14.977813 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920672    2569 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 14 13:27:14.977813 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920675    2569 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 14 13:27:14.977813 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920677    2569 flags.go:64] FLAG: --image-service-endpoint=""
Jul 14 13:27:14.977813 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920680    2569 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 14 13:27:14.977813 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920683    2569 flags.go:64] FLAG: --kube-api-burst="100"
Jul 14 13:27:14.977813 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920685    2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 14 13:27:14.977813 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920689    2569 flags.go:64] FLAG: --kube-api-qps="50"
Jul 14 13:27:14.977813 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920693    2569 flags.go:64] FLAG: --kube-reserved=""
Jul 14 13:27:14.977813 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920696    2569 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 14 13:27:14.977813 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920698    2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 14 13:27:14.977813 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920701    2569 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 14 13:27:14.977813 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920704    2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 14 13:27:14.977813 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920707    2569 flags.go:64] FLAG: --lock-file=""
Jul 14 13:27:14.977813 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920709    2569 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 14 13:27:14.982053 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920712    2569 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 14 13:27:14.982053 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920715    2569 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 14 13:27:14.982053 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920720    2569 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 14 13:27:14.982053 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920723    2569 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 14 13:27:14.982053 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920726    2569 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 14 13:27:14.982053 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920728    2569 flags.go:64] FLAG: --logging-format="text"
Jul 14 13:27:14.982053 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920731    2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 14 13:27:14.982053 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920734    2569 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 14 13:27:14.982053 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920738    2569 flags.go:64] FLAG: --manifest-url=""
Jul 14 13:27:14.982053 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920741    2569 flags.go:64] FLAG: --manifest-url-header=""
Jul 14 13:27:14.982053 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920745    2569 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 14 13:27:14.982053 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920748    2569 flags.go:64] FLAG: --max-open-files="1000000"
Jul 14 13:27:14.982053 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920751    2569 flags.go:64] FLAG: --max-pods="110"
Jul 14 13:27:14.982053 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920754    2569 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 14 13:27:14.982053 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920757    2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 14 13:27:14.982053 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920760    2569 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 14 13:27:14.982053 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920763    2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 14 13:27:14.982053 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920766    2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 14 13:27:14.982053 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920769    2569 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 14 13:27:14.982053 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920772    2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 14 13:27:14.982053 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920779    2569 flags.go:64] FLAG: --node-status-max-images="50"
Jul 14 13:27:14.982053 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920782    2569 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 14 13:27:14.982053 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920784    2569 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 14 13:27:14.982053 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920787    2569 flags.go:64] FLAG: --pod-cidr=""
Jul 14 13:27:15.201959 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920790    2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 14 13:27:15.201959 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920796    2569 flags.go:64] FLAG: --pod-manifest-path=""
Jul 14 13:27:15.201959 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920800    2569 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 14 13:27:15.201959 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920803    2569 flags.go:64] FLAG: --pods-per-core="0"
Jul 14 13:27:15.201959 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920806    2569 flags.go:64] FLAG: --port="10250"
Jul 14 13:27:15.201959 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920809    2569 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 14 13:27:15.201959 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920812    2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01ac76a32d25bfe71"
Jul 14 13:27:15.201959 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920815    2569 flags.go:64] FLAG: --qos-reserved=""
Jul 14 13:27:15.201959 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920818    2569 flags.go:64] FLAG: --read-only-port="10255"
Jul 14 13:27:15.201959 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920821    2569 flags.go:64] FLAG: --register-node="true"
Jul 14 13:27:15.201959 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920823    2569 flags.go:64] FLAG: --register-with-taints=""
Jul 14 13:27:15.201959 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920827    2569 flags.go:64] FLAG: --registry-burst="10"
Jul 14 13:27:15.201959 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920830    2569 flags.go:64] FLAG: --registry-qps="5"
Jul 14 13:27:15.201959 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920832    2569 flags.go:64] FLAG: --reserved-cpus=""
Jul 14 13:27:15.201959 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920835    2569 flags.go:64] FLAG: --reserved-memory=""
Jul 14 13:27:15.201959 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920838    2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 14 13:27:15.201959 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920843    2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 14 13:27:15.201959 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920846    2569 flags.go:64] FLAG: --rotate-certificates="false"
Jul 14 13:27:15.201959 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920848    2569 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 14 13:27:15.201959 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920851    2569 flags.go:64] FLAG: --runonce="false"
Jul 14 13:27:15.201959 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920854    2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 14 13:27:15.201959 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920857    2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 14 13:27:15.201959 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920859    2569 flags.go:64] FLAG: --seccomp-default="false"
Jul 14 13:27:15.201959 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920862    2569 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 14 13:27:15.201959 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920865    2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 14 13:27:14.992735 ip-10-0-140-78 systemd[1]: Started Kubernetes Kubelet.
Jul 14 13:27:15.219822 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920868    2569 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 14 13:27:15.219822 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920872    2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 14 13:27:15.219822 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920874    2569 flags.go:64] FLAG: --storage-driver-password="root"
Jul 14 13:27:15.219822 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920877    2569 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 14 13:27:15.219822 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920880    2569 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 14 13:27:15.219822 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920883    2569 flags.go:64] FLAG: --storage-driver-user="root"
Jul 14 13:27:15.219822 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920885    2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 14 13:27:15.219822 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920888    2569 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 14 13:27:15.219822 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920890    2569 flags.go:64] FLAG: --system-cgroups=""
Jul 14 13:27:15.219822 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920894    2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 14 13:27:15.219822 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920900    2569 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 14 13:27:15.219822 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920903    2569 flags.go:64] FLAG: --tls-cert-file=""
Jul 14 13:27:15.219822 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920905    2569 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 14 13:27:15.219822 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920909    2569 flags.go:64] FLAG: --tls-min-version=""
Jul 14 13:27:15.219822 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920911    2569 flags.go:64] FLAG: --tls-private-key-file=""
Jul 14 13:27:15.219822 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920914    2569 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 14 13:27:15.219822 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920917    2569 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 14 13:27:15.219822 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920920    2569 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 14 13:27:15.219822 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920922    2569 flags.go:64] FLAG: --v="2"
Jul 14 13:27:15.219822 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920926    2569 flags.go:64] FLAG: --version="false"
Jul 14 13:27:15.219822 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920930    2569 flags.go:64] FLAG: --vmodule=""
Jul 14 13:27:15.219822 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920934    2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 14 13:27:15.219822 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.920937    2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 14 13:27:15.219822 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921072    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 13:27:15.219822 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921076    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 13:27:15.220759 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921079    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 13:27:15.220759 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921082    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 13:27:15.220759 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921084    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 13:27:15.220759 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921087    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 13:27:15.220759 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921089    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 13:27:15.220759 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921092    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 13:27:15.220759 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921095    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 13:27:15.220759 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921097    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 13:27:15.220759 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921100    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 13:27:15.220759 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921103    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 13:27:15.220759 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921107    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 13:27:15.220759 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921111    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 13:27:15.220759 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921114    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 13:27:15.220759 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921117    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 13:27:15.220759 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921120    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 13:27:15.220759 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921123    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 13:27:15.220759 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921128    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 13:27:15.220759 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921130    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 13:27:15.221404 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921134    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 13:27:15.221404 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921138    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 13:27:15.221404 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921141    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 13:27:15.221404 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921143    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 13:27:15.221404 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921146    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 13:27:15.221404 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921149    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 13:27:15.221404 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921152    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 13:27:15.221404 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921154    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 13:27:15.221404 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921156    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 13:27:15.221404 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921159    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 13:27:15.221404 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921162    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 13:27:15.221404 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921164    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 13:27:15.221404 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921169    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 13:27:15.221404 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921172    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 13:27:15.221404 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921174    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 13:27:15.221404 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921176    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 13:27:15.221404 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921179    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 13:27:15.221404 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921182    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 13:27:15.221404 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921184    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 13:27:15.221404 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921187    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 13:27:15.222059 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921189    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 13:27:15.222059 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921191    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 13:27:15.222059 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921194    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 13:27:15.222059 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921196    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 13:27:15.222059 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921199    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 13:27:15.222059 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921202    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 13:27:15.222059 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921205    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 13:27:15.222059 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921208    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 13:27:15.222059 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921210    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 13:27:15.222059 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921212    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 13:27:15.222059 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921216    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 13:27:15.222059 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921219    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 13:27:15.222059 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921221    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 13:27:15.222059 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921223    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 13:27:15.222059 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921226    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 13:27:15.222059 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921228    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 13:27:15.222059 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921231    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 13:27:15.222059 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921233    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 13:27:15.222059 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921236    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 13:27:15.222059 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921238    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 13:27:15.222668 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921241    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 13:27:15.222668 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921243    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 13:27:15.222668 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921245    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 13:27:15.222668 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921248    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 13:27:15.222668 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921252    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 13:27:15.222668 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921254    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 13:27:15.222668 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921257    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 13:27:15.222668 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921259    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 13:27:15.222668 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921261    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 13:27:15.222668 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921264    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 13:27:15.222668 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921266    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 13:27:15.222668 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921269    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 13:27:15.222668 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921272    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 13:27:15.222668 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921274    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 13:27:15.222668 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921276    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 13:27:15.222668 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921279    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 13:27:15.222668 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921281    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 13:27:15.222668 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921284    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 13:27:15.222668 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921286    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 13:27:15.222668 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921289    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 13:27:15.223382 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921291    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 13:27:15.223382 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921294    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 13:27:15.223382 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921298    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 13:27:15.223382 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921300    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 13:27:15.223382 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921303    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 13:27:15.223382 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921305    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 13:27:15.223382 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921308    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 13:27:15.223382 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921310    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 13:27:15.223382 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921313    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 13:27:15.223382 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921315    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 13:27:15.223382 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921317    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 13:27:15.223382 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921320    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 13:27:15.223382 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921323    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 13:27:15.223382 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921325    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 13:27:15.223382 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921327    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 13:27:15.223382 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921330    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 13:27:15.223382 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921333    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 13:27:15.223382 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921336    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 13:27:15.223382 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.921338    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 13:27:15.223976 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.921885    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 13:27:15.223976 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.929619    2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 14 13:27:15.223976 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.929633    2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 14 13:27:15.223976 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.929652    2569 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 14 13:27:15.223976 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.929660    2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 14 13:27:15.223976 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929730    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 13:27:15.223976 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929735    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 13:27:15.223976 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929738    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 13:27:15.223976 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929741    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 13:27:15.223976 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929746    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 13:27:15.223976 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929750    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 13:27:15.223976 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929755    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 13:27:15.223976 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929759    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 13:27:15.223976 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929764    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 13:27:15.223976 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929767    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 13:27:15.224504 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929770    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 13:27:15.224504 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929774    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 13:27:15.224504 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929776    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 13:27:15.224504 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929779    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 13:27:15.224504 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929782    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 13:27:15.224504 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929784    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 13:27:15.224504 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929787    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 13:27:15.224504 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929790    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 13:27:15.224504 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929792    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 13:27:15.224504 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929797    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 13:27:15.224504 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929800    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 13:27:15.224504 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929802    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 13:27:15.224504 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929806    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 13:27:15.224504 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929810    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 13:27:15.224504 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929813    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 13:27:15.224504 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929817    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 13:27:15.224504 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929820    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 13:27:15.224504 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929825    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 13:27:15.224504 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929827    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 13:27:15.224504 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929832    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 13:27:15.225132 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929834    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 13:27:15.225132 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929837    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 13:27:15.225132 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929841    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 13:27:15.225132 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929844    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 13:27:15.225132 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929846    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 13:27:15.225132 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929849    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 13:27:15.225132 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929851    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 13:27:15.225132 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929854    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 13:27:15.225132 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929856    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 13:27:15.225132 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929859    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 13:27:15.225132 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929861    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 13:27:15.225132 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929863    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 13:27:15.225132 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929866    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 13:27:15.225132 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929872    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 13:27:15.225132 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929875    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 13:27:15.225132 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929878    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 13:27:15.225132 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929880    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 13:27:15.225132 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929883    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 13:27:15.225132 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929886    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 13:27:15.225132 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929889    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 13:27:15.225824 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929891    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 13:27:15.225824 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929893    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 13:27:15.225824 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929896    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 13:27:15.225824 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929898    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 13:27:15.225824 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929901    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 13:27:15.225824 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929903    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 13:27:15.225824 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929906    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 13:27:15.225824 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929908    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 13:27:15.225824 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929911    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 13:27:15.225824 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929914    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 13:27:15.225824 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929916    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 13:27:15.225824 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929919    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 13:27:15.225824 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929921    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 13:27:15.225824 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929925    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 13:27:15.225824 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929929    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 13:27:15.225824 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929931    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 13:27:15.225824 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929933    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 13:27:15.225824 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929936    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 13:27:15.225824 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929938    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 13:27:15.226426 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929941    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 13:27:15.226426 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929943    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 13:27:15.226426 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929945    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 13:27:15.226426 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929948    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 13:27:15.226426 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929950    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 13:27:15.226426 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929953    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 13:27:15.226426 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929959    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 13:27:15.226426 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929961    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 13:27:15.226426 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929964    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 13:27:15.226426 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929966    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 13:27:15.226426 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929972    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 13:27:15.226426 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929975    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 13:27:15.226426 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929978    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 13:27:15.226426 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929981    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 13:27:15.226426 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929984    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 13:27:15.226426 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929987    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 13:27:15.226426 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929991    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 13:27:15.226426 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929994    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 13:27:15.226426 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.929998    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 13:27:15.227039 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930000    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 13:27:15.227039 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930003    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 13:27:15.227039 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930022    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 13:27:15.227039 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930025    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 13:27:15.227039 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930028    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 13:27:15.227039 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930031    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 13:27:15.227039 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930033    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 13:27:15.227039 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930036    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 13:27:15.227039 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930038    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 13:27:15.227039 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930040    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 13:27:15.227039 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930043    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 13:27:15.227039 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.930048    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 13:27:15.227039 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930170    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 13:27:15.227039 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930175    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 13:27:15.227039 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930178    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 13:27:15.227466 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930181    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 13:27:15.227466 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930184    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 13:27:15.227466 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930186    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 13:27:15.227466 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930189    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 13:27:15.227466 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930194    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 13:27:15.227466 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930200    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 13:27:15.227466 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930202    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 13:27:15.227466 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930205    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 13:27:15.227466 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930208    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 13:27:15.227466 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930211    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 13:27:15.227466 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930214    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 13:27:15.227466 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930216    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 13:27:15.227466 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930219    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 13:27:15.227466 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930221    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 13:27:15.227466 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930224    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 13:27:15.227466 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930226    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 13:27:15.227466 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930229    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 13:27:15.227466 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930231    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 13:27:15.227466 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930234    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 13:27:15.227466 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930236    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 13:27:15.228091 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930239    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 13:27:15.228091 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930241    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 13:27:15.228091 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930243    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 13:27:15.228091 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930246    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 13:27:15.228091 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930248    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 13:27:15.228091 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930251    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 13:27:15.228091 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930253    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 13:27:15.228091 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930255    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 13:27:15.228091 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930259    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 13:27:15.228091 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930263    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 13:27:15.228091 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930266    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 13:27:15.228091 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930269    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 13:27:15.228091 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930271    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 13:27:15.228091 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930274    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 13:27:15.228091 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930277    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 13:27:15.228091 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930279    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 13:27:15.228091 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930284    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 13:27:15.228091 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930288    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 13:27:15.228575 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930291    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 13:27:15.228575 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930293    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 13:27:15.228575 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930296    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 13:27:15.228575 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930298    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 13:27:15.228575 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930301    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 13:27:15.228575 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930304    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 13:27:15.228575 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930306    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 13:27:15.228575 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930309    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 13:27:15.228575 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930311    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 13:27:15.228575 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930314    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 13:27:15.228575 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930317    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 13:27:15.228575 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930319    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 13:27:15.228575 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930321    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 13:27:15.228575 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930324    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 13:27:15.228575 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930326    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 13:27:15.228575 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930329    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 13:27:15.228575 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930332    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 13:27:15.228575 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930335    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 13:27:15.228575 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930338    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 13:27:15.228575 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930341    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 13:27:15.229158 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930344    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 13:27:15.229158 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930347    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 13:27:15.229158 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930350    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 13:27:15.229158 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930353    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 13:27:15.229158 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930356    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 13:27:15.229158 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930359    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 13:27:15.229158 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930361    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 13:27:15.229158 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930364    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 13:27:15.229158 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930366    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 13:27:15.229158 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930369    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 13:27:15.229158 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930373    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 13:27:15.229158 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930377    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 13:27:15.229158 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930380    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 13:27:15.229158 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930383    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 13:27:15.229158 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930385    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 13:27:15.229158 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930388    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 13:27:15.229158 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930390    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 13:27:15.229158 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930394    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 13:27:15.229158 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930396    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 13:27:15.229158 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930398    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 13:27:15.229686 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930401    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 13:27:15.229686 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930403    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 13:27:15.229686 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930406    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 13:27:15.229686 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930408    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 13:27:15.229686 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930411    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 13:27:15.229686 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930413    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 13:27:15.229686 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930415    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 13:27:15.229686 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930418    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 13:27:15.229686 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930421    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 13:27:15.229686 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930423    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 13:27:15.229686 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930425    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 13:27:15.229686 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930428    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 13:27:15.229686 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930430    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 13:27:15.229686 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930433    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 13:27:15.229686 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930435    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 13:27:15.229686 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930437    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 13:27:15.229686 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930439    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 13:27:15.229686 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:14.930442    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 13:27:15.230212 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.930447    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 13:27:15.230212 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.930555    2569 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 14 13:27:15.230212 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.933943    2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 14 13:27:15.230212 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.936730    2569 server.go:1019] "Starting client certificate rotation"
Jul 14 13:27:15.230212 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.936832    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 14 13:27:15.230212 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.936862    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 14 13:27:15.230212 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.941513    2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 14 13:27:15.230212 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.942686    2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 14 13:27:15.230212 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.957042    2569 log.go:25] "Validated CRI v1 runtime API"
Jul 14 13:27:15.230212 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.960617    2569 log.go:25] "Validated CRI v1 image API"
Jul 14 13:27:15.230212 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.962096    2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 14 13:27:15.230212 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.967147    2569 fs.go:135] Filesystem UUIDs: map[6f0fb38e-5785-4627-970e-837056cc7499:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 bb2a3e2c-01ac-4fc7-ba02-2bace43adb87:/dev/nvme0n1p4]
Jul 14 13:27:15.232470 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.967176    2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 14 13:27:15.232470 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.971186    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 14 13:27:15.510007 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.973971    2569 manager.go:216] Machine: {Timestamp:2026-07-14 13:27:14.973133707 +0000 UTC m=+0.299453627 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097827 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b15aabd590c33ff8e9e9328818b95 SystemUUID:ec2b15aa-bd59-0c33-ff8e-9e9328818b95 BootID:9ede8f38-fd19-43e8-85f4-4d41cea7adad Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a9:dc:9f:91:37 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a9:dc:9f:91:37 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:d7:48:4b:77:81 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 14 13:27:15.510007 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.974168    2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 14 13:27:15.510007 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.974259    2569 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 14 13:27:15.510007 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.976887    2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 14 13:27:15.510007 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.976916    2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-78.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 14 13:27:15.510007 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.977655    2569 topology_manager.go:138] "Creating topology manager with none policy"
Jul 14 13:27:15.510007 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.977669    2569 container_manager_linux.go:309] "Creating device plugin manager"
Jul 14 13:27:15.510007 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.977681    2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 14 13:27:15.510007 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.978328    2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 14 13:27:15.510007 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.979064    2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 14 13:27:15.510007 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.981831    2569 state_mem.go:36] "Initialized new in-memory state store"
Jul 14 13:27:15.510007 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.981987    2569 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 14 13:27:15.510007 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.982037    2569 kubelet.go:486] "Attempting to sync node with API server"
Jul 14 13:27:15.510007 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.982060    2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 14 13:27:15.510007 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.982086    2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 14 13:27:15.510007 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.982159    2569 kubelet.go:398] "Adding apiserver pod source"
Jul 14 13:27:15.510007 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.982171    2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 14 13:27:15.510007 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.982661    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.983312    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.985709    2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.986952    2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.986979    2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.988104    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.988119    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.988125    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.988131    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.988139    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.988144    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.988152    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.988157    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.988163    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.988168    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.988193    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.988198    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.988858    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.988864    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.991922    2569 server.go:1268] "Started kubelet"
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.992022    2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.992091    2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.992143    2569 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.992374    2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-78.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:14.992447    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-78.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:14.992480    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.993245    2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.994781    2569 server.go:310] "Adding debug handlers to kubelet server"
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:14.995789    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-78.ec2.internal.18c22af53254841b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-78.ec2.internal,UID:ip-10-0-140-78.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-78.ec2.internal,},FirstTimestamp:2026-07-14 13:27:14.991891483 +0000 UTC m=+0.318211402,LastTimestamp:2026-07-14 13:27:14.991891483 +0000 UTC m=+0.318211402,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-78.ec2.internal,}"
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.997914    2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.997926    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:14.998002    2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:14.998055    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-78.ec2.internal\" not found"
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.998537    2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.998547    2569 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.998567    2569 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.998739    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dx627"
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.998790    2569 factory.go:55] Registering systemd factory
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.998802    2569 factory.go:223] Registration of the systemd container factory successfully
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.998816    2569 reconstruct.go:97] "Volume reconstruction finished"
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.998825    2569 reconciler.go:29] "Reconciler: start to sync state"
Jul 14 13:27:15.511278 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.998997    2569 factory.go:153] Registering CRI-O factory
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.999032    2569 factory.go:223] Registration of the crio container factory successfully
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.999101    2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.999129    2569 factory.go:103] Registering Raw factory
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.999144    2569 manager.go:1196] Started watching for new ooms in manager
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:14.999490    2569 manager.go:349] Starting recovery of all containers
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:15.001827    2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-78.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:15.001930    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.009236    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dx627"
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.010066    2569 manager.go:354] Recovery completed
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.018086    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.019833    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-78.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.019857    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-78.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.019867    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-78.ec2.internal" event="NodeHasSufficientPID"
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.020316    2569 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.020324    2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.020338    2569 state_mem.go:36] "Initialized new in-memory state store"
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.020338    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.020982    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-78.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.021003    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-78.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.021028    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-78.ec2.internal" event="NodeHasSufficientPID"
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.021286    2569 policy_none.go:49] "None policy: Start"
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.021298    2569 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.021308    2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.022135    2569 policy_none.go:47] "Start"
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:15.022286    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-78.ec2.internal.18c22af533ff0c9a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-78.ec2.internal,UID:ip-10-0-140-78.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-78.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-78.ec2.internal,},FirstTimestamp:2026-07-14 13:27:15.019844762 +0000 UTC m=+0.346164683,LastTimestamp:2026-07-14 13:27:15.019844762 +0000 UTC m=+0.346164683,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-78.ec2.internal,}"
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.050303    2569 manager.go:337] "Starting Device Plugin manager"
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:15.050466    2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.050480    2569 server.go:87] "Starting device plugin registration server"
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.050675    2569 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.050685    2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.050755    2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.050822    2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.050831    2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:15.051410    2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:15.051448    2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-78.ec2.internal\" not found"
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.151389    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.152147    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-78.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 13:27:15.512568 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.152165    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-78.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 13:27:15.513734 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.152174    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-78.ec2.internal" event="NodeHasSufficientPID"
Jul 14 13:27:15.513734 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.152197    2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-78.ec2.internal"
Jul 14 13:27:15.513734 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.161695    2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-78.ec2.internal"
Jul 14 13:27:15.513734 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:15.161711    2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-78.ec2.internal\": node \"ip-10-0-140-78.ec2.internal\" not found"
Jul 14 13:27:15.513734 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.182280    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 14 13:27:15.513734 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.186503    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 14 13:27:15.513734 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.186532    2569 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 14 13:27:15.513734 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.186548    2569 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 14 13:27:15.513734 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:15.186575    2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 14 13:27:15.513734 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.199152    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 13:27:15.513734 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:15.233732    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-78.ec2.internal\" not found"
Jul 14 13:27:15.513734 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.287588    2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-78.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-78.ec2.internal"]
Jul 14 13:27:15.513734 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.287653    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 13:27:15.513734 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.288483    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-78.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 13:27:15.513734 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.288506    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-78.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 13:27:15.513734 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.288518    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-78.ec2.internal" event="NodeHasSufficientPID"
Jul 14 13:27:15.513734 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.289779    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 13:27:15.513734 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.289889    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-78.ec2.internal"
Jul 14 13:27:15.513734 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.289921    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 13:27:15.513734 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.290436    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-78.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 13:27:15.513734 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.290454    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-78.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 13:27:15.513734 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.290475    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-78.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 13:27:15.513734 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.290484    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-78.ec2.internal" event="NodeHasSufficientPID"
Jul 14 13:27:15.513734 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.290457    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-78.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 13:27:15.513734 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.290539    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-78.ec2.internal" event="NodeHasSufficientPID"
Jul 14 13:27:15.513734 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.291473    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-78.ec2.internal"
Jul 14 13:27:15.513734 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.291494    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 13:27:15.513734 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.292067    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-78.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 13:27:15.513734 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.292093    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-78.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 13:27:15.513734 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.292103    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-78.ec2.internal" event="NodeHasSufficientPID"
Jul 14 13:27:15.513734 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.299344    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/742561d943373579cd7f313233325224-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-78.ec2.internal\" (UID: \"742561d943373579cd7f313233325224\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-78.ec2.internal"
Jul 14 13:27:15.513734 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.299379    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/742561d943373579cd7f313233325224-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-78.ec2.internal\" (UID: \"742561d943373579cd7f313233325224\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-78.ec2.internal"
Jul 14 13:27:15.513734 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.299403    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9dc64344bcf6e13eee2ce8f9a1eefa2b-config\") pod \"kube-apiserver-proxy-ip-10-0-140-78.ec2.internal\" (UID: \"9dc64344bcf6e13eee2ce8f9a1eefa2b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-78.ec2.internal"
Jul 14 13:27:15.513734 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:15.319465    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-78.ec2.internal\" not found" node="ip-10-0-140-78.ec2.internal"
Jul 14 13:27:15.513734 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:15.323715    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-78.ec2.internal\" not found" node="ip-10-0-140-78.ec2.internal"
Jul 14 13:27:15.514782 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:15.334909    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-78.ec2.internal\" not found"
Jul 14 13:27:15.514782 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.399515    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9dc64344bcf6e13eee2ce8f9a1eefa2b-config\") pod \"kube-apiserver-proxy-ip-10-0-140-78.ec2.internal\" (UID: \"9dc64344bcf6e13eee2ce8f9a1eefa2b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-78.ec2.internal"
Jul 14 13:27:15.514782 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.399545    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/742561d943373579cd7f313233325224-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-78.ec2.internal\" (UID: \"742561d943373579cd7f313233325224\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-78.ec2.internal"
Jul 14 13:27:15.514782 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.399565    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/742561d943373579cd7f313233325224-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-78.ec2.internal\" (UID: \"742561d943373579cd7f313233325224\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-78.ec2.internal"
Jul 14 13:27:15.514782 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.399591    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/742561d943373579cd7f313233325224-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-78.ec2.internal\" (UID: \"742561d943373579cd7f313233325224\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-78.ec2.internal"
Jul 14 13:27:15.514782 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.399620    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9dc64344bcf6e13eee2ce8f9a1eefa2b-config\") pod \"kube-apiserver-proxy-ip-10-0-140-78.ec2.internal\" (UID: \"9dc64344bcf6e13eee2ce8f9a1eefa2b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-78.ec2.internal"
Jul 14 13:27:15.514782 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.399623    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/742561d943373579cd7f313233325224-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-78.ec2.internal\" (UID: \"742561d943373579cd7f313233325224\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-78.ec2.internal"
Jul 14 13:27:15.514782 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:15.434995    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-78.ec2.internal\" not found"
Jul 14 13:27:15.733929 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:15.536489    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-78.ec2.internal\" not found"
Jul 14 13:27:15.733929 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.623385    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-78.ec2.internal"
Jul 14 13:27:15.733929 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.626099    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-78.ec2.internal"
Jul 14 13:27:15.733929 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:15.637342    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-78.ec2.internal\" not found"
Jul 14 13:27:16.006363 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:15.737485    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-78.ec2.internal\" not found"
Jul 14 13:27:16.006363 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:15.840049    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-78.ec2.internal\" not found"
Jul 14 13:27:16.006363 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.912695    2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 13:27:16.006363 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.935321    2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 14 13:27:16.006363 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.935420    2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 14 13:27:16.006363 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.935468    2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 14 13:27:16.006363 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.936190    2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 13:27:16.006363 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:15.940133    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-78.ec2.internal\" not found"
Jul 14 13:27:16.006363 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.957854    2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 13:27:16.006363 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.983481    2569 apiserver.go:52] "Watching apiserver"
Jul 14 13:27:16.006363 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.994767    2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 14 13:27:16.006363 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.995747    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9sb4","openshift-cluster-node-tuning-operator/tuned-xwg7t","openshift-multus/multus-mqppd","openshift-network-operator/iptables-alerter-8qsng","openshift-dns/node-resolver-mjgwk","openshift-image-registry/node-ca-7l994","openshift-multus/multus-additional-cni-plugins-dhtvj","openshift-multus/network-metrics-daemon-w4rvb","openshift-network-diagnostics/network-check-target-42hkp","openshift-ovn-kubernetes/ovnkube-node-vq8tk","kube-system/global-pull-secret-syncer-s7ft2","kube-system/konnectivity-agent-kkhvt"]
Jul 14 13:27:16.006363 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.997305    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9sb4"
Jul 14 13:27:16.006363 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.998287    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-78.ec2.internal"
Jul 14 13:27:16.006363 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.998328    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 14 13:27:16.006363 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.998636    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.006363 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:15.999919    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.006363 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.001124    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8qsng"
Jul 14 13:27:16.006363 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.001306    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 14 13:27:16.006363 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.002441    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mjgwk"
Jul 14 13:27:16.006363 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.003668    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7l994"
Jul 14 13:27:16.006363 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.004611    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98bacfcd-4615-4bde-a9e5-6d908367260e-device-dir\") pod \"aws-ebs-csi-driver-node-q9sb4\" (UID: \"98bacfcd-4615-4bde-a9e5-6d908367260e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9sb4"
Jul 14 13:27:16.006363 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.004644    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff284a1c-edb3-4049-a1e2-2c803aceaa62-lib-modules\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.006363 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.004672    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff284a1c-edb3-4049-a1e2-2c803aceaa62-var-lib-kubelet\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.006363 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.004700    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff284a1c-edb3-4049-a1e2-2c803aceaa62-tmp\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.006363 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.004725    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-cni-binary-copy\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.006363 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.004749    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-host-var-lib-cni-multus\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.007916 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.004782    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-host-var-lib-kubelet\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.007916 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.004816    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-host-run-multus-certs\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.007916 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.004835    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ff284a1c-edb3-4049-a1e2-2c803aceaa62-etc-sysctl-d\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.007916 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.004859    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ff284a1c-edb3-4049-a1e2-2c803aceaa62-etc-sysctl-conf\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.007916 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.004878    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff284a1c-edb3-4049-a1e2-2c803aceaa62-sys\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.007916 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.004901    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff284a1c-edb3-4049-a1e2-2c803aceaa62-host\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.007916 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.004917    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ff284a1c-edb3-4049-a1e2-2c803aceaa62-etc-tuned\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.007916 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.004931    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-hostroot\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.007916 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.004965    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98bacfcd-4615-4bde-a9e5-6d908367260e-socket-dir\") pod \"aws-ebs-csi-driver-node-q9sb4\" (UID: \"98bacfcd-4615-4bde-a9e5-6d908367260e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9sb4"
Jul 14 13:27:16.007916 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.004988    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98bacfcd-4615-4bde-a9e5-6d908367260e-etc-selinux\") pod \"aws-ebs-csi-driver-node-q9sb4\" (UID: \"98bacfcd-4615-4bde-a9e5-6d908367260e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9sb4"
Jul 14 13:27:16.007916 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.005015    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vbm5\" (UniqueName: \"kubernetes.io/projected/98bacfcd-4615-4bde-a9e5-6d908367260e-kube-api-access-2vbm5\") pod \"aws-ebs-csi-driver-node-q9sb4\" (UID: \"98bacfcd-4615-4bde-a9e5-6d908367260e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9sb4"
Jul 14 13:27:16.007916 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.005056    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dhtvj"
Jul 14 13:27:16.007916 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.005085    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 14 13:27:16.007916 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.005056    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-os-release\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.007916 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.005180    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-multus-socket-dir-parent\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.007916 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.005200    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98bacfcd-4615-4bde-a9e5-6d908367260e-registration-dir\") pod \"aws-ebs-csi-driver-node-q9sb4\" (UID: \"98bacfcd-4615-4bde-a9e5-6d908367260e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9sb4"
Jul 14 13:27:16.007916 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.005216    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98bacfcd-4615-4bde-a9e5-6d908367260e-sys-fs\") pod \"aws-ebs-csi-driver-node-q9sb4\" (UID: \"98bacfcd-4615-4bde-a9e5-6d908367260e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9sb4"
Jul 14 13:27:16.007916 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.005237    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-cnibin\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.008796 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.005266    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-host-run-netns\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.009925 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.005298    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x67gf\" (UniqueName: \"kubernetes.io/projected/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-kube-api-access-x67gf\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.009925 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.009663    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ff284a1c-edb3-4049-a1e2-2c803aceaa62-etc-modprobe-d\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.009925 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.009686    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98bacfcd-4615-4bde-a9e5-6d908367260e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q9sb4\" (UID: \"98bacfcd-4615-4bde-a9e5-6d908367260e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9sb4"
Jul 14 13:27:16.009925 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.009707    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csc58\" (UniqueName: \"kubernetes.io/projected/ff284a1c-edb3-4049-a1e2-2c803aceaa62-kube-api-access-csc58\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.009925 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.009730    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-system-cni-dir\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.009925 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.009751    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-multus-cni-dir\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.009925 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.009797    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-host-run-k8s-cni-cncf-io\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.009925 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.009856    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-multus-daemon-config\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.009925 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.009892    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ff284a1c-edb3-4049-a1e2-2c803aceaa62-etc-systemd\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.009925 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.009916    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-host-var-lib-cni-bin\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.010567 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.010112    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-etc-kubernetes\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.010567 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.010148    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ff284a1c-edb3-4049-a1e2-2c803aceaa62-etc-sysconfig\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.010567 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.010174    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff284a1c-edb3-4049-a1e2-2c803aceaa62-etc-kubernetes\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.010567 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.010199    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ff284a1c-edb3-4049-a1e2-2c803aceaa62-run\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.010567 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.010227    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-multus-conf-dir\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.010567 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.010408    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4rvb"
Jul 14 13:27:16.010567 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:16.010478    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4rvb" podUID="fd9c8cb5-09e3-4116-b407-01ebe6c06999"
Jul 14 13:27:16.011893 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.011767    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42hkp"
Jul 14 13:27:16.011893 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:16.011828    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42hkp" podUID="d87134ea-5c27-47f7-a629-883bae028590"
Jul 14 13:27:16.188981 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.013032    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.188981 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.013553    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 14 13:27:16.188981 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.013628    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 14 13:27:16.188981 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.013668    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 14 13:27:16.188981 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.013844    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 14 13:27:16.188981 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.013935    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 14 13:27:16.188981 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.013960    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 14 13:27:16.188981 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.014636    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:27:16.188981 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:16.014685    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7ft2" podUID="b33bdbd5-63be-4828-a2d6-91d4bead7b80"
Jul 14 13:27:16.188981 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.014850    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 14 13:27:16.188981 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.014859    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-13 13:22:15 +0000 UTC" deadline="2028-03-20 01:09:53.225436241 +0000 UTC"
Jul 14 13:27:16.188981 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.014878    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14747h42m37.210560591s"
Jul 14 13:27:16.188981 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.015731    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 14 13:27:16.188981 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.015760    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 14 13:27:16.188981 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.016035    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xh88p\""
Jul 14 13:27:16.188981 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.016117    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fwrqk\""
Jul 14 13:27:16.188981 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.016142    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kkhvt"
Jul 14 13:27:16.188981 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.016225    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 14 13:27:16.188981 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.016627    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 14 13:27:16.188981 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.016649    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 14 13:27:16.188981 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.017014    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 14 13:27:16.188981 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.017077    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-r5cv6\""
Jul 14 13:27:16.188981 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.017278    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 14 13:27:16.188981 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.017484    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 14 13:27:16.188981 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.017540    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 14 13:27:16.188981 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.017756    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jz5kp\""
Jul 14 13:27:16.188981 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.017882    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kj27l\""
Jul 14 13:27:16.188981 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.018231    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bksxw\""
Jul 14 13:27:16.188981 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.018285    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 14 13:27:16.188981 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.018400    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-25p99\""
Jul 14 13:27:16.188981 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.026777    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 14 13:27:16.188981 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.031652    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 14 13:27:16.188981 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.031663    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 14 13:27:16.188981 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.032900    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 14 13:27:16.188981 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.032904    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 14 13:27:16.190101 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.033339    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 14 13:27:16.190101 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.033402    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 14 13:27:16.190101 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.033460    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-78.ec2.internal"
Jul 14 13:27:16.190101 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.033467    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 14 13:27:16.190101 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.033604    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-ts6z6\""
Jul 14 13:27:16.190101 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.033641    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-78.ec2.internal"]
Jul 14 13:27:16.190101 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.034395    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cstbr\""
Jul 14 13:27:16.190101 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.034588    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 14 13:27:16.190101 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.037709    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 14 13:27:16.190101 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.046033    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-78.ec2.internal"]
Jul 14 13:27:16.190101 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.046435    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 14 13:27:16.190101 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.057665    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gktkx"
Jul 14 13:27:16.190101 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.075103    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gktkx"
Jul 14 13:27:16.190101 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.100334    2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 14 13:27:16.190101 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.110846    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-host-run-netns\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.190101 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.110870    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98bacfcd-4615-4bde-a9e5-6d908367260e-sys-fs\") pod \"aws-ebs-csi-driver-node-q9sb4\" (UID: \"98bacfcd-4615-4bde-a9e5-6d908367260e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9sb4"
Jul 14 13:27:16.190101 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.110886    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-cnibin\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.190101 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.110903    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-var-lib-openvswitch\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.190101 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.110927    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-run-ovn\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.190101 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.110957    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98bacfcd-4615-4bde-a9e5-6d908367260e-sys-fs\") pod \"aws-ebs-csi-driver-node-q9sb4\" (UID: \"98bacfcd-4615-4bde-a9e5-6d908367260e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9sb4"
Jul 14 13:27:16.190101 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.110970    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ff284a1c-edb3-4049-a1e2-2c803aceaa62-etc-modprobe-d\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.190101 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.110990    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-cnibin\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.190101 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.110993    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/88a55d43-cfd1-433c-944b-8de4720da8af-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dhtvj\" (UID: \"88a55d43-cfd1-433c-944b-8de4720da8af\") " pod="openshift-multus/multus-additional-cni-plugins-dhtvj"
Jul 14 13:27:16.190101 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111029    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j929x\" (UniqueName: \"kubernetes.io/projected/88a55d43-cfd1-433c-944b-8de4720da8af-kube-api-access-j929x\") pod \"multus-additional-cni-plugins-dhtvj\" (UID: \"88a55d43-cfd1-433c-944b-8de4720da8af\") " pod="openshift-multus/multus-additional-cni-plugins-dhtvj"
Jul 14 13:27:16.190991 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111049    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-log-socket\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.190991 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111065    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e33a489f-7d68-4e6c-9ffe-e7252049e15d-ovnkube-script-lib\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.190991 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111081    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ff284a1c-edb3-4049-a1e2-2c803aceaa62-etc-modprobe-d\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.190991 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111081    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dxbb\" (UniqueName: \"kubernetes.io/projected/e33a489f-7d68-4e6c-9ffe-e7252049e15d-kube-api-access-6dxbb\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.190991 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111128    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98bacfcd-4615-4bde-a9e5-6d908367260e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q9sb4\" (UID: \"98bacfcd-4615-4bde-a9e5-6d908367260e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9sb4"
Jul 14 13:27:16.190991 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111148    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-system-cni-dir\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.190991 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111163    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-multus-cni-dir\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.190991 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111191    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98bacfcd-4615-4bde-a9e5-6d908367260e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q9sb4\" (UID: \"98bacfcd-4615-4bde-a9e5-6d908367260e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9sb4"
Jul 14 13:27:16.190991 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111195    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-host-run-k8s-cni-cncf-io\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.190991 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111212    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-system-cni-dir\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.190991 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111223    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-host-run-k8s-cni-cncf-io\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.190991 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111225    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfqnw\" (UniqueName: \"kubernetes.io/projected/03018596-d0b1-43da-af02-ede52df67ceb-kube-api-access-qfqnw\") pod \"node-resolver-mjgwk\" (UID: \"03018596-d0b1-43da-af02-ede52df67ceb\") " pod="openshift-dns/node-resolver-mjgwk"
Jul 14 13:27:16.190991 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111243    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q27h2\" (UniqueName: \"kubernetes.io/projected/fd9c8cb5-09e3-4116-b407-01ebe6c06999-kube-api-access-q27h2\") pod \"network-metrics-daemon-w4rvb\" (UID: \"fd9c8cb5-09e3-4116-b407-01ebe6c06999\") " pod="openshift-multus/network-metrics-daemon-w4rvb"
Jul 14 13:27:16.190991 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111259    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ff4m\" (UniqueName: \"kubernetes.io/projected/d87134ea-5c27-47f7-a629-883bae028590-kube-api-access-9ff4m\") pod \"network-check-target-42hkp\" (UID: \"d87134ea-5c27-47f7-a629-883bae028590\") " pod="openshift-network-diagnostics/network-check-target-42hkp"
Jul 14 13:27:16.190991 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111275    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/88a55d43-cfd1-433c-944b-8de4720da8af-cnibin\") pod \"multus-additional-cni-plugins-dhtvj\" (UID: \"88a55d43-cfd1-433c-944b-8de4720da8af\") " pod="openshift-multus/multus-additional-cni-plugins-dhtvj"
Jul 14 13:27:16.190991 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111304    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ff284a1c-edb3-4049-a1e2-2c803aceaa62-etc-systemd\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.190991 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111323    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-host-var-lib-cni-bin\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.191686 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111360    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-host-var-lib-cni-bin\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.191686 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111364    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ff284a1c-edb3-4049-a1e2-2c803aceaa62-etc-systemd\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.191686 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111365    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-multus-cni-dir\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.191686 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111379    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-etc-kubernetes\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.191686 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111415    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-etc-kubernetes\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.191686 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111412    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/88a55d43-cfd1-433c-944b-8de4720da8af-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dhtvj\" (UID: \"88a55d43-cfd1-433c-944b-8de4720da8af\") " pod="openshift-multus/multus-additional-cni-plugins-dhtvj"
Jul 14 13:27:16.191686 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111448    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.191686 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111471    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff284a1c-edb3-4049-a1e2-2c803aceaa62-etc-kubernetes\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.191686 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111497    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-multus-conf-dir\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.191686 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111524    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff284a1c-edb3-4049-a1e2-2c803aceaa62-etc-kubernetes\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.191686 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111534    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/62b825ef-ccb4-4f79-bd95-c0cb2753b4f6-agent-certs\") pod \"konnectivity-agent-kkhvt\" (UID: \"62b825ef-ccb4-4f79-bd95-c0cb2753b4f6\") " pod="kube-system/konnectivity-agent-kkhvt"
Jul 14 13:27:16.191686 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111566    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-multus-conf-dir\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.191686 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111585    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/62b825ef-ccb4-4f79-bd95-c0cb2753b4f6-konnectivity-ca\") pod \"konnectivity-agent-kkhvt\" (UID: \"62b825ef-ccb4-4f79-bd95-c0cb2753b4f6\") " pod="kube-system/konnectivity-agent-kkhvt"
Jul 14 13:27:16.191686 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111607    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61151274-edf7-4e83-adfd-be582e30ad29-host-slash\") pod \"iptables-alerter-8qsng\" (UID: \"61151274-edf7-4e83-adfd-be582e30ad29\") " pod="openshift-network-operator/iptables-alerter-8qsng"
Jul 14 13:27:16.191686 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111629    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bf204a1f-63ca-40ac-825b-e8684c67e36f-serviceca\") pod \"node-ca-7l994\" (UID: \"bf204a1f-63ca-40ac-825b-e8684c67e36f\") " pod="openshift-image-registry/node-ca-7l994"
Jul 14 13:27:16.191686 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111643    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-run-systemd\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.191686 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111675    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98bacfcd-4615-4bde-a9e5-6d908367260e-device-dir\") pod \"aws-ebs-csi-driver-node-q9sb4\" (UID: \"98bacfcd-4615-4bde-a9e5-6d908367260e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9sb4"
Jul 14 13:27:16.192457 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111706    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff284a1c-edb3-4049-a1e2-2c803aceaa62-lib-modules\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.192457 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111729    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff284a1c-edb3-4049-a1e2-2c803aceaa62-var-lib-kubelet\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.192457 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111750    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-cni-binary-copy\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.192457 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111769    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98bacfcd-4615-4bde-a9e5-6d908367260e-device-dir\") pod \"aws-ebs-csi-driver-node-q9sb4\" (UID: \"98bacfcd-4615-4bde-a9e5-6d908367260e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9sb4"
Jul 14 13:27:16.192457 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111801    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff284a1c-edb3-4049-a1e2-2c803aceaa62-var-lib-kubelet\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.192457 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111801    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ff284a1c-edb3-4049-a1e2-2c803aceaa62-lib-modules\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.192457 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111833    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-host-var-lib-cni-multus\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.192457 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111925    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-host-var-lib-kubelet\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.192457 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111935    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-host-var-lib-cni-multus\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.192457 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111975    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03018596-d0b1-43da-af02-ede52df67ceb-tmp-dir\") pod \"node-resolver-mjgwk\" (UID: \"03018596-d0b1-43da-af02-ede52df67ceb\") " pod="openshift-dns/node-resolver-mjgwk"
Jul 14 13:27:16.192457 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.111992    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e33a489f-7d68-4e6c-9ffe-e7252049e15d-ovn-node-metrics-cert\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.192457 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112010    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ff284a1c-edb3-4049-a1e2-2c803aceaa62-etc-sysctl-d\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.192457 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112040    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-host-var-lib-kubelet\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.192457 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112086    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ff284a1c-edb3-4049-a1e2-2c803aceaa62-etc-sysctl-conf\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.192457 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112122    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ff284a1c-edb3-4049-a1e2-2c803aceaa62-etc-sysctl-d\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.192457 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112128    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff284a1c-edb3-4049-a1e2-2c803aceaa62-sys\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.192457 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112174    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff284a1c-edb3-4049-a1e2-2c803aceaa62-host\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.192457 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112198    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ff284a1c-edb3-4049-a1e2-2c803aceaa62-etc-tuned\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.193588 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112230    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ff284a1c-edb3-4049-a1e2-2c803aceaa62-etc-sysctl-conf\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.193588 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112242    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd9c8cb5-09e3-4116-b407-01ebe6c06999-metrics-certs\") pod \"network-metrics-daemon-w4rvb\" (UID: \"fd9c8cb5-09e3-4116-b407-01ebe6c06999\") " pod="openshift-multus/network-metrics-daemon-w4rvb"
Jul 14 13:27:16.193588 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112251    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff284a1c-edb3-4049-a1e2-2c803aceaa62-host\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.193588 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112302    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff284a1c-edb3-4049-a1e2-2c803aceaa62-sys\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.193588 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112325    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf204a1f-63ca-40ac-825b-e8684c67e36f-host\") pod \"node-ca-7l994\" (UID: \"bf204a1f-63ca-40ac-825b-e8684c67e36f\") " pod="openshift-image-registry/node-ca-7l994"
Jul 14 13:27:16.193588 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112349    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vbm5\" (UniqueName: \"kubernetes.io/projected/98bacfcd-4615-4bde-a9e5-6d908367260e-kube-api-access-2vbm5\") pod \"aws-ebs-csi-driver-node-q9sb4\" (UID: \"98bacfcd-4615-4bde-a9e5-6d908367260e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9sb4"
Jul 14 13:27:16.193588 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112371    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b33bdbd5-63be-4828-a2d6-91d4bead7b80-dbus\") pod \"global-pull-secret-syncer-s7ft2\" (UID: \"b33bdbd5-63be-4828-a2d6-91d4bead7b80\") " pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:27:16.193588 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112388    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4sgk\" (UniqueName: \"kubernetes.io/projected/bf204a1f-63ca-40ac-825b-e8684c67e36f-kube-api-access-z4sgk\") pod \"node-ca-7l994\" (UID: \"bf204a1f-63ca-40ac-825b-e8684c67e36f\") " pod="openshift-image-registry/node-ca-7l994"
Jul 14 13:27:16.193588 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112404    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-systemd-units\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.193588 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112428    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-run-openvswitch\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.193588 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112465    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-host-cni-bin\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.193588 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112487    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-host-cni-netd\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.193588 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112520    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98bacfcd-4615-4bde-a9e5-6d908367260e-registration-dir\") pod \"aws-ebs-csi-driver-node-q9sb4\" (UID: \"98bacfcd-4615-4bde-a9e5-6d908367260e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9sb4"
Jul 14 13:27:16.193588 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112543    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-host-run-netns\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.193588 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112559    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x67gf\" (UniqueName: \"kubernetes.io/projected/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-kube-api-access-x67gf\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.193588 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112575    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/88a55d43-cfd1-433c-944b-8de4720da8af-cni-binary-copy\") pod \"multus-additional-cni-plugins-dhtvj\" (UID: \"88a55d43-cfd1-433c-944b-8de4720da8af\") " pod="openshift-multus/multus-additional-cni-plugins-dhtvj"
Jul 14 13:27:16.193588 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112594    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-node-log\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.194659 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112602    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-host-run-netns\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.194659 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112621    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/03018596-d0b1-43da-af02-ede52df67ceb-hosts-file\") pod \"node-resolver-mjgwk\" (UID: \"03018596-d0b1-43da-af02-ede52df67ceb\") " pod="openshift-dns/node-resolver-mjgwk"
Jul 14 13:27:16.194659 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112652    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98bacfcd-4615-4bde-a9e5-6d908367260e-registration-dir\") pod \"aws-ebs-csi-driver-node-q9sb4\" (UID: \"98bacfcd-4615-4bde-a9e5-6d908367260e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9sb4"
Jul 14 13:27:16.194659 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112664    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b33bdbd5-63be-4828-a2d6-91d4bead7b80-kubelet-config\") pod \"global-pull-secret-syncer-s7ft2\" (UID: \"b33bdbd5-63be-4828-a2d6-91d4bead7b80\") " pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:27:16.194659 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112706    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b33bdbd5-63be-4828-a2d6-91d4bead7b80-original-pull-secret\") pod \"global-pull-secret-syncer-s7ft2\" (UID: \"b33bdbd5-63be-4828-a2d6-91d4bead7b80\") " pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:27:16.194659 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112738    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/61151274-edf7-4e83-adfd-be582e30ad29-iptables-alerter-script\") pod \"iptables-alerter-8qsng\" (UID: \"61151274-edf7-4e83-adfd-be582e30ad29\") " pod="openshift-network-operator/iptables-alerter-8qsng"
Jul 14 13:27:16.194659 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112756    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csc58\" (UniqueName: \"kubernetes.io/projected/ff284a1c-edb3-4049-a1e2-2c803aceaa62-kube-api-access-csc58\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.194659 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112787    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-multus-daemon-config\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.194659 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112807    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/88a55d43-cfd1-433c-944b-8de4720da8af-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dhtvj\" (UID: \"88a55d43-cfd1-433c-944b-8de4720da8af\") " pod="openshift-multus/multus-additional-cni-plugins-dhtvj"
Jul 14 13:27:16.194659 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112836    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-etc-openvswitch\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.194659 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112853    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e33a489f-7d68-4e6c-9ffe-e7252049e15d-env-overrides\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.194659 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112893    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-host-run-ovn-kubernetes\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.194659 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112911    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ff284a1c-edb3-4049-a1e2-2c803aceaa62-etc-sysconfig\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.194659 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112983    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ff284a1c-edb3-4049-a1e2-2c803aceaa62-run\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.194659 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.112998    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-host-kubelet\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.194659 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.113015    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff284a1c-edb3-4049-a1e2-2c803aceaa62-tmp\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.194659 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.113038    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-host-run-multus-certs\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.195746 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.113053    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/88a55d43-cfd1-433c-944b-8de4720da8af-os-release\") pod \"multus-additional-cni-plugins-dhtvj\" (UID: \"88a55d43-cfd1-433c-944b-8de4720da8af\") " pod="openshift-multus/multus-additional-cni-plugins-dhtvj"
Jul 14 13:27:16.195746 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.113064    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ff284a1c-edb3-4049-a1e2-2c803aceaa62-run\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.195746 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.113068    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-hostroot\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.195746 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.113067    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ff284a1c-edb3-4049-a1e2-2c803aceaa62-etc-sysconfig\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.195746 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.113111    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e33a489f-7d68-4e6c-9ffe-e7252049e15d-ovnkube-config\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.195746 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.113133    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-host-run-multus-certs\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.195746 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.113139    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-hostroot\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.195746 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.113144    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98bacfcd-4615-4bde-a9e5-6d908367260e-socket-dir\") pod \"aws-ebs-csi-driver-node-q9sb4\" (UID: \"98bacfcd-4615-4bde-a9e5-6d908367260e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9sb4"
Jul 14 13:27:16.195746 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.113178    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98bacfcd-4615-4bde-a9e5-6d908367260e-etc-selinux\") pod \"aws-ebs-csi-driver-node-q9sb4\" (UID: \"98bacfcd-4615-4bde-a9e5-6d908367260e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9sb4"
Jul 14 13:27:16.195746 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.113194    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-os-release\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.195746 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.113220    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-multus-socket-dir-parent\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.195746 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.113232    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98bacfcd-4615-4bde-a9e5-6d908367260e-socket-dir\") pod \"aws-ebs-csi-driver-node-q9sb4\" (UID: \"98bacfcd-4615-4bde-a9e5-6d908367260e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9sb4"
Jul 14 13:27:16.195746 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.113243    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn67h\" (UniqueName: \"kubernetes.io/projected/61151274-edf7-4e83-adfd-be582e30ad29-kube-api-access-kn67h\") pod \"iptables-alerter-8qsng\" (UID: \"61151274-edf7-4e83-adfd-be582e30ad29\") " pod="openshift-network-operator/iptables-alerter-8qsng"
Jul 14 13:27:16.195746 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.113272    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-multus-socket-dir-parent\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.195746 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.113272    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88a55d43-cfd1-433c-944b-8de4720da8af-system-cni-dir\") pod \"multus-additional-cni-plugins-dhtvj\" (UID: \"88a55d43-cfd1-433c-944b-8de4720da8af\") " pod="openshift-multus/multus-additional-cni-plugins-dhtvj"
Jul 14 13:27:16.195746 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.113285    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-os-release\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.195746 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.113293    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98bacfcd-4615-4bde-a9e5-6d908367260e-etc-selinux\") pod \"aws-ebs-csi-driver-node-q9sb4\" (UID: \"98bacfcd-4615-4bde-a9e5-6d908367260e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9sb4"
Jul 14 13:27:16.196448 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.113303    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-host-slash\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.196448 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.113769    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-cni-binary-copy\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.196448 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.113771    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-multus-daemon-config\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.196448 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.113675    2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 14 13:27:16.196448 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.127711    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ff284a1c-edb3-4049-a1e2-2c803aceaa62-etc-tuned\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.196448 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.127774    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff284a1c-edb3-4049-a1e2-2c803aceaa62-tmp\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.196448 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.175507    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csc58\" (UniqueName: \"kubernetes.io/projected/ff284a1c-edb3-4049-a1e2-2c803aceaa62-kube-api-access-csc58\") pod \"tuned-xwg7t\" (UID: \"ff284a1c-edb3-4049-a1e2-2c803aceaa62\") " pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.196448 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.177055    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x67gf\" (UniqueName: \"kubernetes.io/projected/8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b-kube-api-access-x67gf\") pod \"multus-mqppd\" (UID: \"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b\") " pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.196448 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.192545    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vbm5\" (UniqueName: \"kubernetes.io/projected/98bacfcd-4615-4bde-a9e5-6d908367260e-kube-api-access-2vbm5\") pod \"aws-ebs-csi-driver-node-q9sb4\" (UID: \"98bacfcd-4615-4bde-a9e5-6d908367260e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9sb4"
Jul 14 13:27:16.359945 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214146    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/88a55d43-cfd1-433c-944b-8de4720da8af-cni-binary-copy\") pod \"multus-additional-cni-plugins-dhtvj\" (UID: \"88a55d43-cfd1-433c-944b-8de4720da8af\") " pod="openshift-multus/multus-additional-cni-plugins-dhtvj"
Jul 14 13:27:16.359945 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214173    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-node-log\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.359945 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214189    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/03018596-d0b1-43da-af02-ede52df67ceb-hosts-file\") pod \"node-resolver-mjgwk\" (UID: \"03018596-d0b1-43da-af02-ede52df67ceb\") " pod="openshift-dns/node-resolver-mjgwk"
Jul 14 13:27:16.359945 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214204    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b33bdbd5-63be-4828-a2d6-91d4bead7b80-kubelet-config\") pod \"global-pull-secret-syncer-s7ft2\" (UID: \"b33bdbd5-63be-4828-a2d6-91d4bead7b80\") " pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:27:16.359945 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214220    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b33bdbd5-63be-4828-a2d6-91d4bead7b80-original-pull-secret\") pod \"global-pull-secret-syncer-s7ft2\" (UID: \"b33bdbd5-63be-4828-a2d6-91d4bead7b80\") " pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:27:16.359945 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214239    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/61151274-edf7-4e83-adfd-be582e30ad29-iptables-alerter-script\") pod \"iptables-alerter-8qsng\" (UID: \"61151274-edf7-4e83-adfd-be582e30ad29\") " pod="openshift-network-operator/iptables-alerter-8qsng"
Jul 14 13:27:16.359945 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214263    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-node-log\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.359945 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214265    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/88a55d43-cfd1-433c-944b-8de4720da8af-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dhtvj\" (UID: \"88a55d43-cfd1-433c-944b-8de4720da8af\") " pod="openshift-multus/multus-additional-cni-plugins-dhtvj"
Jul 14 13:27:16.359945 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214302    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-etc-openvswitch\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.359945 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214304    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b33bdbd5-63be-4828-a2d6-91d4bead7b80-kubelet-config\") pod \"global-pull-secret-syncer-s7ft2\" (UID: \"b33bdbd5-63be-4828-a2d6-91d4bead7b80\") " pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:27:16.359945 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214320    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e33a489f-7d68-4e6c-9ffe-e7252049e15d-env-overrides\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.359945 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214323    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/03018596-d0b1-43da-af02-ede52df67ceb-hosts-file\") pod \"node-resolver-mjgwk\" (UID: \"03018596-d0b1-43da-af02-ede52df67ceb\") " pod="openshift-dns/node-resolver-mjgwk"
Jul 14 13:27:16.359945 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214353    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-host-run-ovn-kubernetes\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.359945 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214367    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-etc-openvswitch\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.359945 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:16.214370    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 13:27:16.359945 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214382    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-host-kubelet\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.359945 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214406    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/88a55d43-cfd1-433c-944b-8de4720da8af-os-release\") pod \"multus-additional-cni-plugins-dhtvj\" (UID: \"88a55d43-cfd1-433c-944b-8de4720da8af\") " pod="openshift-multus/multus-additional-cni-plugins-dhtvj"
Jul 14 13:27:16.359945 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214434    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e33a489f-7d68-4e6c-9ffe-e7252049e15d-ovnkube-config\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.360926 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214441    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-host-kubelet\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.360926 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214405    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-host-run-ovn-kubernetes\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.360926 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:16.214495    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b33bdbd5-63be-4828-a2d6-91d4bead7b80-original-pull-secret podName:b33bdbd5-63be-4828-a2d6-91d4bead7b80 nodeName:}" failed. No retries permitted until 2026-07-14 13:27:16.714457244 +0000 UTC m=+2.040777167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b33bdbd5-63be-4828-a2d6-91d4bead7b80-original-pull-secret") pod "global-pull-secret-syncer-s7ft2" (UID: "b33bdbd5-63be-4828-a2d6-91d4bead7b80") : object "kube-system"/"original-pull-secret" not registered
Jul 14 13:27:16.360926 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214507    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/88a55d43-cfd1-433c-944b-8de4720da8af-os-release\") pod \"multus-additional-cni-plugins-dhtvj\" (UID: \"88a55d43-cfd1-433c-944b-8de4720da8af\") " pod="openshift-multus/multus-additional-cni-plugins-dhtvj"
Jul 14 13:27:16.360926 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214517    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn67h\" (UniqueName: \"kubernetes.io/projected/61151274-edf7-4e83-adfd-be582e30ad29-kube-api-access-kn67h\") pod \"iptables-alerter-8qsng\" (UID: \"61151274-edf7-4e83-adfd-be582e30ad29\") " pod="openshift-network-operator/iptables-alerter-8qsng"
Jul 14 13:27:16.360926 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214553    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88a55d43-cfd1-433c-944b-8de4720da8af-system-cni-dir\") pod \"multus-additional-cni-plugins-dhtvj\" (UID: \"88a55d43-cfd1-433c-944b-8de4720da8af\") " pod="openshift-multus/multus-additional-cni-plugins-dhtvj"
Jul 14 13:27:16.360926 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214580    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-host-slash\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.360926 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214605    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-host-run-netns\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.360926 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214632    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-var-lib-openvswitch\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.360926 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214656    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-run-ovn\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.360926 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214683    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/88a55d43-cfd1-433c-944b-8de4720da8af-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dhtvj\" (UID: \"88a55d43-cfd1-433c-944b-8de4720da8af\") " pod="openshift-multus/multus-additional-cni-plugins-dhtvj"
Jul 14 13:27:16.360926 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214687    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-host-slash\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.360926 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214721    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j929x\" (UniqueName: \"kubernetes.io/projected/88a55d43-cfd1-433c-944b-8de4720da8af-kube-api-access-j929x\") pod \"multus-additional-cni-plugins-dhtvj\" (UID: \"88a55d43-cfd1-433c-944b-8de4720da8af\") " pod="openshift-multus/multus-additional-cni-plugins-dhtvj"
Jul 14 13:27:16.360926 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214731    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88a55d43-cfd1-433c-944b-8de4720da8af-system-cni-dir\") pod \"multus-additional-cni-plugins-dhtvj\" (UID: \"88a55d43-cfd1-433c-944b-8de4720da8af\") " pod="openshift-multus/multus-additional-cni-plugins-dhtvj"
Jul 14 13:27:16.360926 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214748    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-log-socket\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.360926 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214771    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-run-ovn\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.362485 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214773    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e33a489f-7d68-4e6c-9ffe-e7252049e15d-ovnkube-script-lib\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.362485 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214807    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dxbb\" (UniqueName: \"kubernetes.io/projected/e33a489f-7d68-4e6c-9ffe-e7252049e15d-kube-api-access-6dxbb\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.362485 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214831    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/88a55d43-cfd1-433c-944b-8de4720da8af-cni-binary-copy\") pod \"multus-additional-cni-plugins-dhtvj\" (UID: \"88a55d43-cfd1-433c-944b-8de4720da8af\") " pod="openshift-multus/multus-additional-cni-plugins-dhtvj"
Jul 14 13:27:16.362485 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214838    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfqnw\" (UniqueName: \"kubernetes.io/projected/03018596-d0b1-43da-af02-ede52df67ceb-kube-api-access-qfqnw\") pod \"node-resolver-mjgwk\" (UID: \"03018596-d0b1-43da-af02-ede52df67ceb\") " pod="openshift-dns/node-resolver-mjgwk"
Jul 14 13:27:16.362485 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214899    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q27h2\" (UniqueName: \"kubernetes.io/projected/fd9c8cb5-09e3-4116-b407-01ebe6c06999-kube-api-access-q27h2\") pod \"network-metrics-daemon-w4rvb\" (UID: \"fd9c8cb5-09e3-4116-b407-01ebe6c06999\") " pod="openshift-multus/network-metrics-daemon-w4rvb"
Jul 14 13:27:16.362485 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214904    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/88a55d43-cfd1-433c-944b-8de4720da8af-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dhtvj\" (UID: \"88a55d43-cfd1-433c-944b-8de4720da8af\") " pod="openshift-multus/multus-additional-cni-plugins-dhtvj"
Jul 14 13:27:16.362485 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214912    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/88a55d43-cfd1-433c-944b-8de4720da8af-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dhtvj\" (UID: \"88a55d43-cfd1-433c-944b-8de4720da8af\") " pod="openshift-multus/multus-additional-cni-plugins-dhtvj"
Jul 14 13:27:16.362485 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214927    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ff4m\" (UniqueName: \"kubernetes.io/projected/d87134ea-5c27-47f7-a629-883bae028590-kube-api-access-9ff4m\") pod \"network-check-target-42hkp\" (UID: \"d87134ea-5c27-47f7-a629-883bae028590\") " pod="openshift-network-diagnostics/network-check-target-42hkp"
Jul 14 13:27:16.362485 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214938    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/61151274-edf7-4e83-adfd-be582e30ad29-iptables-alerter-script\") pod \"iptables-alerter-8qsng\" (UID: \"61151274-edf7-4e83-adfd-be582e30ad29\") " pod="openshift-network-operator/iptables-alerter-8qsng"
Jul 14 13:27:16.362485 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214957    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/88a55d43-cfd1-433c-944b-8de4720da8af-cnibin\") pod \"multus-additional-cni-plugins-dhtvj\" (UID: \"88a55d43-cfd1-433c-944b-8de4720da8af\") " pod="openshift-multus/multus-additional-cni-plugins-dhtvj"
Jul 14 13:27:16.362485 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214965    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-var-lib-openvswitch\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.362485 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.214962    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e33a489f-7d68-4e6c-9ffe-e7252049e15d-env-overrides\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.362485 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.215011    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/88a55d43-cfd1-433c-944b-8de4720da8af-cnibin\") pod \"multus-additional-cni-plugins-dhtvj\" (UID: \"88a55d43-cfd1-433c-944b-8de4720da8af\") " pod="openshift-multus/multus-additional-cni-plugins-dhtvj"
Jul 14 13:27:16.362485 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.215019    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-host-run-netns\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.362485 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.215052    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-log-socket\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.362485 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.215051    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/88a55d43-cfd1-433c-944b-8de4720da8af-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dhtvj\" (UID: \"88a55d43-cfd1-433c-944b-8de4720da8af\") " pod="openshift-multus/multus-additional-cni-plugins-dhtvj"
Jul 14 13:27:16.362485 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.215079    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e33a489f-7d68-4e6c-9ffe-e7252049e15d-ovnkube-config\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.363071 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.215106    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.363071 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.215141    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/62b825ef-ccb4-4f79-bd95-c0cb2753b4f6-agent-certs\") pod \"konnectivity-agent-kkhvt\" (UID: \"62b825ef-ccb4-4f79-bd95-c0cb2753b4f6\") " pod="kube-system/konnectivity-agent-kkhvt"
Jul 14 13:27:16.363071 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.215160    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/62b825ef-ccb4-4f79-bd95-c0cb2753b4f6-konnectivity-ca\") pod \"konnectivity-agent-kkhvt\" (UID: \"62b825ef-ccb4-4f79-bd95-c0cb2753b4f6\") " pod="kube-system/konnectivity-agent-kkhvt"
Jul 14 13:27:16.363071 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.215181    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61151274-edf7-4e83-adfd-be582e30ad29-host-slash\") pod \"iptables-alerter-8qsng\" (UID: \"61151274-edf7-4e83-adfd-be582e30ad29\") " pod="openshift-network-operator/iptables-alerter-8qsng"
Jul 14 13:27:16.363071 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.215211    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61151274-edf7-4e83-adfd-be582e30ad29-host-slash\") pod \"iptables-alerter-8qsng\" (UID: \"61151274-edf7-4e83-adfd-be582e30ad29\") " pod="openshift-network-operator/iptables-alerter-8qsng"
Jul 14 13:27:16.363071 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.215253    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.363071 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.215291    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bf204a1f-63ca-40ac-825b-e8684c67e36f-serviceca\") pod \"node-ca-7l994\" (UID: \"bf204a1f-63ca-40ac-825b-e8684c67e36f\") " pod="openshift-image-registry/node-ca-7l994"
Jul 14 13:27:16.363071 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.215318    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-run-systemd\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.363071 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.215349    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03018596-d0b1-43da-af02-ede52df67ceb-tmp-dir\") pod \"node-resolver-mjgwk\" (UID: \"03018596-d0b1-43da-af02-ede52df67ceb\") " pod="openshift-dns/node-resolver-mjgwk"
Jul 14 13:27:16.363071 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.215376    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e33a489f-7d68-4e6c-9ffe-e7252049e15d-ovn-node-metrics-cert\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.363071 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.215411    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd9c8cb5-09e3-4116-b407-01ebe6c06999-metrics-certs\") pod \"network-metrics-daemon-w4rvb\" (UID: \"fd9c8cb5-09e3-4116-b407-01ebe6c06999\") " pod="openshift-multus/network-metrics-daemon-w4rvb"
Jul 14 13:27:16.363071 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.215420    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-run-systemd\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.363071 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.215435    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf204a1f-63ca-40ac-825b-e8684c67e36f-host\") pod \"node-ca-7l994\" (UID: \"bf204a1f-63ca-40ac-825b-e8684c67e36f\") " pod="openshift-image-registry/node-ca-7l994"
Jul 14 13:27:16.363071 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.215461    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b33bdbd5-63be-4828-a2d6-91d4bead7b80-dbus\") pod \"global-pull-secret-syncer-s7ft2\" (UID: \"b33bdbd5-63be-4828-a2d6-91d4bead7b80\") " pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:27:16.363071 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.215489    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4sgk\" (UniqueName: \"kubernetes.io/projected/bf204a1f-63ca-40ac-825b-e8684c67e36f-kube-api-access-z4sgk\") pod \"node-ca-7l994\" (UID: \"bf204a1f-63ca-40ac-825b-e8684c67e36f\") " pod="openshift-image-registry/node-ca-7l994"
Jul 14 13:27:16.363071 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:16.215504    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 13:27:16.363071 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.215512    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/88a55d43-cfd1-433c-944b-8de4720da8af-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dhtvj\" (UID: \"88a55d43-cfd1-433c-944b-8de4720da8af\") " pod="openshift-multus/multus-additional-cni-plugins-dhtvj"
Jul 14 13:27:16.363071 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.215514    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-systemd-units\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.363675 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:16.215561    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd9c8cb5-09e3-4116-b407-01ebe6c06999-metrics-certs podName:fd9c8cb5-09e3-4116-b407-01ebe6c06999 nodeName:}" failed. No retries permitted until 2026-07-14 13:27:16.715544797 +0000 UTC m=+2.041864719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd9c8cb5-09e3-4116-b407-01ebe6c06999-metrics-certs") pod "network-metrics-daemon-w4rvb" (UID: "fd9c8cb5-09e3-4116-b407-01ebe6c06999") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 13:27:16.363675 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.215554    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-systemd-units\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.363675 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.215582    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-run-openvswitch\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.363675 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.215593    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b33bdbd5-63be-4828-a2d6-91d4bead7b80-dbus\") pod \"global-pull-secret-syncer-s7ft2\" (UID: \"b33bdbd5-63be-4828-a2d6-91d4bead7b80\") " pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:27:16.363675 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.215613    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-host-cni-bin\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.363675 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.215635    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf204a1f-63ca-40ac-825b-e8684c67e36f-host\") pod \"node-ca-7l994\" (UID: \"bf204a1f-63ca-40ac-825b-e8684c67e36f\") " pod="openshift-image-registry/node-ca-7l994"
Jul 14 13:27:16.363675 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.215641    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-host-cni-netd\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.363675 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.215676    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-run-openvswitch\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.363675 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.215668    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03018596-d0b1-43da-af02-ede52df67ceb-tmp-dir\") pod \"node-resolver-mjgwk\" (UID: \"03018596-d0b1-43da-af02-ede52df67ceb\") " pod="openshift-dns/node-resolver-mjgwk"
Jul 14 13:27:16.363675 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.215709    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-host-cni-netd\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.363675 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.215710    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bf204a1f-63ca-40ac-825b-e8684c67e36f-serviceca\") pod \"node-ca-7l994\" (UID: \"bf204a1f-63ca-40ac-825b-e8684c67e36f\") " pod="openshift-image-registry/node-ca-7l994"
Jul 14 13:27:16.363675 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.215731    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/62b825ef-ccb4-4f79-bd95-c0cb2753b4f6-konnectivity-ca\") pod \"konnectivity-agent-kkhvt\" (UID: \"62b825ef-ccb4-4f79-bd95-c0cb2753b4f6\") " pod="kube-system/konnectivity-agent-kkhvt"
Jul 14 13:27:16.363675 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.215748    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e33a489f-7d68-4e6c-9ffe-e7252049e15d-host-cni-bin\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.363675 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.215990    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e33a489f-7d68-4e6c-9ffe-e7252049e15d-ovnkube-script-lib\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.363675 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.218160    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/62b825ef-ccb4-4f79-bd95-c0cb2753b4f6-agent-certs\") pod \"konnectivity-agent-kkhvt\" (UID: \"62b825ef-ccb4-4f79-bd95-c0cb2753b4f6\") " pod="kube-system/konnectivity-agent-kkhvt"
Jul 14 13:27:16.363675 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.224995    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e33a489f-7d68-4e6c-9ffe-e7252049e15d-ovn-node-metrics-cert\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.363675 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.247381    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q27h2\" (UniqueName: \"kubernetes.io/projected/fd9c8cb5-09e3-4116-b407-01ebe6c06999-kube-api-access-q27h2\") pod \"network-metrics-daemon-w4rvb\" (UID: \"fd9c8cb5-09e3-4116-b407-01ebe6c06999\") " pod="openshift-multus/network-metrics-daemon-w4rvb"
Jul 14 13:27:16.364306 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.247577    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j929x\" (UniqueName: \"kubernetes.io/projected/88a55d43-cfd1-433c-944b-8de4720da8af-kube-api-access-j929x\") pod \"multus-additional-cni-plugins-dhtvj\" (UID: \"88a55d43-cfd1-433c-944b-8de4720da8af\") " pod="openshift-multus/multus-additional-cni-plugins-dhtvj"
Jul 14 13:27:16.364306 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.260621    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4sgk\" (UniqueName: \"kubernetes.io/projected/bf204a1f-63ca-40ac-825b-e8684c67e36f-kube-api-access-z4sgk\") pod \"node-ca-7l994\" (UID: \"bf204a1f-63ca-40ac-825b-e8684c67e36f\") " pod="openshift-image-registry/node-ca-7l994"
Jul 14 13:27:16.364306 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.260628    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dxbb\" (UniqueName: \"kubernetes.io/projected/e33a489f-7d68-4e6c-9ffe-e7252049e15d-kube-api-access-6dxbb\") pod \"ovnkube-node-vq8tk\" (UID: \"e33a489f-7d68-4e6c-9ffe-e7252049e15d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.364306 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.261131    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn67h\" (UniqueName: \"kubernetes.io/projected/61151274-edf7-4e83-adfd-be582e30ad29-kube-api-access-kn67h\") pod \"iptables-alerter-8qsng\" (UID: \"61151274-edf7-4e83-adfd-be582e30ad29\") " pod="openshift-network-operator/iptables-alerter-8qsng"
Jul 14 13:27:16.364306 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.264913    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfqnw\" (UniqueName: \"kubernetes.io/projected/03018596-d0b1-43da-af02-ede52df67ceb-kube-api-access-qfqnw\") pod \"node-resolver-mjgwk\" (UID: \"03018596-d0b1-43da-af02-ede52df67ceb\") " pod="openshift-dns/node-resolver-mjgwk"
Jul 14 13:27:16.364306 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:16.281932    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 13:27:16.364306 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:16.281947    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 13:27:16.364306 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:16.281957    2569 projected.go:196] Error preparing data for projected volume kube-api-access-9ff4m for pod openshift-network-diagnostics/network-check-target-42hkp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 13:27:16.364306 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:16.282010    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d87134ea-5c27-47f7-a629-883bae028590-kube-api-access-9ff4m podName:d87134ea-5c27-47f7-a629-883bae028590 nodeName:}" failed. No retries permitted until 2026-07-14 13:27:16.781996293 +0000 UTC m=+2.108316202 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9ff4m" (UniqueName: "kubernetes.io/projected/d87134ea-5c27-47f7-a629-883bae028590-kube-api-access-9ff4m") pod "network-check-target-42hkp" (UID: "d87134ea-5c27-47f7-a629-883bae028590") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 13:27:16.364306 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.314121    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9sb4"
Jul 14 13:27:16.364306 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.332632    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xwg7t"
Jul 14 13:27:16.364306 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.356295    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mqppd"
Jul 14 13:27:16.653581 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.377506    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8qsng"
Jul 14 13:27:16.653581 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.395441    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mjgwk"
Jul 14 13:27:16.653581 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.418930    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7l994"
Jul 14 13:27:16.653581 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.447002    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dhtvj"
Jul 14 13:27:16.653581 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.467713    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:16.653581 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.483080    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kkhvt"
Jul 14 13:27:16.901994 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.718827    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd9c8cb5-09e3-4116-b407-01ebe6c06999-metrics-certs\") pod \"network-metrics-daemon-w4rvb\" (UID: \"fd9c8cb5-09e3-4116-b407-01ebe6c06999\") " pod="openshift-multus/network-metrics-daemon-w4rvb"
Jul 14 13:27:16.901994 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.718858    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b33bdbd5-63be-4828-a2d6-91d4bead7b80-original-pull-secret\") pod \"global-pull-secret-syncer-s7ft2\" (UID: \"b33bdbd5-63be-4828-a2d6-91d4bead7b80\") " pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:27:16.901994 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:16.718946    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 13:27:16.901994 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:16.718959    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 13:27:16.901994 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:16.718993    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b33bdbd5-63be-4828-a2d6-91d4bead7b80-original-pull-secret podName:b33bdbd5-63be-4828-a2d6-91d4bead7b80 nodeName:}" failed. No retries permitted until 2026-07-14 13:27:17.718981594 +0000 UTC m=+3.045301503 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b33bdbd5-63be-4828-a2d6-91d4bead7b80-original-pull-secret") pod "global-pull-secret-syncer-s7ft2" (UID: "b33bdbd5-63be-4828-a2d6-91d4bead7b80") : object "kube-system"/"original-pull-secret" not registered
Jul 14 13:27:16.901994 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:16.719008    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd9c8cb5-09e3-4116-b407-01ebe6c06999-metrics-certs podName:fd9c8cb5-09e3-4116-b407-01ebe6c06999 nodeName:}" failed. No retries permitted until 2026-07-14 13:27:17.719000108 +0000 UTC m=+3.045320016 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd9c8cb5-09e3-4116-b407-01ebe6c06999-metrics-certs") pod "network-metrics-daemon-w4rvb" (UID: "fd9c8cb5-09e3-4116-b407-01ebe6c06999") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 13:27:16.901994 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.819471    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ff4m\" (UniqueName: \"kubernetes.io/projected/d87134ea-5c27-47f7-a629-883bae028590-kube-api-access-9ff4m\") pod \"network-check-target-42hkp\" (UID: \"d87134ea-5c27-47f7-a629-883bae028590\") " pod="openshift-network-diagnostics/network-check-target-42hkp"
Jul 14 13:27:16.901994 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:16.819595    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 13:27:16.901994 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:16.819610    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 13:27:16.901994 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:16.819620    2569 projected.go:196] Error preparing data for projected volume kube-api-access-9ff4m for pod openshift-network-diagnostics/network-check-target-42hkp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 13:27:16.901994 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:16.819676    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d87134ea-5c27-47f7-a629-883bae028590-kube-api-access-9ff4m podName:d87134ea-5c27-47f7-a629-883bae028590 nodeName:}" failed. No retries permitted until 2026-07-14 13:27:17.819656185 +0000 UTC m=+3.145976100 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9ff4m" (UniqueName: "kubernetes.io/projected/d87134ea-5c27-47f7-a629-883bae028590-kube-api-access-9ff4m") pod "network-check-target-42hkp" (UID: "d87134ea-5c27-47f7-a629-883bae028590") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 13:27:16.901994 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.843300    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 13:27:17.110752 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:16.909447    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dc64344bcf6e13eee2ce8f9a1eefa2b.slice/crio-a436ee37468b727c52da42e59ce37262424273bcbe13e0399a7d4e1efc7b59e6 WatchSource:0}: Error finding container a436ee37468b727c52da42e59ce37262424273bcbe13e0399a7d4e1efc7b59e6: Status 404 returned error can't find the container with id a436ee37468b727c52da42e59ce37262424273bcbe13e0399a7d4e1efc7b59e6
Jul 14 13:27:17.110752 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:16.913767    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff284a1c_edb3_4049_a1e2_2c803aceaa62.slice/crio-7db1580371e641bc3b3fb101c34f2ad120961ffb62e07666d84871d792e08ab6 WatchSource:0}: Error finding container 7db1580371e641bc3b3fb101c34f2ad120961ffb62e07666d84871d792e08ab6: Status 404 returned error can't find the container with id 7db1580371e641bc3b3fb101c34f2ad120961ffb62e07666d84871d792e08ab6
Jul 14 13:27:17.110752 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:16.915959    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 13:27:17.110752 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:16.919381    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode33a489f_7d68_4e6c_9ffe_e7252049e15d.slice/crio-f88d70da1931c0a32cc1f1b771d7253c56f262dafdedbb08f18ec228ced4a3b1 WatchSource:0}: Error finding container f88d70da1931c0a32cc1f1b771d7253c56f262dafdedbb08f18ec228ced4a3b1: Status 404 returned error can't find the container with id f88d70da1931c0a32cc1f1b771d7253c56f262dafdedbb08f18ec228ced4a3b1
Jul 14 13:27:17.110752 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:16.939507    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61151274_edf7_4e83_adfd_be582e30ad29.slice/crio-1841e10c40ae5cf89507a277f572eaf9e75144bc088f11f3dd20577402937f73 WatchSource:0}: Error finding container 1841e10c40ae5cf89507a277f572eaf9e75144bc088f11f3dd20577402937f73: Status 404 returned error can't find the container with id 1841e10c40ae5cf89507a277f572eaf9e75144bc088f11f3dd20577402937f73
Jul 14 13:27:17.110752 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:16.971371    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a55d43_cfd1_433c_944b_8de4720da8af.slice/crio-2f3c11aa5852d28cb68722887e84359f65705efde7f9c773d80dc6735dd93e74 WatchSource:0}: Error finding container 2f3c11aa5852d28cb68722887e84359f65705efde7f9c773d80dc6735dd93e74: Status 404 returned error can't find the container with id 2f3c11aa5852d28cb68722887e84359f65705efde7f9c773d80dc6735dd93e74
Jul 14 13:27:17.110752 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:17.015214    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf204a1f_63ca_40ac_825b_e8684c67e36f.slice/crio-22930f924cc5606a9cc8f69e311d12cbd18e0807b12dc7fd8fc5d2bcffea6dbf WatchSource:0}: Error finding container 22930f924cc5606a9cc8f69e311d12cbd18e0807b12dc7fd8fc5d2bcffea6dbf: Status 404 returned error can't find the container with id 22930f924cc5606a9cc8f69e311d12cbd18e0807b12dc7fd8fc5d2bcffea6dbf
Jul 14 13:27:17.110752 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:17.039282    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98bacfcd_4615_4bde_a9e5_6d908367260e.slice/crio-ab0f09211a9566d61c74e735c4a316b28a54cf70ba768bb7c0af6462d1ebaad7 WatchSource:0}: Error finding container ab0f09211a9566d61c74e735c4a316b28a54cf70ba768bb7c0af6462d1ebaad7: Status 404 returned error can't find the container with id ab0f09211a9566d61c74e735c4a316b28a54cf70ba768bb7c0af6462d1ebaad7
Jul 14 13:27:17.110752 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:17.066930    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c3ae302_40b4_42eb_9a67_5fd1cfcbd33b.slice/crio-9725233ce46833a0808fa18f2881349b7c6685aa930f6628d71e789430488346 WatchSource:0}: Error finding container 9725233ce46833a0808fa18f2881349b7c6685aa930f6628d71e789430488346: Status 404 returned error can't find the container with id 9725233ce46833a0808fa18f2881349b7c6685aa930f6628d71e789430488346
Jul 14 13:27:17.110752 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:17.077674    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 13:22:16 +0000 UTC" deadline="2028-03-06 16:07:21.929054622 +0000 UTC"
Jul 14 13:27:17.110752 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:17.077707    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14426h40m4.851349987s"
Jul 14 13:27:17.289750 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:17.129901    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62b825ef_ccb4_4f79_bd95_c0cb2753b4f6.slice/crio-d48b900dc7f0aecbaf4bb8016bc4389c7a32663b00179dbe8f8a83a0af35e4d8 WatchSource:0}: Error finding container d48b900dc7f0aecbaf4bb8016bc4389c7a32663b00179dbe8f8a83a0af35e4d8: Status 404 returned error can't find the container with id d48b900dc7f0aecbaf4bb8016bc4389c7a32663b00179dbe8f8a83a0af35e4d8
Jul 14 13:27:17.289750 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:17.187828    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:27:17.289750 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:17.187918    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7ft2" podUID="b33bdbd5-63be-4828-a2d6-91d4bead7b80"
Jul 14 13:27:17.289750 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:17.190378    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mqppd" event={"ID":"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b","Type":"ContainerStarted","Data":"9725233ce46833a0808fa18f2881349b7c6685aa930f6628d71e789430488346"}
Jul 14 13:27:17.289750 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:17.191288    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9sb4" event={"ID":"98bacfcd-4615-4bde-a9e5-6d908367260e","Type":"ContainerStarted","Data":"ab0f09211a9566d61c74e735c4a316b28a54cf70ba768bb7c0af6462d1ebaad7"}
Jul 14 13:27:17.289750 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:17.192285    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7l994" event={"ID":"bf204a1f-63ca-40ac-825b-e8684c67e36f","Type":"ContainerStarted","Data":"22930f924cc5606a9cc8f69e311d12cbd18e0807b12dc7fd8fc5d2bcffea6dbf"}
Jul 14 13:27:17.289750 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:17.193234    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dhtvj" event={"ID":"88a55d43-cfd1-433c-944b-8de4720da8af","Type":"ContainerStarted","Data":"2f3c11aa5852d28cb68722887e84359f65705efde7f9c773d80dc6735dd93e74"}
Jul 14 13:27:17.289750 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:17.194244    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xwg7t" event={"ID":"ff284a1c-edb3-4049-a1e2-2c803aceaa62","Type":"ContainerStarted","Data":"7db1580371e641bc3b3fb101c34f2ad120961ffb62e07666d84871d792e08ab6"}
Jul 14 13:27:17.289750 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:17.195255    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kkhvt" event={"ID":"62b825ef-ccb4-4f79-bd95-c0cb2753b4f6","Type":"ContainerStarted","Data":"d48b900dc7f0aecbaf4bb8016bc4389c7a32663b00179dbe8f8a83a0af35e4d8"}
Jul 14 13:27:17.289750 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:17.196233    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8qsng" event={"ID":"61151274-edf7-4e83-adfd-be582e30ad29","Type":"ContainerStarted","Data":"1841e10c40ae5cf89507a277f572eaf9e75144bc088f11f3dd20577402937f73"}
Jul 14 13:27:17.289750 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:17.197359    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk" event={"ID":"e33a489f-7d68-4e6c-9ffe-e7252049e15d","Type":"ContainerStarted","Data":"f88d70da1931c0a32cc1f1b771d7253c56f262dafdedbb08f18ec228ced4a3b1"}
Jul 14 13:27:17.289750 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:17.198343    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-78.ec2.internal" event={"ID":"9dc64344bcf6e13eee2ce8f9a1eefa2b","Type":"ContainerStarted","Data":"a436ee37468b727c52da42e59ce37262424273bcbe13e0399a7d4e1efc7b59e6"}
Jul 14 13:27:17.514654 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:17.392764    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod742561d943373579cd7f313233325224.slice/crio-6f75eab85d6400f70cddd1b4aea59b65b1c7d80006ccb30cdcbe4c13420b66d2 WatchSource:0}: Error finding container 6f75eab85d6400f70cddd1b4aea59b65b1c7d80006ccb30cdcbe4c13420b66d2: Status 404 returned error can't find the container with id 6f75eab85d6400f70cddd1b4aea59b65b1c7d80006ccb30cdcbe4c13420b66d2
Jul 14 13:27:17.768783 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:17.727709    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd9c8cb5-09e3-4116-b407-01ebe6c06999-metrics-certs\") pod \"network-metrics-daemon-w4rvb\" (UID: \"fd9c8cb5-09e3-4116-b407-01ebe6c06999\") " pod="openshift-multus/network-metrics-daemon-w4rvb"
Jul 14 13:27:17.768783 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:17.727765    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b33bdbd5-63be-4828-a2d6-91d4bead7b80-original-pull-secret\") pod \"global-pull-secret-syncer-s7ft2\" (UID: \"b33bdbd5-63be-4828-a2d6-91d4bead7b80\") " pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:27:17.768783 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:17.727878    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 13:27:17.768783 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:17.727921    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 13:27:17.768783 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:17.727957    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd9c8cb5-09e3-4116-b407-01ebe6c06999-metrics-certs podName:fd9c8cb5-09e3-4116-b407-01ebe6c06999 nodeName:}" failed. No retries permitted until 2026-07-14 13:27:19.727933817 +0000 UTC m=+5.054253731 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd9c8cb5-09e3-4116-b407-01ebe6c06999-metrics-certs") pod "network-metrics-daemon-w4rvb" (UID: "fd9c8cb5-09e3-4116-b407-01ebe6c06999") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 13:27:17.768783 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:17.727979    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b33bdbd5-63be-4828-a2d6-91d4bead7b80-original-pull-secret podName:b33bdbd5-63be-4828-a2d6-91d4bead7b80 nodeName:}" failed. No retries permitted until 2026-07-14 13:27:19.727967722 +0000 UTC m=+5.054287635 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b33bdbd5-63be-4828-a2d6-91d4bead7b80-original-pull-secret") pod "global-pull-secret-syncer-s7ft2" (UID: "b33bdbd5-63be-4828-a2d6-91d4bead7b80") : object "kube-system"/"original-pull-secret" not registered
Jul 14 13:27:17.768783 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:17.745787    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03018596_d0b1_43da_af02_ede52df67ceb.slice/crio-69b1a37a97f564403c9f1f0ff84e4604d7674d100bd9804843f6d7e14c2334d4 WatchSource:0}: Error finding container 69b1a37a97f564403c9f1f0ff84e4604d7674d100bd9804843f6d7e14c2334d4: Status 404 returned error can't find the container with id 69b1a37a97f564403c9f1f0ff84e4604d7674d100bd9804843f6d7e14c2334d4
Jul 14 13:27:17.849982 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:17.828679    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ff4m\" (UniqueName: \"kubernetes.io/projected/d87134ea-5c27-47f7-a629-883bae028590-kube-api-access-9ff4m\") pod \"network-check-target-42hkp\" (UID: \"d87134ea-5c27-47f7-a629-883bae028590\") " pod="openshift-network-diagnostics/network-check-target-42hkp"
Jul 14 13:27:17.849982 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:17.828849    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 13:27:17.849982 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:17.828867    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 13:27:17.849982 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:17.828878    2569 projected.go:196] Error preparing data for projected volume kube-api-access-9ff4m for pod openshift-network-diagnostics/network-check-target-42hkp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 13:27:17.849982 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:17.828927    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d87134ea-5c27-47f7-a629-883bae028590-kube-api-access-9ff4m podName:d87134ea-5c27-47f7-a629-883bae028590 nodeName:}" failed. No retries permitted until 2026-07-14 13:27:19.828911197 +0000 UTC m=+5.155231104 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9ff4m" (UniqueName: "kubernetes.io/projected/d87134ea-5c27-47f7-a629-883bae028590-kube-api-access-9ff4m") pod "network-check-target-42hkp" (UID: "d87134ea-5c27-47f7-a629-883bae028590") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 13:27:18.080806 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:18.078358    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 13:22:16 +0000 UTC" deadline="2028-03-29 05:23:05.14259943 +0000 UTC"
Jul 14 13:27:18.080806 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:18.078395    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14967h55m47.064209778s"
Jul 14 13:27:18.186814 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:18.186775    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42hkp"
Jul 14 13:27:18.187394 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:18.186919    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42hkp" podUID="d87134ea-5c27-47f7-a629-883bae028590"
Jul 14 13:27:18.187394 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:18.187361    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4rvb"
Jul 14 13:27:18.187531 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:18.187474    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4rvb" podUID="fd9c8cb5-09e3-4116-b407-01ebe6c06999"
Jul 14 13:27:18.206765 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:18.206695    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mjgwk" event={"ID":"03018596-d0b1-43da-af02-ede52df67ceb","Type":"ContainerStarted","Data":"69b1a37a97f564403c9f1f0ff84e4604d7674d100bd9804843f6d7e14c2334d4"}
Jul 14 13:27:18.221931 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:18.221898    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-78.ec2.internal" event={"ID":"742561d943373579cd7f313233325224","Type":"ContainerStarted","Data":"6f75eab85d6400f70cddd1b4aea59b65b1c7d80006ccb30cdcbe4c13420b66d2"}
Jul 14 13:27:18.685892 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:18.685856    2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 13:27:19.188314 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:19.188278    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:27:19.188796 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:19.188428    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7ft2" podUID="b33bdbd5-63be-4828-a2d6-91d4bead7b80"
Jul 14 13:27:19.751124 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:19.751051    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd9c8cb5-09e3-4116-b407-01ebe6c06999-metrics-certs\") pod \"network-metrics-daemon-w4rvb\" (UID: \"fd9c8cb5-09e3-4116-b407-01ebe6c06999\") " pod="openshift-multus/network-metrics-daemon-w4rvb"
Jul 14 13:27:19.751124 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:19.751120    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b33bdbd5-63be-4828-a2d6-91d4bead7b80-original-pull-secret\") pod \"global-pull-secret-syncer-s7ft2\" (UID: \"b33bdbd5-63be-4828-a2d6-91d4bead7b80\") " pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:27:19.751377 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:19.751249    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 13:27:19.751377 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:19.751314    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b33bdbd5-63be-4828-a2d6-91d4bead7b80-original-pull-secret podName:b33bdbd5-63be-4828-a2d6-91d4bead7b80 nodeName:}" failed. No retries permitted until 2026-07-14 13:27:23.751295057 +0000 UTC m=+9.077614968 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b33bdbd5-63be-4828-a2d6-91d4bead7b80-original-pull-secret") pod "global-pull-secret-syncer-s7ft2" (UID: "b33bdbd5-63be-4828-a2d6-91d4bead7b80") : object "kube-system"/"original-pull-secret" not registered
Jul 14 13:27:19.751774 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:19.751749    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 13:27:19.751863 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:19.751816    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd9c8cb5-09e3-4116-b407-01ebe6c06999-metrics-certs podName:fd9c8cb5-09e3-4116-b407-01ebe6c06999 nodeName:}" failed. No retries permitted until 2026-07-14 13:27:23.751798073 +0000 UTC m=+9.078117984 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd9c8cb5-09e3-4116-b407-01ebe6c06999-metrics-certs") pod "network-metrics-daemon-w4rvb" (UID: "fd9c8cb5-09e3-4116-b407-01ebe6c06999") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 13:27:19.852121 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:19.851927    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ff4m\" (UniqueName: \"kubernetes.io/projected/d87134ea-5c27-47f7-a629-883bae028590-kube-api-access-9ff4m\") pod \"network-check-target-42hkp\" (UID: \"d87134ea-5c27-47f7-a629-883bae028590\") " pod="openshift-network-diagnostics/network-check-target-42hkp"
Jul 14 13:27:19.852361 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:19.852157    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 13:27:19.852361 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:19.852188    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 13:27:19.852361 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:19.852202    2569 projected.go:196] Error preparing data for projected volume kube-api-access-9ff4m for pod openshift-network-diagnostics/network-check-target-42hkp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 13:27:19.852361 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:19.852266    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d87134ea-5c27-47f7-a629-883bae028590-kube-api-access-9ff4m podName:d87134ea-5c27-47f7-a629-883bae028590 nodeName:}" failed. No retries permitted until 2026-07-14 13:27:23.852244478 +0000 UTC m=+9.178564408 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9ff4m" (UniqueName: "kubernetes.io/projected/d87134ea-5c27-47f7-a629-883bae028590-kube-api-access-9ff4m") pod "network-check-target-42hkp" (UID: "d87134ea-5c27-47f7-a629-883bae028590") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 13:27:20.186826 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:20.186737    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42hkp"
Jul 14 13:27:20.187011 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:20.186908    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42hkp" podUID="d87134ea-5c27-47f7-a629-883bae028590"
Jul 14 13:27:20.187511 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:20.187348    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4rvb"
Jul 14 13:27:20.187511 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:20.187471    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4rvb" podUID="fd9c8cb5-09e3-4116-b407-01ebe6c06999"
Jul 14 13:27:21.187795 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:21.187723    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:27:21.188184 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:21.187856    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7ft2" podUID="b33bdbd5-63be-4828-a2d6-91d4bead7b80"
Jul 14 13:27:22.188087 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:22.187387    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42hkp"
Jul 14 13:27:22.188087 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:22.187505    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42hkp" podUID="d87134ea-5c27-47f7-a629-883bae028590"
Jul 14 13:27:22.188087 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:22.187918    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4rvb"
Jul 14 13:27:22.188087 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:22.188042    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4rvb" podUID="fd9c8cb5-09e3-4116-b407-01ebe6c06999"
Jul 14 13:27:23.187070 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:23.187032    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:27:23.187229 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:23.187174    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7ft2" podUID="b33bdbd5-63be-4828-a2d6-91d4bead7b80"
Jul 14 13:27:23.786343 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:23.786294    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd9c8cb5-09e3-4116-b407-01ebe6c06999-metrics-certs\") pod \"network-metrics-daemon-w4rvb\" (UID: \"fd9c8cb5-09e3-4116-b407-01ebe6c06999\") " pod="openshift-multus/network-metrics-daemon-w4rvb"
Jul 14 13:27:23.786767 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:23.786353    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b33bdbd5-63be-4828-a2d6-91d4bead7b80-original-pull-secret\") pod \"global-pull-secret-syncer-s7ft2\" (UID: \"b33bdbd5-63be-4828-a2d6-91d4bead7b80\") " pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:27:23.786767 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:23.786488    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 13:27:23.786767 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:23.786550    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b33bdbd5-63be-4828-a2d6-91d4bead7b80-original-pull-secret podName:b33bdbd5-63be-4828-a2d6-91d4bead7b80 nodeName:}" failed. No retries permitted until 2026-07-14 13:27:31.786531847 +0000 UTC m=+17.112851757 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b33bdbd5-63be-4828-a2d6-91d4bead7b80-original-pull-secret") pod "global-pull-secret-syncer-s7ft2" (UID: "b33bdbd5-63be-4828-a2d6-91d4bead7b80") : object "kube-system"/"original-pull-secret" not registered
Jul 14 13:27:23.786985 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:23.786965    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 13:27:23.787079 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:23.787044    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd9c8cb5-09e3-4116-b407-01ebe6c06999-metrics-certs podName:fd9c8cb5-09e3-4116-b407-01ebe6c06999 nodeName:}" failed. No retries permitted until 2026-07-14 13:27:31.787026263 +0000 UTC m=+17.113346178 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd9c8cb5-09e3-4116-b407-01ebe6c06999-metrics-certs") pod "network-metrics-daemon-w4rvb" (UID: "fd9c8cb5-09e3-4116-b407-01ebe6c06999") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 13:27:23.887429 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:23.886815    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ff4m\" (UniqueName: \"kubernetes.io/projected/d87134ea-5c27-47f7-a629-883bae028590-kube-api-access-9ff4m\") pod \"network-check-target-42hkp\" (UID: \"d87134ea-5c27-47f7-a629-883bae028590\") " pod="openshift-network-diagnostics/network-check-target-42hkp"
Jul 14 13:27:23.887429 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:23.887029    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 13:27:23.887429 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:23.887052    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 13:27:23.887429 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:23.887066    2569 projected.go:196] Error preparing data for projected volume kube-api-access-9ff4m for pod openshift-network-diagnostics/network-check-target-42hkp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 13:27:23.887429 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:23.887121    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d87134ea-5c27-47f7-a629-883bae028590-kube-api-access-9ff4m podName:d87134ea-5c27-47f7-a629-883bae028590 nodeName:}" failed. No retries permitted until 2026-07-14 13:27:31.887102923 +0000 UTC m=+17.213422843 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9ff4m" (UniqueName: "kubernetes.io/projected/d87134ea-5c27-47f7-a629-883bae028590-kube-api-access-9ff4m") pod "network-check-target-42hkp" (UID: "d87134ea-5c27-47f7-a629-883bae028590") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 13:27:24.187738 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:24.187249    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42hkp"
Jul 14 13:27:24.187738 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:24.187377    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42hkp" podUID="d87134ea-5c27-47f7-a629-883bae028590"
Jul 14 13:27:24.187738 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:24.187536    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4rvb"
Jul 14 13:27:24.187738 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:24.187644    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4rvb" podUID="fd9c8cb5-09e3-4116-b407-01ebe6c06999"
Jul 14 13:27:25.188279 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:25.187800    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:27:25.188279 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:25.187939    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7ft2" podUID="b33bdbd5-63be-4828-a2d6-91d4bead7b80"
Jul 14 13:27:26.187371 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:26.186864    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42hkp"
Jul 14 13:27:26.187371 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:26.186864    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4rvb"
Jul 14 13:27:26.187371 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:26.186997    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42hkp" podUID="d87134ea-5c27-47f7-a629-883bae028590"
Jul 14 13:27:26.187371 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:26.187110    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4rvb" podUID="fd9c8cb5-09e3-4116-b407-01ebe6c06999"
Jul 14 13:27:27.187705 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:27.187675    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:27:27.188172 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:27.187812    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7ft2" podUID="b33bdbd5-63be-4828-a2d6-91d4bead7b80"
Jul 14 13:27:28.187103 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:28.187068    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42hkp"
Jul 14 13:27:28.187294 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:28.187170    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42hkp" podUID="d87134ea-5c27-47f7-a629-883bae028590"
Jul 14 13:27:28.187294 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:28.187227    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4rvb"
Jul 14 13:27:28.187406 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:28.187341    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4rvb" podUID="fd9c8cb5-09e3-4116-b407-01ebe6c06999"
Jul 14 13:27:29.187748 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:29.187726    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:27:29.188047 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:29.187822    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7ft2" podUID="b33bdbd5-63be-4828-a2d6-91d4bead7b80"
Jul 14 13:27:30.187096 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:30.186868    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42hkp"
Jul 14 13:27:30.187282 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:30.186877    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4rvb"
Jul 14 13:27:30.187282 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:30.187184    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42hkp" podUID="d87134ea-5c27-47f7-a629-883bae028590"
Jul 14 13:27:30.187401 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:30.187308    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4rvb" podUID="fd9c8cb5-09e3-4116-b407-01ebe6c06999"
Jul 14 13:27:31.187699 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:31.187668    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:27:31.188179 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:31.187798    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7ft2" podUID="b33bdbd5-63be-4828-a2d6-91d4bead7b80"
Jul 14 13:27:31.848141 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:31.848108    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd9c8cb5-09e3-4116-b407-01ebe6c06999-metrics-certs\") pod \"network-metrics-daemon-w4rvb\" (UID: \"fd9c8cb5-09e3-4116-b407-01ebe6c06999\") " pod="openshift-multus/network-metrics-daemon-w4rvb"
Jul 14 13:27:31.848311 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:31.848159    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b33bdbd5-63be-4828-a2d6-91d4bead7b80-original-pull-secret\") pod \"global-pull-secret-syncer-s7ft2\" (UID: \"b33bdbd5-63be-4828-a2d6-91d4bead7b80\") " pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:27:31.848311 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:31.848275    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 13:27:31.848311 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:31.848280    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 13:27:31.848474 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:31.848331    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b33bdbd5-63be-4828-a2d6-91d4bead7b80-original-pull-secret podName:b33bdbd5-63be-4828-a2d6-91d4bead7b80 nodeName:}" failed. No retries permitted until 2026-07-14 13:27:47.848316009 +0000 UTC m=+33.174635932 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b33bdbd5-63be-4828-a2d6-91d4bead7b80-original-pull-secret") pod "global-pull-secret-syncer-s7ft2" (UID: "b33bdbd5-63be-4828-a2d6-91d4bead7b80") : object "kube-system"/"original-pull-secret" not registered
Jul 14 13:27:31.848474 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:31.848351    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd9c8cb5-09e3-4116-b407-01ebe6c06999-metrics-certs podName:fd9c8cb5-09e3-4116-b407-01ebe6c06999 nodeName:}" failed. No retries permitted until 2026-07-14 13:27:47.848341124 +0000 UTC m=+33.174661030 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd9c8cb5-09e3-4116-b407-01ebe6c06999-metrics-certs") pod "network-metrics-daemon-w4rvb" (UID: "fd9c8cb5-09e3-4116-b407-01ebe6c06999") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 13:27:31.949217 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:31.949186    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ff4m\" (UniqueName: \"kubernetes.io/projected/d87134ea-5c27-47f7-a629-883bae028590-kube-api-access-9ff4m\") pod \"network-check-target-42hkp\" (UID: \"d87134ea-5c27-47f7-a629-883bae028590\") " pod="openshift-network-diagnostics/network-check-target-42hkp"
Jul 14 13:27:31.949385 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:31.949340    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 13:27:31.949385 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:31.949365    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 13:27:31.949385 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:31.949377    2569 projected.go:196] Error preparing data for projected volume kube-api-access-9ff4m for pod openshift-network-diagnostics/network-check-target-42hkp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 13:27:31.949510 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:31.949427    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d87134ea-5c27-47f7-a629-883bae028590-kube-api-access-9ff4m podName:d87134ea-5c27-47f7-a629-883bae028590 nodeName:}" failed. No retries permitted until 2026-07-14 13:27:47.949410303 +0000 UTC m=+33.275730209 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9ff4m" (UniqueName: "kubernetes.io/projected/d87134ea-5c27-47f7-a629-883bae028590-kube-api-access-9ff4m") pod "network-check-target-42hkp" (UID: "d87134ea-5c27-47f7-a629-883bae028590") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 13:27:32.187317 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:32.187233    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42hkp"
Jul 14 13:27:32.187317 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:32.187256    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4rvb"
Jul 14 13:27:32.187526 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:32.187359    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42hkp" podUID="d87134ea-5c27-47f7-a629-883bae028590"
Jul 14 13:27:32.187526 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:32.187508    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4rvb" podUID="fd9c8cb5-09e3-4116-b407-01ebe6c06999"
Jul 14 13:27:33.187209 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:33.187183    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:27:33.187567 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:33.187321    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7ft2" podUID="b33bdbd5-63be-4828-a2d6-91d4bead7b80"
Jul 14 13:27:34.186969 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:34.186938    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42hkp"
Jul 14 13:27:34.187149 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:34.187073    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42hkp" podUID="d87134ea-5c27-47f7-a629-883bae028590"
Jul 14 13:27:34.187149 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:34.187107    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4rvb"
Jul 14 13:27:34.187254 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:34.187237    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4rvb" podUID="fd9c8cb5-09e3-4116-b407-01ebe6c06999"
Jul 14 13:27:35.187777 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:35.187747    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:27:35.188198 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:35.187839    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7ft2" podUID="b33bdbd5-63be-4828-a2d6-91d4bead7b80"
Jul 14 13:27:36.187725 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:36.187696    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42hkp"
Jul 14 13:27:36.187900 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:36.187707    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4rvb"
Jul 14 13:27:36.187900 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:36.187830    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42hkp" podUID="d87134ea-5c27-47f7-a629-883bae028590"
Jul 14 13:27:36.188313 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:36.187914    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4rvb" podUID="fd9c8cb5-09e3-4116-b407-01ebe6c06999"
Jul 14 13:27:37.187485 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:37.187450    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:27:37.187651 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:37.187589    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7ft2" podUID="b33bdbd5-63be-4828-a2d6-91d4bead7b80"
Jul 14 13:27:38.187664 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:38.187640    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42hkp"
Jul 14 13:27:38.188057 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:38.187744    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42hkp" podUID="d87134ea-5c27-47f7-a629-883bae028590"
Jul 14 13:27:38.188057 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:38.187804    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4rvb"
Jul 14 13:27:38.188057 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:38.187922    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4rvb" podUID="fd9c8cb5-09e3-4116-b407-01ebe6c06999"
Jul 14 13:27:38.279470 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:38.279438    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xwg7t" event={"ID":"ff284a1c-edb3-4049-a1e2-2c803aceaa62","Type":"ContainerStarted","Data":"9172ed6afb56d4b7c8cf2e9067942578d495e308699919d1cb06a92b253cc205"}
Jul 14 13:27:38.280983 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:38.280961    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-78.ec2.internal" event={"ID":"9dc64344bcf6e13eee2ce8f9a1eefa2b","Type":"ContainerStarted","Data":"8acf42c36af7b6d07c8f068f361ec49282ac3b6b0f9422c37f2ee5a27281d88b"}
Jul 14 13:27:38.282516 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:38.282496    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mqppd" event={"ID":"8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b","Type":"ContainerStarted","Data":"20841ee4fde28b5e49de33c10f426f2c55525166c2551bb48c4f3d39281c49c4"}
Jul 14 13:27:38.308025 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:38.307959    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xwg7t" podStartSLOduration=2.7511456819999998 podStartE2EDuration="23.307941184s" totalImagesPullingTime="20.556795502s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 13:27:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:27:38.306534434 +0000 UTC m=+23.632854363" watchObservedRunningTime="2026-07-14 13:27:38.307941184 +0000 UTC m=+23.634261135"
Jul 14 13:27:38.325315 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:38.325279    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-78.ec2.internal" podStartSLOduration=22.32526707 podStartE2EDuration="22.32526707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 13:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:27:38.324738031 +0000 UTC m=+23.651057959" watchObservedRunningTime="2026-07-14 13:27:38.32526707 +0000 UTC m=+23.651586999"
Jul 14 13:27:38.346733 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:38.346692    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-mqppd" podStartSLOduration=2.908740221 podStartE2EDuration="23.346675288s" totalImagesPullingTime="20.437935067s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 13:27:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:27:38.346536672 +0000 UTC m=+23.672856600" watchObservedRunningTime="2026-07-14 13:27:38.346675288 +0000 UTC m=+23.672995216"
Jul 14 13:27:39.187879 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:39.187536    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:27:39.188506 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:39.187962    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7ft2" podUID="b33bdbd5-63be-4828-a2d6-91d4bead7b80"
Jul 14 13:27:39.286199 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:39.286161    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kkhvt" event={"ID":"62b825ef-ccb4-4f79-bd95-c0cb2753b4f6","Type":"ContainerStarted","Data":"3fda463911f056913041261c4dbbee08c2a067c10dc968354747d8bf9fe1253f"}
Jul 14 13:27:39.288123 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:39.288089    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8qsng" event={"ID":"61151274-edf7-4e83-adfd-be582e30ad29","Type":"ContainerStarted","Data":"4117fb8777333587ab27fae31b17370bf6bb26774912823843a454566d88bc47"}
Jul 14 13:27:39.291545 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:39.291522    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vq8tk_e33a489f-7d68-4e6c-9ffe-e7252049e15d/ovn-acl-logging/0.log"
Jul 14 13:27:39.291903 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:39.291879    2569 generic.go:356] "Generic (PLEG): container finished" podID="e33a489f-7d68-4e6c-9ffe-e7252049e15d" containerID="cdb552943f5591ba5e9fb3fbd69d628f6f06424aa16b77f86296444c59cf5f9e" exitCode=1
Jul 14 13:27:39.292030 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:39.291937    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk" event={"ID":"e33a489f-7d68-4e6c-9ffe-e7252049e15d","Type":"ContainerStarted","Data":"f58889db5ce2f42c858e30d916e889ce445ad41ab3bac5915dd0a2e89174eaa4"}
Jul 14 13:27:39.292103 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:39.292030    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk" event={"ID":"e33a489f-7d68-4e6c-9ffe-e7252049e15d","Type":"ContainerStarted","Data":"adff7fe6885c07cced061cf5acd1e313a5781500def3cef1363f3e387e49e324"}
Jul 14 13:27:39.292103 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:39.292074    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk" event={"ID":"e33a489f-7d68-4e6c-9ffe-e7252049e15d","Type":"ContainerStarted","Data":"a113b95473140d6319663454874813088ce48e50d1b7ce93fac8e6d8989dde2a"}
Jul 14 13:27:39.292203 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:39.292108    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk" event={"ID":"e33a489f-7d68-4e6c-9ffe-e7252049e15d","Type":"ContainerStarted","Data":"fa0d66fd86cd6d4f14f70c5b934f9967198cdf737c3df462bdb232eeb2b61032"}
Jul 14 13:27:39.292203 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:39.292136    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk" event={"ID":"e33a489f-7d68-4e6c-9ffe-e7252049e15d","Type":"ContainerDied","Data":"cdb552943f5591ba5e9fb3fbd69d628f6f06424aa16b77f86296444c59cf5f9e"}
Jul 14 13:27:39.292203 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:39.292176    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk" event={"ID":"e33a489f-7d68-4e6c-9ffe-e7252049e15d","Type":"ContainerStarted","Data":"5f2442ed632b040530c50c5d82d215059b9b547227863f689a872c272b8ce112"}
Jul 14 13:27:39.293473 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:39.293448    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mjgwk" event={"ID":"03018596-d0b1-43da-af02-ede52df67ceb","Type":"ContainerStarted","Data":"39f8630ed063ac801bd722461071c7737d37964549ccbc8aafa4ffb91fa96b99"}
Jul 14 13:27:39.295005 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:39.294978    2569 generic.go:356] "Generic (PLEG): container finished" podID="742561d943373579cd7f313233325224" containerID="dd5afe9c5e0da591fd08ab4d9e8cc7aeffa3c78a8b729d8b414c03dda38ebc18" exitCode=0
Jul 14 13:27:39.295112 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:39.295040    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-78.ec2.internal" event={"ID":"742561d943373579cd7f313233325224","Type":"ContainerDied","Data":"dd5afe9c5e0da591fd08ab4d9e8cc7aeffa3c78a8b729d8b414c03dda38ebc18"}
Jul 14 13:27:39.296737 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:39.296710    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9sb4" event={"ID":"98bacfcd-4615-4bde-a9e5-6d908367260e","Type":"ContainerStarted","Data":"a4a34ee3d992410abefaca584e170605ce541cefd37619c3a362a7d175ddfc3b"}
Jul 14 13:27:39.298275 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:39.298254    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7l994" event={"ID":"bf204a1f-63ca-40ac-825b-e8684c67e36f","Type":"ContainerStarted","Data":"e2639b73fb455eb8199794497fac4f14d57acded517e7b5aad51b0f8c4fa4688"}
Jul 14 13:27:39.299988 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:39.299964    2569 generic.go:356] "Generic (PLEG): container finished" podID="88a55d43-cfd1-433c-944b-8de4720da8af" containerID="56447de051b5be1508a36351888b6ec5eabc82f0b8bbb4d538084fbe50470ec0" exitCode=0
Jul 14 13:27:39.300096 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:39.300069    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dhtvj" event={"ID":"88a55d43-cfd1-433c-944b-8de4720da8af","Type":"ContainerDied","Data":"56447de051b5be1508a36351888b6ec5eabc82f0b8bbb4d538084fbe50470ec0"}
Jul 14 13:27:39.319586 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:39.319546    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kkhvt" podStartSLOduration=3.9797089530000003 podStartE2EDuration="24.319535258s" totalImagesPullingTime="20.339826305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:27:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:27:39.319022601 +0000 UTC m=+24.645342527" watchObservedRunningTime="2026-07-14 13:27:39.319535258 +0000 UTC m=+24.645855185"
Jul 14 13:27:39.340440 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:39.340396    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7l994" podStartSLOduration=3.885997907 podStartE2EDuration="24.340380743s" totalImagesPullingTime="20.454382836s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 13:27:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:27:39.340338415 +0000 UTC m=+24.666658345" watchObservedRunningTime="2026-07-14 13:27:39.340380743 +0000 UTC m=+24.666700672"
Jul 14 13:27:39.482380 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:39.482298    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8qsng" podStartSLOduration=3.950241177 podStartE2EDuration="24.482278372s" totalImagesPullingTime="20.532037195s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 13:27:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:27:39.444539365 +0000 UTC m=+24.770859292" watchObservedRunningTime="2026-07-14 13:27:39.482278372 +0000 UTC m=+24.808598300"
Jul 14 13:27:39.896124 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:39.896105    2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 14 13:27:40.066292 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:40.066171    2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-14T13:27:39.896122803Z","UUID":"67e4fe66-905f-4a13-b2a3-8eb5c37d3d54","Handler":null,"Name":"","Endpoint":""}
Jul 14 13:27:40.067570 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:40.067553    2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 14 13:27:40.067671 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:40.067585    2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 14 13:27:40.187633 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:40.187605    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42hkp"
Jul 14 13:27:40.187797 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:40.187696    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42hkp" podUID="d87134ea-5c27-47f7-a629-883bae028590"
Jul 14 13:27:40.187797 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:40.187748    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4rvb"
Jul 14 13:27:40.187869 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:40.187838    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4rvb" podUID="fd9c8cb5-09e3-4116-b407-01ebe6c06999"
Jul 14 13:27:40.304155 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:40.304121    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-78.ec2.internal" event={"ID":"742561d943373579cd7f313233325224","Type":"ContainerStarted","Data":"333f5e224534cf6a44c420e979bc791423f473dfef0d3d2a66377a6942a1f2c1"}
Jul 14 13:27:40.306328 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:40.306277    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9sb4" event={"ID":"98bacfcd-4615-4bde-a9e5-6d908367260e","Type":"ContainerStarted","Data":"736fd97ed63e61350dc5e8bd692878b8759ba49f6de2fffd72a49e931f670b00"}
Jul 14 13:27:40.322472 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:40.322398    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-78.ec2.internal" podStartSLOduration=24.322382973 podStartE2EDuration="24.322382973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 13:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:27:40.322003445 +0000 UTC m=+25.648323373" watchObservedRunningTime="2026-07-14 13:27:40.322382973 +0000 UTC m=+25.648702902"
Jul 14 13:27:40.322584 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:40.322479    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mjgwk" podStartSLOduration=5.611737935 podStartE2EDuration="25.322473354s" totalImagesPullingTime="19.710735419s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 13:27:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:27:39.483116245 +0000 UTC m=+24.809436174" watchObservedRunningTime="2026-07-14 13:27:40.322473354 +0000 UTC m=+25.648793282"
Jul 14 13:27:41.187643 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:41.187416    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:27:41.187808 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:41.187748    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7ft2" podUID="b33bdbd5-63be-4828-a2d6-91d4bead7b80"
Jul 14 13:27:41.311611 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:41.311537    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vq8tk_e33a489f-7d68-4e6c-9ffe-e7252049e15d/ovn-acl-logging/0.log"
Jul 14 13:27:41.312072 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:41.311994    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk" event={"ID":"e33a489f-7d68-4e6c-9ffe-e7252049e15d","Type":"ContainerStarted","Data":"cc8822c469aaf751a07a0ec119812b9224f27ec0c122688ab1bfdcf8b3717a27"}
Jul 14 13:27:42.187739 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:42.187557    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42hkp"
Jul 14 13:27:42.187739 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:42.187662    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42hkp" podUID="d87134ea-5c27-47f7-a629-883bae028590"
Jul 14 13:27:42.187918 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:42.187749    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4rvb"
Jul 14 13:27:42.187918 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:42.187891    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4rvb" podUID="fd9c8cb5-09e3-4116-b407-01ebe6c06999"
Jul 14 13:27:42.316637 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:42.316600    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9sb4" event={"ID":"98bacfcd-4615-4bde-a9e5-6d908367260e","Type":"ContainerStarted","Data":"9f317a80285a936d5e02816ecd1c619d9252c71715316b1f1672c21815e60f34"}
Jul 14 13:27:42.342801 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:42.342756    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9sb4" podStartSLOduration=3.389950987 podStartE2EDuration="27.342740104s" totalImagesPullingTime="23.952789117s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 13:27:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:27:42.341355264 +0000 UTC m=+27.667675193" watchObservedRunningTime="2026-07-14 13:27:42.342740104 +0000 UTC m=+27.669060033"
Jul 14 13:27:42.354476 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:42.354452    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kkhvt"
Jul 14 13:27:43.187526 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:43.187501    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:27:43.187668 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:43.187623    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7ft2" podUID="b33bdbd5-63be-4828-a2d6-91d4bead7b80"
Jul 14 13:27:43.499378 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:43.499341    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kkhvt"
Jul 14 13:27:43.500049 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:43.500026    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kkhvt"
Jul 14 13:27:44.187650 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:44.187473    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4rvb"
Jul 14 13:27:44.187812 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:44.187495    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42hkp"
Jul 14 13:27:44.187878 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:44.187811    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42hkp" podUID="d87134ea-5c27-47f7-a629-883bae028590"
Jul 14 13:27:44.187878 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:44.187719    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4rvb" podUID="fd9c8cb5-09e3-4116-b407-01ebe6c06999"
Jul 14 13:27:44.324124 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:44.324077    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vq8tk_e33a489f-7d68-4e6c-9ffe-e7252049e15d/ovn-acl-logging/0.log"
Jul 14 13:27:44.324450 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:44.324428    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk" event={"ID":"e33a489f-7d68-4e6c-9ffe-e7252049e15d","Type":"ContainerStarted","Data":"659238ec99541848493ae89766e0196eeda74251e266886ef1b81b545262267f"}
Jul 14 13:27:44.324681 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:44.324660    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:44.324802 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:44.324732    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:44.324862 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:44.324835    2569 scope.go:117] "RemoveContainer" containerID="cdb552943f5591ba5e9fb3fbd69d628f6f06424aa16b77f86296444c59cf5f9e"
Jul 14 13:27:44.326516 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:44.326496    2569 generic.go:356] "Generic (PLEG): container finished" podID="88a55d43-cfd1-433c-944b-8de4720da8af" containerID="c6fff9d049bdd7d41246ea45e3afe7d0a86dd5f721feb717db37c4622aa2107f" exitCode=0
Jul 14 13:27:44.326625 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:44.326572    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dhtvj" event={"ID":"88a55d43-cfd1-433c-944b-8de4720da8af","Type":"ContainerDied","Data":"c6fff9d049bdd7d41246ea45e3afe7d0a86dd5f721feb717db37c4622aa2107f"}
Jul 14 13:27:44.327613 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:44.327256    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kkhvt"
Jul 14 13:27:44.340256 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:44.340238    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:44.340913 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:44.340898    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:45.039691 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:45.039650    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:27:45.188621 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:45.188594    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:27:45.188774 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:45.188703    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7ft2" podUID="b33bdbd5-63be-4828-a2d6-91d4bead7b80"
Jul 14 13:27:45.332432 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:45.332364    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vq8tk_e33a489f-7d68-4e6c-9ffe-e7252049e15d/ovn-acl-logging/0.log"
Jul 14 13:27:45.332760 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:45.332736    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk" event={"ID":"e33a489f-7d68-4e6c-9ffe-e7252049e15d","Type":"ContainerStarted","Data":"c94001e617dde50321a14be5332405c165a22f94954b4bf2e8902e3aed9c1ef6"}
Jul 14 13:27:45.374330 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:45.374284    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk" podStartSLOduration=8.991024807 podStartE2EDuration="30.374270587s" totalImagesPullingTime="21.38324578s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 13:27:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:27:45.374075974 +0000 UTC m=+30.700395902" watchObservedRunningTime="2026-07-14 13:27:45.374270587 +0000 UTC m=+30.700590515"
Jul 14 13:27:46.135327 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:46.135136    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-s7ft2"]
Jul 14 13:27:46.135629 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:46.135369    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:27:46.135629 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:46.135454    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7ft2" podUID="b33bdbd5-63be-4828-a2d6-91d4bead7b80"
Jul 14 13:27:46.137822 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:46.137796    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-42hkp"]
Jul 14 13:27:46.137932 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:46.137890    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42hkp"
Jul 14 13:27:46.138002 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:46.137985    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42hkp" podUID="d87134ea-5c27-47f7-a629-883bae028590"
Jul 14 13:27:46.138385 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:46.138366    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-w4rvb"]
Jul 14 13:27:46.138470 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:46.138458    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4rvb"
Jul 14 13:27:46.138553 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:46.138535    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4rvb" podUID="fd9c8cb5-09e3-4116-b407-01ebe6c06999"
Jul 14 13:27:46.336544 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:46.336515    2569 generic.go:356] "Generic (PLEG): container finished" podID="88a55d43-cfd1-433c-944b-8de4720da8af" containerID="3f553b9b1baae1efef5d0fa42480f95510baf7d9e4c8ec80b8154097f28441c9" exitCode=0
Jul 14 13:27:46.336684 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:46.336596    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dhtvj" event={"ID":"88a55d43-cfd1-433c-944b-8de4720da8af","Type":"ContainerDied","Data":"3f553b9b1baae1efef5d0fa42480f95510baf7d9e4c8ec80b8154097f28441c9"}
Jul 14 13:27:47.871211 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:47.871178    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b33bdbd5-63be-4828-a2d6-91d4bead7b80-original-pull-secret\") pod \"global-pull-secret-syncer-s7ft2\" (UID: \"b33bdbd5-63be-4828-a2d6-91d4bead7b80\") " pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:27:47.871712 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:47.871261    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd9c8cb5-09e3-4116-b407-01ebe6c06999-metrics-certs\") pod \"network-metrics-daemon-w4rvb\" (UID: \"fd9c8cb5-09e3-4116-b407-01ebe6c06999\") " pod="openshift-multus/network-metrics-daemon-w4rvb"
Jul 14 13:27:47.871712 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:47.871278    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 13:27:47.871712 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:47.871337    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b33bdbd5-63be-4828-a2d6-91d4bead7b80-original-pull-secret podName:b33bdbd5-63be-4828-a2d6-91d4bead7b80 nodeName:}" failed. No retries permitted until 2026-07-14 13:28:19.871317653 +0000 UTC m=+65.197637560 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b33bdbd5-63be-4828-a2d6-91d4bead7b80-original-pull-secret") pod "global-pull-secret-syncer-s7ft2" (UID: "b33bdbd5-63be-4828-a2d6-91d4bead7b80") : object "kube-system"/"original-pull-secret" not registered
Jul 14 13:27:47.871712 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:47.871360    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 13:27:47.871712 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:47.871407    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd9c8cb5-09e3-4116-b407-01ebe6c06999-metrics-certs podName:fd9c8cb5-09e3-4116-b407-01ebe6c06999 nodeName:}" failed. No retries permitted until 2026-07-14 13:28:19.871391105 +0000 UTC m=+65.197711014 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd9c8cb5-09e3-4116-b407-01ebe6c06999-metrics-certs") pod "network-metrics-daemon-w4rvb" (UID: "fd9c8cb5-09e3-4116-b407-01ebe6c06999") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 13:27:47.972273 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:47.972247    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ff4m\" (UniqueName: \"kubernetes.io/projected/d87134ea-5c27-47f7-a629-883bae028590-kube-api-access-9ff4m\") pod \"network-check-target-42hkp\" (UID: \"d87134ea-5c27-47f7-a629-883bae028590\") " pod="openshift-network-diagnostics/network-check-target-42hkp"
Jul 14 13:27:47.972402 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:47.972386    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 13:27:47.972498 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:47.972406    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 13:27:47.972498 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:47.972416    2569 projected.go:196] Error preparing data for projected volume kube-api-access-9ff4m for pod openshift-network-diagnostics/network-check-target-42hkp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 13:27:47.972498 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:47.972461    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d87134ea-5c27-47f7-a629-883bae028590-kube-api-access-9ff4m podName:d87134ea-5c27-47f7-a629-883bae028590 nodeName:}" failed. No retries permitted until 2026-07-14 13:28:19.972443438 +0000 UTC m=+65.298763346 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9ff4m" (UniqueName: "kubernetes.io/projected/d87134ea-5c27-47f7-a629-883bae028590-kube-api-access-9ff4m") pod "network-check-target-42hkp" (UID: "d87134ea-5c27-47f7-a629-883bae028590") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 13:27:48.187351 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:48.187288    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4rvb"
Jul 14 13:27:48.187351 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:48.187305    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42hkp"
Jul 14 13:27:48.187351 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:48.187324    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:27:48.187524 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:48.187396    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4rvb" podUID="fd9c8cb5-09e3-4116-b407-01ebe6c06999"
Jul 14 13:27:48.187524 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:48.187441    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7ft2" podUID="b33bdbd5-63be-4828-a2d6-91d4bead7b80"
Jul 14 13:27:48.187524 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:48.187500    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42hkp" podUID="d87134ea-5c27-47f7-a629-883bae028590"
Jul 14 13:27:48.343947 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:48.343924    2569 generic.go:356] "Generic (PLEG): container finished" podID="88a55d43-cfd1-433c-944b-8de4720da8af" containerID="4c81430944d70b727e8ec2fd22c0bb504e8cd9c2b571e217dcb10b00ca4cae43" exitCode=0
Jul 14 13:27:48.344070 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:48.343959    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dhtvj" event={"ID":"88a55d43-cfd1-433c-944b-8de4720da8af","Type":"ContainerDied","Data":"4c81430944d70b727e8ec2fd22c0bb504e8cd9c2b571e217dcb10b00ca4cae43"}
Jul 14 13:27:50.187446 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.187399    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:27:50.187921 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.187493    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42hkp"
Jul 14 13:27:50.187921 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.187605    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4rvb"
Jul 14 13:27:50.187921 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:50.187607    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7ft2" podUID="b33bdbd5-63be-4828-a2d6-91d4bead7b80"
Jul 14 13:27:50.187921 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:50.187728    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-42hkp" podUID="d87134ea-5c27-47f7-a629-883bae028590"
Jul 14 13:27:50.187921 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:50.187896    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w4rvb" podUID="fd9c8cb5-09e3-4116-b407-01ebe6c06999"
Jul 14 13:27:50.508707 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.508679    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-78.ec2.internal" event="NodeReady"
Jul 14 13:27:50.508876 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.508823    2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 14 13:27:50.564778 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.564753    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lkjjk"]
Jul 14 13:27:50.566676 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.566658    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lkjjk"
Jul 14 13:27:50.568654 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.568636    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vvmd5"]
Jul 14 13:27:50.570797 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.570774    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vvmd5"
Jul 14 13:27:50.571056 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.570956    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 14 13:27:50.571056 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.570979    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6rczr\""
Jul 14 13:27:50.571307 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.571292    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 14 13:27:50.573694 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.573658    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-755dc\""
Jul 14 13:27:50.574033 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.573799    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 14 13:27:50.574033 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.573826    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 14 13:27:50.574033 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.573945    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 14 13:27:50.580591 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.580246    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lkjjk"]
Jul 14 13:27:50.582100 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.582082    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vvmd5"]
Jul 14 13:27:50.694172 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.694141    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5mqq\" (UniqueName: \"kubernetes.io/projected/5cdbc484-7145-45f0-9060-a4a818174ff5-kube-api-access-b5mqq\") pod \"ingress-canary-vvmd5\" (UID: \"5cdbc484-7145-45f0-9060-a4a818174ff5\") " pod="openshift-ingress-canary/ingress-canary-vvmd5"
Jul 14 13:27:50.694318 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.694180    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5cdbc484-7145-45f0-9060-a4a818174ff5-cert\") pod \"ingress-canary-vvmd5\" (UID: \"5cdbc484-7145-45f0-9060-a4a818174ff5\") " pod="openshift-ingress-canary/ingress-canary-vvmd5"
Jul 14 13:27:50.694318 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.694209    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b89a211f-60b9-4eab-84cb-7aad1dcd94b1-metrics-tls\") pod \"dns-default-lkjjk\" (UID: \"b89a211f-60b9-4eab-84cb-7aad1dcd94b1\") " pod="openshift-dns/dns-default-lkjjk"
Jul 14 13:27:50.694318 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.694296    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b89a211f-60b9-4eab-84cb-7aad1dcd94b1-config-volume\") pod \"dns-default-lkjjk\" (UID: \"b89a211f-60b9-4eab-84cb-7aad1dcd94b1\") " pod="openshift-dns/dns-default-lkjjk"
Jul 14 13:27:50.694473 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.694346    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b89a211f-60b9-4eab-84cb-7aad1dcd94b1-tmp-dir\") pod \"dns-default-lkjjk\" (UID: \"b89a211f-60b9-4eab-84cb-7aad1dcd94b1\") " pod="openshift-dns/dns-default-lkjjk"
Jul 14 13:27:50.694473 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.694377    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgz97\" (UniqueName: \"kubernetes.io/projected/b89a211f-60b9-4eab-84cb-7aad1dcd94b1-kube-api-access-rgz97\") pod \"dns-default-lkjjk\" (UID: \"b89a211f-60b9-4eab-84cb-7aad1dcd94b1\") " pod="openshift-dns/dns-default-lkjjk"
Jul 14 13:27:50.795474 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.795407    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b89a211f-60b9-4eab-84cb-7aad1dcd94b1-tmp-dir\") pod \"dns-default-lkjjk\" (UID: \"b89a211f-60b9-4eab-84cb-7aad1dcd94b1\") " pod="openshift-dns/dns-default-lkjjk"
Jul 14 13:27:50.795474 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.795442    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgz97\" (UniqueName: \"kubernetes.io/projected/b89a211f-60b9-4eab-84cb-7aad1dcd94b1-kube-api-access-rgz97\") pod \"dns-default-lkjjk\" (UID: \"b89a211f-60b9-4eab-84cb-7aad1dcd94b1\") " pod="openshift-dns/dns-default-lkjjk"
Jul 14 13:27:50.795677 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.795513    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5mqq\" (UniqueName: \"kubernetes.io/projected/5cdbc484-7145-45f0-9060-a4a818174ff5-kube-api-access-b5mqq\") pod \"ingress-canary-vvmd5\" (UID: \"5cdbc484-7145-45f0-9060-a4a818174ff5\") " pod="openshift-ingress-canary/ingress-canary-vvmd5"
Jul 14 13:27:50.795677 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.795544    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5cdbc484-7145-45f0-9060-a4a818174ff5-cert\") pod \"ingress-canary-vvmd5\" (UID: \"5cdbc484-7145-45f0-9060-a4a818174ff5\") " pod="openshift-ingress-canary/ingress-canary-vvmd5"
Jul 14 13:27:50.795677 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.795626    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b89a211f-60b9-4eab-84cb-7aad1dcd94b1-metrics-tls\") pod \"dns-default-lkjjk\" (UID: \"b89a211f-60b9-4eab-84cb-7aad1dcd94b1\") " pod="openshift-dns/dns-default-lkjjk"
Jul 14 13:27:50.795677 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.795658    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b89a211f-60b9-4eab-84cb-7aad1dcd94b1-config-volume\") pod \"dns-default-lkjjk\" (UID: \"b89a211f-60b9-4eab-84cb-7aad1dcd94b1\") " pod="openshift-dns/dns-default-lkjjk"
Jul 14 13:27:50.795860 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:50.795763    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 13:27:50.795860 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.795787    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b89a211f-60b9-4eab-84cb-7aad1dcd94b1-tmp-dir\") pod \"dns-default-lkjjk\" (UID: \"b89a211f-60b9-4eab-84cb-7aad1dcd94b1\") " pod="openshift-dns/dns-default-lkjjk"
Jul 14 13:27:50.795860 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:50.795849    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b89a211f-60b9-4eab-84cb-7aad1dcd94b1-metrics-tls podName:b89a211f-60b9-4eab-84cb-7aad1dcd94b1 nodeName:}" failed. No retries permitted until 2026-07-14 13:27:51.295825906 +0000 UTC m=+36.622145819 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b89a211f-60b9-4eab-84cb-7aad1dcd94b1-metrics-tls") pod "dns-default-lkjjk" (UID: "b89a211f-60b9-4eab-84cb-7aad1dcd94b1") : secret "dns-default-metrics-tls" not found
Jul 14 13:27:50.796239 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.796217    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b89a211f-60b9-4eab-84cb-7aad1dcd94b1-config-volume\") pod \"dns-default-lkjjk\" (UID: \"b89a211f-60b9-4eab-84cb-7aad1dcd94b1\") " pod="openshift-dns/dns-default-lkjjk"
Jul 14 13:27:50.799911 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.799894    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5cdbc484-7145-45f0-9060-a4a818174ff5-cert\") pod \"ingress-canary-vvmd5\" (UID: \"5cdbc484-7145-45f0-9060-a4a818174ff5\") " pod="openshift-ingress-canary/ingress-canary-vvmd5"
Jul 14 13:27:50.806230 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.806212    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5mqq\" (UniqueName: \"kubernetes.io/projected/5cdbc484-7145-45f0-9060-a4a818174ff5-kube-api-access-b5mqq\") pod \"ingress-canary-vvmd5\" (UID: \"5cdbc484-7145-45f0-9060-a4a818174ff5\") " pod="openshift-ingress-canary/ingress-canary-vvmd5"
Jul 14 13:27:50.806390 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.806371    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgz97\" (UniqueName: \"kubernetes.io/projected/b89a211f-60b9-4eab-84cb-7aad1dcd94b1-kube-api-access-rgz97\") pod \"dns-default-lkjjk\" (UID: \"b89a211f-60b9-4eab-84cb-7aad1dcd94b1\") " pod="openshift-dns/dns-default-lkjjk"
Jul 14 13:27:50.886490 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:50.886463    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vvmd5"
Jul 14 13:27:51.300391 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:51.300359    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b89a211f-60b9-4eab-84cb-7aad1dcd94b1-metrics-tls\") pod \"dns-default-lkjjk\" (UID: \"b89a211f-60b9-4eab-84cb-7aad1dcd94b1\") " pod="openshift-dns/dns-default-lkjjk"
Jul 14 13:27:51.300837 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:51.300509    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 13:27:51.300837 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:51.300603    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b89a211f-60b9-4eab-84cb-7aad1dcd94b1-metrics-tls podName:b89a211f-60b9-4eab-84cb-7aad1dcd94b1 nodeName:}" failed. No retries permitted until 2026-07-14 13:27:52.300580387 +0000 UTC m=+37.626900299 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b89a211f-60b9-4eab-84cb-7aad1dcd94b1-metrics-tls") pod "dns-default-lkjjk" (UID: "b89a211f-60b9-4eab-84cb-7aad1dcd94b1") : secret "dns-default-metrics-tls" not found
Jul 14 13:27:52.187549 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:52.187515    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42hkp"
Jul 14 13:27:52.187733 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:52.187515    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:27:52.187832 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:52.187515    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4rvb"
Jul 14 13:27:52.192672 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:52.192460    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 14 13:27:52.192672 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:52.192471    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l7sfg\""
Jul 14 13:27:52.192672 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:52.192514    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 14 13:27:52.192672 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:52.192519    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 14 13:27:52.192672 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:52.192571    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 14 13:27:52.192959 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:52.192852    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rgn8m\""
Jul 14 13:27:52.307859 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:52.307832    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b89a211f-60b9-4eab-84cb-7aad1dcd94b1-metrics-tls\") pod \"dns-default-lkjjk\" (UID: \"b89a211f-60b9-4eab-84cb-7aad1dcd94b1\") " pod="openshift-dns/dns-default-lkjjk"
Jul 14 13:27:52.308267 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:52.308025    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 13:27:52.308267 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:52.308095    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b89a211f-60b9-4eab-84cb-7aad1dcd94b1-metrics-tls podName:b89a211f-60b9-4eab-84cb-7aad1dcd94b1 nodeName:}" failed. No retries permitted until 2026-07-14 13:27:54.308072976 +0000 UTC m=+39.634392905 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b89a211f-60b9-4eab-84cb-7aad1dcd94b1-metrics-tls") pod "dns-default-lkjjk" (UID: "b89a211f-60b9-4eab-84cb-7aad1dcd94b1") : secret "dns-default-metrics-tls" not found
Jul 14 13:27:52.357492 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:52.357463    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dhtvj" event={"ID":"88a55d43-cfd1-433c-944b-8de4720da8af","Type":"ContainerStarted","Data":"0c32f37958cc8ad7bdc89b86cd30c23c45e58eeac8f59c09f052f5804d40d5b9"}
Jul 14 13:27:52.381671 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:52.381645    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vvmd5"]
Jul 14 13:27:52.385292 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:27:52.385265    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cdbc484_7145_45f0_9060_a4a818174ff5.slice/crio-4442d509405e36b7ecaf59fe20b7ec372b88bf6f7c8bed2c5471b4f859807846 WatchSource:0}: Error finding container 4442d509405e36b7ecaf59fe20b7ec372b88bf6f7c8bed2c5471b4f859807846: Status 404 returned error can't find the container with id 4442d509405e36b7ecaf59fe20b7ec372b88bf6f7c8bed2c5471b4f859807846
Jul 14 13:27:53.360745 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:53.360710    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vvmd5" event={"ID":"5cdbc484-7145-45f0-9060-a4a818174ff5","Type":"ContainerStarted","Data":"4442d509405e36b7ecaf59fe20b7ec372b88bf6f7c8bed2c5471b4f859807846"}
Jul 14 13:27:53.363528 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:53.363499    2569 generic.go:356] "Generic (PLEG): container finished" podID="88a55d43-cfd1-433c-944b-8de4720da8af" containerID="0c32f37958cc8ad7bdc89b86cd30c23c45e58eeac8f59c09f052f5804d40d5b9" exitCode=0
Jul 14 13:27:53.363654 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:53.363532    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dhtvj" event={"ID":"88a55d43-cfd1-433c-944b-8de4720da8af","Type":"ContainerDied","Data":"0c32f37958cc8ad7bdc89b86cd30c23c45e58eeac8f59c09f052f5804d40d5b9"}
Jul 14 13:27:54.323235 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:54.322957    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b89a211f-60b9-4eab-84cb-7aad1dcd94b1-metrics-tls\") pod \"dns-default-lkjjk\" (UID: \"b89a211f-60b9-4eab-84cb-7aad1dcd94b1\") " pod="openshift-dns/dns-default-lkjjk"
Jul 14 13:27:54.323433 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:54.323119    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 13:27:54.323433 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:54.323328    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b89a211f-60b9-4eab-84cb-7aad1dcd94b1-metrics-tls podName:b89a211f-60b9-4eab-84cb-7aad1dcd94b1 nodeName:}" failed. No retries permitted until 2026-07-14 13:27:58.323303768 +0000 UTC m=+43.649623678 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b89a211f-60b9-4eab-84cb-7aad1dcd94b1-metrics-tls") pod "dns-default-lkjjk" (UID: "b89a211f-60b9-4eab-84cb-7aad1dcd94b1") : secret "dns-default-metrics-tls" not found
Jul 14 13:27:54.368311 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:54.368281    2569 generic.go:356] "Generic (PLEG): container finished" podID="88a55d43-cfd1-433c-944b-8de4720da8af" containerID="c45437119c10bcdac997fb757161a1a04567e80d079a5dafa73ec5be1f7ff8e7" exitCode=0
Jul 14 13:27:54.368734 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:54.368347    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dhtvj" event={"ID":"88a55d43-cfd1-433c-944b-8de4720da8af","Type":"ContainerDied","Data":"c45437119c10bcdac997fb757161a1a04567e80d079a5dafa73ec5be1f7ff8e7"}
Jul 14 13:27:55.373553 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:55.373502    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dhtvj" event={"ID":"88a55d43-cfd1-433c-944b-8de4720da8af","Type":"ContainerStarted","Data":"0708953e3b10fd3c85004722f11b203ac70877fd3d834be4e2ddf6d1136009c7"}
Jul 14 13:27:55.401949 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:55.401868    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dhtvj" podStartSLOduration=8.587999447 podStartE2EDuration="40.401856579s" totalImagesPullingTime="31.813857132s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 13:27:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:27:55.401518865 +0000 UTC m=+40.727838794" watchObservedRunningTime="2026-07-14 13:27:55.401856579 +0000 UTC m=+40.728176485"
Jul 14 13:27:56.377702 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:56.377670    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vvmd5" event={"ID":"5cdbc484-7145-45f0-9060-a4a818174ff5","Type":"ContainerStarted","Data":"575e348518a9dba2b1629efefcf547d41144d1ee6cee0a225ee456754df9bd1e"}
Jul 14 13:27:56.399273 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:56.399232    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vvmd5" podStartSLOduration=3.261265243 podStartE2EDuration="6.399217488s" totalImagesPullingTime="3.137952245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:27:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:27:56.398337418 +0000 UTC m=+41.724657348" watchObservedRunningTime="2026-07-14 13:27:56.399217488 +0000 UTC m=+41.725537416"
Jul 14 13:27:58.350989 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:58.350957    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b89a211f-60b9-4eab-84cb-7aad1dcd94b1-metrics-tls\") pod \"dns-default-lkjjk\" (UID: \"b89a211f-60b9-4eab-84cb-7aad1dcd94b1\") " pod="openshift-dns/dns-default-lkjjk"
Jul 14 13:27:58.351386 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:58.351069    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 13:27:58.351386 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:58.351122    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b89a211f-60b9-4eab-84cb-7aad1dcd94b1-metrics-tls podName:b89a211f-60b9-4eab-84cb-7aad1dcd94b1 nodeName:}" failed. No retries permitted until 2026-07-14 13:28:06.351109407 +0000 UTC m=+51.677429313 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b89a211f-60b9-4eab-84cb-7aad1dcd94b1-metrics-tls") pod "dns-default-lkjjk" (UID: "b89a211f-60b9-4eab-84cb-7aad1dcd94b1") : secret "dns-default-metrics-tls" not found
Jul 14 13:27:58.548378 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:58.548353    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-584b976c99-8blm8"]
Jul 14 13:27:58.575118 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:58.575096    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-584b976c99-8blm8"]
Jul 14 13:27:58.575246 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:58.575181    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-584b976c99-8blm8"
Jul 14 13:27:58.577904 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:58.577885    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jzn49\""
Jul 14 13:27:58.578062 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:58.577883    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 14 13:27:58.578062 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:58.577883    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 14 13:27:58.578456 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:58.578430    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 14 13:27:58.585149 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:58.585128    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 14 13:27:58.754290 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:58.754261    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d137293e-b342-4b7e-938a-bc40f150ae6b-registry-certificates\") pod \"image-registry-584b976c99-8blm8\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") " pod="openshift-image-registry/image-registry-584b976c99-8blm8"
Jul 14 13:27:58.754290 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:58.754292    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d137293e-b342-4b7e-938a-bc40f150ae6b-trusted-ca\") pod \"image-registry-584b976c99-8blm8\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") " pod="openshift-image-registry/image-registry-584b976c99-8blm8"
Jul 14 13:27:58.754444 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:58.754311    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d137293e-b342-4b7e-938a-bc40f150ae6b-image-registry-private-configuration\") pod \"image-registry-584b976c99-8blm8\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") " pod="openshift-image-registry/image-registry-584b976c99-8blm8"
Jul 14 13:27:58.754444 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:58.754337    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d137293e-b342-4b7e-938a-bc40f150ae6b-ca-trust-extracted\") pod \"image-registry-584b976c99-8blm8\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") " pod="openshift-image-registry/image-registry-584b976c99-8blm8"
Jul 14 13:27:58.754444 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:58.754418    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d137293e-b342-4b7e-938a-bc40f150ae6b-registry-tls\") pod \"image-registry-584b976c99-8blm8\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") " pod="openshift-image-registry/image-registry-584b976c99-8blm8"
Jul 14 13:27:58.754444 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:58.754442    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d137293e-b342-4b7e-938a-bc40f150ae6b-installation-pull-secrets\") pod \"image-registry-584b976c99-8blm8\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") " pod="openshift-image-registry/image-registry-584b976c99-8blm8"
Jul 14 13:27:58.754623 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:58.754458    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d137293e-b342-4b7e-938a-bc40f150ae6b-bound-sa-token\") pod \"image-registry-584b976c99-8blm8\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") " pod="openshift-image-registry/image-registry-584b976c99-8blm8"
Jul 14 13:27:58.754623 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:58.754547    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5t5p\" (UniqueName: \"kubernetes.io/projected/d137293e-b342-4b7e-938a-bc40f150ae6b-kube-api-access-q5t5p\") pod \"image-registry-584b976c99-8blm8\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") " pod="openshift-image-registry/image-registry-584b976c99-8blm8"
Jul 14 13:27:58.854977 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:58.854949    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d137293e-b342-4b7e-938a-bc40f150ae6b-registry-tls\") pod \"image-registry-584b976c99-8blm8\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") " pod="openshift-image-registry/image-registry-584b976c99-8blm8"
Jul 14 13:27:58.854977 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:58.854978    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d137293e-b342-4b7e-938a-bc40f150ae6b-installation-pull-secrets\") pod \"image-registry-584b976c99-8blm8\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") " pod="openshift-image-registry/image-registry-584b976c99-8blm8"
Jul 14 13:27:58.855137 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:58.854996    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d137293e-b342-4b7e-938a-bc40f150ae6b-bound-sa-token\") pod \"image-registry-584b976c99-8blm8\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") " pod="openshift-image-registry/image-registry-584b976c99-8blm8"
Jul 14 13:27:58.855137 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:58.855059    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5t5p\" (UniqueName: \"kubernetes.io/projected/d137293e-b342-4b7e-938a-bc40f150ae6b-kube-api-access-q5t5p\") pod \"image-registry-584b976c99-8blm8\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") " pod="openshift-image-registry/image-registry-584b976c99-8blm8"
Jul 14 13:27:58.855137 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:58.855078    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d137293e-b342-4b7e-938a-bc40f150ae6b-registry-certificates\") pod \"image-registry-584b976c99-8blm8\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") " pod="openshift-image-registry/image-registry-584b976c99-8blm8"
Jul 14 13:27:58.855137 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:58.855093    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d137293e-b342-4b7e-938a-bc40f150ae6b-trusted-ca\") pod \"image-registry-584b976c99-8blm8\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") " pod="openshift-image-registry/image-registry-584b976c99-8blm8"
Jul 14 13:27:58.855137 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:58.855111    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d137293e-b342-4b7e-938a-bc40f150ae6b-image-registry-private-configuration\") pod \"image-registry-584b976c99-8blm8\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") " pod="openshift-image-registry/image-registry-584b976c99-8blm8"
Jul 14 13:27:58.855137 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:58.855128    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d137293e-b342-4b7e-938a-bc40f150ae6b-ca-trust-extracted\") pod \"image-registry-584b976c99-8blm8\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") " pod="openshift-image-registry/image-registry-584b976c99-8blm8"
Jul 14 13:27:58.855137 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:58.855127    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 14 13:27:58.855443 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:58.855147    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-584b976c99-8blm8: secret "image-registry-tls" not found
Jul 14 13:27:58.855443 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:58.855205    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d137293e-b342-4b7e-938a-bc40f150ae6b-registry-tls podName:d137293e-b342-4b7e-938a-bc40f150ae6b nodeName:}" failed. No retries permitted until 2026-07-14 13:27:59.355186412 +0000 UTC m=+44.681506337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d137293e-b342-4b7e-938a-bc40f150ae6b-registry-tls") pod "image-registry-584b976c99-8blm8" (UID: "d137293e-b342-4b7e-938a-bc40f150ae6b") : secret "image-registry-tls" not found
Jul 14 13:27:58.855649 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:58.855623    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d137293e-b342-4b7e-938a-bc40f150ae6b-ca-trust-extracted\") pod \"image-registry-584b976c99-8blm8\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") " pod="openshift-image-registry/image-registry-584b976c99-8blm8"
Jul 14 13:27:58.855750 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:58.855734    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d137293e-b342-4b7e-938a-bc40f150ae6b-registry-certificates\") pod \"image-registry-584b976c99-8blm8\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") " pod="openshift-image-registry/image-registry-584b976c99-8blm8"
Jul 14 13:27:58.855941 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:58.855923    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d137293e-b342-4b7e-938a-bc40f150ae6b-trusted-ca\") pod \"image-registry-584b976c99-8blm8\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") " pod="openshift-image-registry/image-registry-584b976c99-8blm8"
Jul 14 13:27:58.858498 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:58.858471    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d137293e-b342-4b7e-938a-bc40f150ae6b-installation-pull-secrets\") pod \"image-registry-584b976c99-8blm8\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") " pod="openshift-image-registry/image-registry-584b976c99-8blm8"
Jul 14 13:27:58.858702 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:58.858681    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d137293e-b342-4b7e-938a-bc40f150ae6b-image-registry-private-configuration\") pod \"image-registry-584b976c99-8blm8\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") " pod="openshift-image-registry/image-registry-584b976c99-8blm8"
Jul 14 13:27:58.864828 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:58.864807    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d137293e-b342-4b7e-938a-bc40f150ae6b-bound-sa-token\") pod \"image-registry-584b976c99-8blm8\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") " pod="openshift-image-registry/image-registry-584b976c99-8blm8"
Jul 14 13:27:58.865001 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:58.864982    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5t5p\" (UniqueName: \"kubernetes.io/projected/d137293e-b342-4b7e-938a-bc40f150ae6b-kube-api-access-q5t5p\") pod \"image-registry-584b976c99-8blm8\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") " pod="openshift-image-registry/image-registry-584b976c99-8blm8"
Jul 14 13:27:59.359132 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:27:59.359105    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d137293e-b342-4b7e-938a-bc40f150ae6b-registry-tls\") pod \"image-registry-584b976c99-8blm8\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") " pod="openshift-image-registry/image-registry-584b976c99-8blm8"
Jul 14 13:27:59.359451 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:59.359221    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 14 13:27:59.359451 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:59.359233    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-584b976c99-8blm8: secret "image-registry-tls" not found
Jul 14 13:27:59.359451 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:27:59.359307    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d137293e-b342-4b7e-938a-bc40f150ae6b-registry-tls podName:d137293e-b342-4b7e-938a-bc40f150ae6b nodeName:}" failed. No retries permitted until 2026-07-14 13:28:00.3592896 +0000 UTC m=+45.685609506 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d137293e-b342-4b7e-938a-bc40f150ae6b-registry-tls") pod "image-registry-584b976c99-8blm8" (UID: "d137293e-b342-4b7e-938a-bc40f150ae6b") : secret "image-registry-tls" not found
Jul 14 13:28:00.366241 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:00.366203    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d137293e-b342-4b7e-938a-bc40f150ae6b-registry-tls\") pod \"image-registry-584b976c99-8blm8\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") " pod="openshift-image-registry/image-registry-584b976c99-8blm8"
Jul 14 13:28:00.366595 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:00.366320    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 14 13:28:00.366595 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:00.366331    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-584b976c99-8blm8: secret "image-registry-tls" not found
Jul 14 13:28:00.366595 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:00.366394    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d137293e-b342-4b7e-938a-bc40f150ae6b-registry-tls podName:d137293e-b342-4b7e-938a-bc40f150ae6b nodeName:}" failed. No retries permitted until 2026-07-14 13:28:02.366372199 +0000 UTC m=+47.692692120 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d137293e-b342-4b7e-938a-bc40f150ae6b-registry-tls") pod "image-registry-584b976c99-8blm8" (UID: "d137293e-b342-4b7e-938a-bc40f150ae6b") : secret "image-registry-tls" not found
Jul 14 13:28:00.546664 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:00.546634    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xz2lp"]
Jul 14 13:28:00.578838 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:00.578811    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xz2lp"]
Jul 14 13:28:00.578967 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:00.578923    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xz2lp"
Jul 14 13:28:00.581450 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:00.581427    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 14 13:28:00.581570 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:00.581463    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 14 13:28:00.581570 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:00.581499    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 14 13:28:00.582719 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:00.582701    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tbwpd\""
Jul 14 13:28:00.582814 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:00.582772    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 14 13:28:00.768649 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:00.768624    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xz2lp\" (UID: \"c650b1a4-6066-4d12-b4cb-0217c8f4ebfe\") " pod="openshift-insights/insights-runtime-extractor-xz2lp"
Jul 14 13:28:00.768791 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:00.768748    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-crio-socket\") pod \"insights-runtime-extractor-xz2lp\" (UID: \"c650b1a4-6066-4d12-b4cb-0217c8f4ebfe\") " pod="openshift-insights/insights-runtime-extractor-xz2lp"
Jul 14 13:28:00.768852 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:00.768795    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xz2lp\" (UID: \"c650b1a4-6066-4d12-b4cb-0217c8f4ebfe\") " pod="openshift-insights/insights-runtime-extractor-xz2lp"
Jul 14 13:28:00.768852 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:00.768820    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-data-volume\") pod \"insights-runtime-extractor-xz2lp\" (UID: \"c650b1a4-6066-4d12-b4cb-0217c8f4ebfe\") " pod="openshift-insights/insights-runtime-extractor-xz2lp"
Jul 14 13:28:00.768940 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:00.768854    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppd5k\" (UniqueName: \"kubernetes.io/projected/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-kube-api-access-ppd5k\") pod \"insights-runtime-extractor-xz2lp\" (UID: \"c650b1a4-6066-4d12-b4cb-0217c8f4ebfe\") " pod="openshift-insights/insights-runtime-extractor-xz2lp"
Jul 14 13:28:00.870091 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:00.870070    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-crio-socket\") pod \"insights-runtime-extractor-xz2lp\" (UID: \"c650b1a4-6066-4d12-b4cb-0217c8f4ebfe\") " pod="openshift-insights/insights-runtime-extractor-xz2lp"
Jul 14 13:28:00.870212 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:00.870111    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xz2lp\" (UID: \"c650b1a4-6066-4d12-b4cb-0217c8f4ebfe\") " pod="openshift-insights/insights-runtime-extractor-xz2lp"
Jul 14 13:28:00.870212 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:00.870130    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-data-volume\") pod \"insights-runtime-extractor-xz2lp\" (UID: \"c650b1a4-6066-4d12-b4cb-0217c8f4ebfe\") " pod="openshift-insights/insights-runtime-extractor-xz2lp"
Jul 14 13:28:00.870212 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:00.870148    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppd5k\" (UniqueName: \"kubernetes.io/projected/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-kube-api-access-ppd5k\") pod \"insights-runtime-extractor-xz2lp\" (UID: \"c650b1a4-6066-4d12-b4cb-0217c8f4ebfe\") " pod="openshift-insights/insights-runtime-extractor-xz2lp"
Jul 14 13:28:00.870212 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:00.870171    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xz2lp\" (UID: \"c650b1a4-6066-4d12-b4cb-0217c8f4ebfe\") " pod="openshift-insights/insights-runtime-extractor-xz2lp"
Jul 14 13:28:00.870359 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:00.870245    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 14 13:28:00.870359 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:00.870283    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-crio-socket\") pod \"insights-runtime-extractor-xz2lp\" (UID: \"c650b1a4-6066-4d12-b4cb-0217c8f4ebfe\") " pod="openshift-insights/insights-runtime-extractor-xz2lp"
Jul 14 13:28:00.870359 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:00.870248    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 14 13:28:00.870359 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:00.870319    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-kube-rbac-proxy-cm podName:c650b1a4-6066-4d12-b4cb-0217c8f4ebfe nodeName:}" failed. No retries permitted until 2026-07-14 13:28:01.370299895 +0000 UTC m=+46.696619802 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xz2lp" (UID: "c650b1a4-6066-4d12-b4cb-0217c8f4ebfe") : configmap "kube-rbac-proxy" not found
Jul 14 13:28:00.870518 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:00.870368    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-insights-runtime-extractor-tls podName:c650b1a4-6066-4d12-b4cb-0217c8f4ebfe nodeName:}" failed. No retries permitted until 2026-07-14 13:28:01.370352788 +0000 UTC m=+46.696672697 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xz2lp" (UID: "c650b1a4-6066-4d12-b4cb-0217c8f4ebfe") : secret "insights-runtime-extractor-tls" not found
Jul 14 13:28:00.870518 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:00.870447    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-data-volume\") pod \"insights-runtime-extractor-xz2lp\" (UID: \"c650b1a4-6066-4d12-b4cb-0217c8f4ebfe\") " pod="openshift-insights/insights-runtime-extractor-xz2lp"
Jul 14 13:28:00.883620 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:00.883591    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppd5k\" (UniqueName: \"kubernetes.io/projected/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-kube-api-access-ppd5k\") pod \"insights-runtime-extractor-xz2lp\" (UID: \"c650b1a4-6066-4d12-b4cb-0217c8f4ebfe\") " pod="openshift-insights/insights-runtime-extractor-xz2lp"
Jul 14 13:28:01.375185 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:01.375159    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xz2lp\" (UID: \"c650b1a4-6066-4d12-b4cb-0217c8f4ebfe\") " pod="openshift-insights/insights-runtime-extractor-xz2lp"
Jul 14 13:28:01.375543 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:01.375199    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xz2lp\" (UID: \"c650b1a4-6066-4d12-b4cb-0217c8f4ebfe\") " pod="openshift-insights/insights-runtime-extractor-xz2lp"
Jul 14 13:28:01.375543 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:01.375264    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 14 13:28:01.375543 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:01.375296    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 14 13:28:01.375543 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:01.375314    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-kube-rbac-proxy-cm podName:c650b1a4-6066-4d12-b4cb-0217c8f4ebfe nodeName:}" failed. No retries permitted until 2026-07-14 13:28:02.375296037 +0000 UTC m=+47.701615945 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xz2lp" (UID: "c650b1a4-6066-4d12-b4cb-0217c8f4ebfe") : configmap "kube-rbac-proxy" not found
Jul 14 13:28:01.375543 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:01.375354    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-insights-runtime-extractor-tls podName:c650b1a4-6066-4d12-b4cb-0217c8f4ebfe nodeName:}" failed. No retries permitted until 2026-07-14 13:28:02.375339928 +0000 UTC m=+47.701659834 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xz2lp" (UID: "c650b1a4-6066-4d12-b4cb-0217c8f4ebfe") : secret "insights-runtime-extractor-tls" not found
Jul 14 13:28:02.380512 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:02.380476    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xz2lp\" (UID: \"c650b1a4-6066-4d12-b4cb-0217c8f4ebfe\") " pod="openshift-insights/insights-runtime-extractor-xz2lp"
Jul 14 13:28:02.380986 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:02.380523    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xz2lp\" (UID: \"c650b1a4-6066-4d12-b4cb-0217c8f4ebfe\") " pod="openshift-insights/insights-runtime-extractor-xz2lp"
Jul 14 13:28:02.380986 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:02.380567    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d137293e-b342-4b7e-938a-bc40f150ae6b-registry-tls\") pod \"image-registry-584b976c99-8blm8\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") " pod="openshift-image-registry/image-registry-584b976c99-8blm8"
Jul 14 13:28:02.380986 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:02.380644    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 14 13:28:02.380986 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:02.380655    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 14 13:28:02.380986 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:02.380667    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-584b976c99-8blm8: secret "image-registry-tls" not found
Jul 14 13:28:02.380986 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:02.380670    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 14 13:28:02.380986 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:02.380710    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d137293e-b342-4b7e-938a-bc40f150ae6b-registry-tls podName:d137293e-b342-4b7e-938a-bc40f150ae6b nodeName:}" failed. No retries permitted until 2026-07-14 13:28:06.380697989 +0000 UTC m=+51.707017912 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d137293e-b342-4b7e-938a-bc40f150ae6b-registry-tls") pod "image-registry-584b976c99-8blm8" (UID: "d137293e-b342-4b7e-938a-bc40f150ae6b") : secret "image-registry-tls" not found
Jul 14 13:28:02.380986 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:02.380723    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-kube-rbac-proxy-cm podName:c650b1a4-6066-4d12-b4cb-0217c8f4ebfe nodeName:}" failed. No retries permitted until 2026-07-14 13:28:04.380716988 +0000 UTC m=+49.707036894 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xz2lp" (UID: "c650b1a4-6066-4d12-b4cb-0217c8f4ebfe") : configmap "kube-rbac-proxy" not found
Jul 14 13:28:02.380986 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:02.380734    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-insights-runtime-extractor-tls podName:c650b1a4-6066-4d12-b4cb-0217c8f4ebfe nodeName:}" failed. No retries permitted until 2026-07-14 13:28:04.380729579 +0000 UTC m=+49.707049484 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xz2lp" (UID: "c650b1a4-6066-4d12-b4cb-0217c8f4ebfe") : secret "insights-runtime-extractor-tls" not found
Jul 14 13:28:03.327510 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:03.327484    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mjgwk_03018596-d0b1-43da-af02-ede52df67ceb/dns-node-resolver/0.log"
Jul 14 13:28:04.120306 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:04.120282    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7l994_bf204a1f-63ca-40ac-825b-e8684c67e36f/node-ca/0.log"
Jul 14 13:28:04.394803 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:04.394732    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xz2lp\" (UID: \"c650b1a4-6066-4d12-b4cb-0217c8f4ebfe\") " pod="openshift-insights/insights-runtime-extractor-xz2lp"
Jul 14 13:28:04.394803 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:04.394770    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xz2lp\" (UID: \"c650b1a4-6066-4d12-b4cb-0217c8f4ebfe\") " pod="openshift-insights/insights-runtime-extractor-xz2lp"
Jul 14 13:28:04.395001 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:04.394841    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 14 13:28:04.395001 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:04.394886    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 14 13:28:04.395001 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:04.394892    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-kube-rbac-proxy-cm podName:c650b1a4-6066-4d12-b4cb-0217c8f4ebfe nodeName:}" failed. No retries permitted until 2026-07-14 13:28:08.394879494 +0000 UTC m=+53.721199400 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xz2lp" (UID: "c650b1a4-6066-4d12-b4cb-0217c8f4ebfe") : configmap "kube-rbac-proxy" not found
Jul 14 13:28:04.395001 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:04.394952    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-insights-runtime-extractor-tls podName:c650b1a4-6066-4d12-b4cb-0217c8f4ebfe nodeName:}" failed. No retries permitted until 2026-07-14 13:28:08.394933247 +0000 UTC m=+53.721253152 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xz2lp" (UID: "c650b1a4-6066-4d12-b4cb-0217c8f4ebfe") : secret "insights-runtime-extractor-tls" not found
Jul 14 13:28:05.320628 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:05.320590    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vvmd5_5cdbc484-7145-45f0-9060-a4a818174ff5/serve-healthcheck-canary/0.log"
Jul 14 13:28:06.408512 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:06.408484    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b89a211f-60b9-4eab-84cb-7aad1dcd94b1-metrics-tls\") pod \"dns-default-lkjjk\" (UID: \"b89a211f-60b9-4eab-84cb-7aad1dcd94b1\") " pod="openshift-dns/dns-default-lkjjk"
Jul 14 13:28:06.408873 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:06.408547    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d137293e-b342-4b7e-938a-bc40f150ae6b-registry-tls\") pod \"image-registry-584b976c99-8blm8\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") " pod="openshift-image-registry/image-registry-584b976c99-8blm8"
Jul 14 13:28:06.408873 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:06.408627    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 14 13:28:06.408873 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:06.408630    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 13:28:06.408873 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:06.408638    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-584b976c99-8blm8: secret "image-registry-tls" not found
Jul 14 13:28:06.408873 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:06.408691    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b89a211f-60b9-4eab-84cb-7aad1dcd94b1-metrics-tls podName:b89a211f-60b9-4eab-84cb-7aad1dcd94b1 nodeName:}" failed. No retries permitted until 2026-07-14 13:28:22.408672359 +0000 UTC m=+67.734992485 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b89a211f-60b9-4eab-84cb-7aad1dcd94b1-metrics-tls") pod "dns-default-lkjjk" (UID: "b89a211f-60b9-4eab-84cb-7aad1dcd94b1") : secret "dns-default-metrics-tls" not found
Jul 14 13:28:06.408873 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:06.408710    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d137293e-b342-4b7e-938a-bc40f150ae6b-registry-tls podName:d137293e-b342-4b7e-938a-bc40f150ae6b nodeName:}" failed. No retries permitted until 2026-07-14 13:28:14.408701864 +0000 UTC m=+59.735021797 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d137293e-b342-4b7e-938a-bc40f150ae6b-registry-tls") pod "image-registry-584b976c99-8blm8" (UID: "d137293e-b342-4b7e-938a-bc40f150ae6b") : secret "image-registry-tls" not found
Jul 14 13:28:08.422430 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:08.422394    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xz2lp\" (UID: \"c650b1a4-6066-4d12-b4cb-0217c8f4ebfe\") " pod="openshift-insights/insights-runtime-extractor-xz2lp"
Jul 14 13:28:08.422948 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:08.422449    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xz2lp\" (UID: \"c650b1a4-6066-4d12-b4cb-0217c8f4ebfe\") " pod="openshift-insights/insights-runtime-extractor-xz2lp"
Jul 14 13:28:08.422948 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:08.422534    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 14 13:28:08.422948 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:08.422590    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-insights-runtime-extractor-tls podName:c650b1a4-6066-4d12-b4cb-0217c8f4ebfe nodeName:}" failed. No retries permitted until 2026-07-14 13:28:16.422576416 +0000 UTC m=+61.748896322 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xz2lp" (UID: "c650b1a4-6066-4d12-b4cb-0217c8f4ebfe") : secret "insights-runtime-extractor-tls" not found
Jul 14 13:28:08.422948 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:08.422534    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 14 13:28:08.422948 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:08.422621    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-kube-rbac-proxy-cm podName:c650b1a4-6066-4d12-b4cb-0217c8f4ebfe nodeName:}" failed. No retries permitted until 2026-07-14 13:28:16.422615463 +0000 UTC m=+61.748935369 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xz2lp" (UID: "c650b1a4-6066-4d12-b4cb-0217c8f4ebfe") : configmap "kube-rbac-proxy" not found
Jul 14 13:28:14.470424 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:14.470394    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d137293e-b342-4b7e-938a-bc40f150ae6b-registry-tls\") pod \"image-registry-584b976c99-8blm8\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") " pod="openshift-image-registry/image-registry-584b976c99-8blm8"
Jul 14 13:28:14.472546 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:14.472523    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d137293e-b342-4b7e-938a-bc40f150ae6b-registry-tls\") pod \"image-registry-584b976c99-8blm8\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") " pod="openshift-image-registry/image-registry-584b976c99-8blm8"
Jul 14 13:28:14.488467 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:14.488444    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-584b976c99-8blm8"
Jul 14 13:28:14.607806 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:14.607779    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-584b976c99-8blm8"]
Jul 14 13:28:14.610532 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:28:14.610500    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd137293e_b342_4b7e_938a_bc40f150ae6b.slice/crio-8f3ed3a35445138efb7e0056c2e675a5688110cb8dc74c9e0afa3bb4656e1bc3 WatchSource:0}: Error finding container 8f3ed3a35445138efb7e0056c2e675a5688110cb8dc74c9e0afa3bb4656e1bc3: Status 404 returned error can't find the container with id 8f3ed3a35445138efb7e0056c2e675a5688110cb8dc74c9e0afa3bb4656e1bc3
Jul 14 13:28:15.422058 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:15.422021    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-584b976c99-8blm8" event={"ID":"d137293e-b342-4b7e-938a-bc40f150ae6b","Type":"ContainerStarted","Data":"c1782b12cc6ed6d43659b9a5f324ed868158156191dade404083d7c1e3d974db"}
Jul 14 13:28:15.422218 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:15.422074    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-584b976c99-8blm8" event={"ID":"d137293e-b342-4b7e-938a-bc40f150ae6b","Type":"ContainerStarted","Data":"8f3ed3a35445138efb7e0056c2e675a5688110cb8dc74c9e0afa3bb4656e1bc3"}
Jul 14 13:28:15.422218 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:15.422108    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-584b976c99-8blm8"
Jul 14 13:28:15.446661 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:15.446617    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-584b976c99-8blm8" podStartSLOduration=17.446602895 podStartE2EDuration="17.446602895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:27:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:28:15.446274392 +0000 UTC m=+60.772594345" watchObservedRunningTime="2026-07-14 13:28:15.446602895 +0000 UTC m=+60.772922822"
Jul 14 13:28:16.346568 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:16.346540    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vq8tk"
Jul 14 13:28:16.493207 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:16.493176    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xz2lp\" (UID: \"c650b1a4-6066-4d12-b4cb-0217c8f4ebfe\") " pod="openshift-insights/insights-runtime-extractor-xz2lp"
Jul 14 13:28:16.493368 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:16.493348    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xz2lp\" (UID: \"c650b1a4-6066-4d12-b4cb-0217c8f4ebfe\") " pod="openshift-insights/insights-runtime-extractor-xz2lp"
Jul 14 13:28:16.493602 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:16.493582    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 14 13:28:16.493693 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:16.493650    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-insights-runtime-extractor-tls podName:c650b1a4-6066-4d12-b4cb-0217c8f4ebfe nodeName:}" failed. No retries permitted until 2026-07-14 13:28:32.493631355 +0000 UTC m=+77.819951268 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xz2lp" (UID: "c650b1a4-6066-4d12-b4cb-0217c8f4ebfe") : secret "insights-runtime-extractor-tls" not found
Jul 14 13:28:16.496685 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:16.494148    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xz2lp\" (UID: \"c650b1a4-6066-4d12-b4cb-0217c8f4ebfe\") " pod="openshift-insights/insights-runtime-extractor-xz2lp"
Jul 14 13:28:19.917307 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:19.917268    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd9c8cb5-09e3-4116-b407-01ebe6c06999-metrics-certs\") pod \"network-metrics-daemon-w4rvb\" (UID: \"fd9c8cb5-09e3-4116-b407-01ebe6c06999\") " pod="openshift-multus/network-metrics-daemon-w4rvb"
Jul 14 13:28:19.917307 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:19.917313    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b33bdbd5-63be-4828-a2d6-91d4bead7b80-original-pull-secret\") pod \"global-pull-secret-syncer-s7ft2\" (UID: \"b33bdbd5-63be-4828-a2d6-91d4bead7b80\") " pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:28:19.920374 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:19.920352    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 14 13:28:19.921399 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:19.921383    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 14 13:28:19.929866 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:19.929838    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd9c8cb5-09e3-4116-b407-01ebe6c06999-metrics-certs\") pod \"network-metrics-daemon-w4rvb\" (UID: \"fd9c8cb5-09e3-4116-b407-01ebe6c06999\") " pod="openshift-multus/network-metrics-daemon-w4rvb"
Jul 14 13:28:19.929962 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:19.929849    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b33bdbd5-63be-4828-a2d6-91d4bead7b80-original-pull-secret\") pod \"global-pull-secret-syncer-s7ft2\" (UID: \"b33bdbd5-63be-4828-a2d6-91d4bead7b80\") " pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:28:20.018039 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:20.017999    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ff4m\" (UniqueName: \"kubernetes.io/projected/d87134ea-5c27-47f7-a629-883bae028590-kube-api-access-9ff4m\") pod \"network-check-target-42hkp\" (UID: \"d87134ea-5c27-47f7-a629-883bae028590\") " pod="openshift-network-diagnostics/network-check-target-42hkp"
Jul 14 13:28:20.020677 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:20.020662    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 14 13:28:20.031696 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:20.031677    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 14 13:28:20.041962 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:20.041938    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ff4m\" (UniqueName: \"kubernetes.io/projected/d87134ea-5c27-47f7-a629-883bae028590-kube-api-access-9ff4m\") pod \"network-check-target-42hkp\" (UID: \"d87134ea-5c27-47f7-a629-883bae028590\") " pod="openshift-network-diagnostics/network-check-target-42hkp"
Jul 14 13:28:20.103021 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:20.102997    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rgn8m\""
Jul 14 13:28:20.107510 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:20.107498    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7ft2"
Jul 14 13:28:20.110515 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:20.110499    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-42hkp"
Jul 14 13:28:20.116924 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:20.116906    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l7sfg\""
Jul 14 13:28:20.124455 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:20.124437    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w4rvb"
Jul 14 13:28:20.247626 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:20.247594    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-42hkp"]
Jul 14 13:28:20.251264 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:28:20.251234    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd87134ea_5c27_47f7_a629_883bae028590.slice/crio-e6b2ae63b4c3f7b924c91b059c8c8a81dc82556e83deac87e7ab9d24ebd298e9 WatchSource:0}: Error finding container e6b2ae63b4c3f7b924c91b059c8c8a81dc82556e83deac87e7ab9d24ebd298e9: Status 404 returned error can't find the container with id e6b2ae63b4c3f7b924c91b059c8c8a81dc82556e83deac87e7ab9d24ebd298e9
Jul 14 13:28:20.265786 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:20.265761    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-s7ft2"]
Jul 14 13:28:20.268183 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:28:20.268152    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb33bdbd5_63be_4828_a2d6_91d4bead7b80.slice/crio-36249eb451df637414ff22cf624d27111efe00b1da1eba0d6706bedc601eff84 WatchSource:0}: Error finding container 36249eb451df637414ff22cf624d27111efe00b1da1eba0d6706bedc601eff84: Status 404 returned error can't find the container with id 36249eb451df637414ff22cf624d27111efe00b1da1eba0d6706bedc601eff84
Jul 14 13:28:20.281914 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:20.281893    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-w4rvb"]
Jul 14 13:28:20.286449 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:28:20.286427    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd9c8cb5_09e3_4116_b407_01ebe6c06999.slice/crio-ad234dc1c8e24a58307f79e08ed842e9a1649075684d9e6ae7ffa2293e752adb WatchSource:0}: Error finding container ad234dc1c8e24a58307f79e08ed842e9a1649075684d9e6ae7ffa2293e752adb: Status 404 returned error can't find the container with id ad234dc1c8e24a58307f79e08ed842e9a1649075684d9e6ae7ffa2293e752adb
Jul 14 13:28:20.433785 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:20.433724    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w4rvb" event={"ID":"fd9c8cb5-09e3-4116-b407-01ebe6c06999","Type":"ContainerStarted","Data":"ad234dc1c8e24a58307f79e08ed842e9a1649075684d9e6ae7ffa2293e752adb"}
Jul 14 13:28:20.434779 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:20.434756    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-s7ft2" event={"ID":"b33bdbd5-63be-4828-a2d6-91d4bead7b80","Type":"ContainerStarted","Data":"36249eb451df637414ff22cf624d27111efe00b1da1eba0d6706bedc601eff84"}
Jul 14 13:28:20.435693 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:20.435677    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-42hkp" event={"ID":"d87134ea-5c27-47f7-a629-883bae028590","Type":"ContainerStarted","Data":"e6b2ae63b4c3f7b924c91b059c8c8a81dc82556e83deac87e7ab9d24ebd298e9"}
Jul 14 13:28:22.434383 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:22.434347    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b89a211f-60b9-4eab-84cb-7aad1dcd94b1-metrics-tls\") pod \"dns-default-lkjjk\" (UID: \"b89a211f-60b9-4eab-84cb-7aad1dcd94b1\") " pod="openshift-dns/dns-default-lkjjk"
Jul 14 13:28:22.437137 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:22.437108    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b89a211f-60b9-4eab-84cb-7aad1dcd94b1-metrics-tls\") pod \"dns-default-lkjjk\" (UID: \"b89a211f-60b9-4eab-84cb-7aad1dcd94b1\") " pod="openshift-dns/dns-default-lkjjk"
Jul 14 13:28:22.449515 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:22.449486    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w4rvb" event={"ID":"fd9c8cb5-09e3-4116-b407-01ebe6c06999","Type":"ContainerStarted","Data":"e1b847d1ee233bde0bb3ced28ecccd77984522f8e33ebd8f39d04a5e284d292a"}
Jul 14 13:28:22.449640 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:22.449581    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w4rvb" event={"ID":"fd9c8cb5-09e3-4116-b407-01ebe6c06999","Type":"ContainerStarted","Data":"4314a82a76a4d05d0a45bc388afe2b5121ce731a853607c648def75b79ee0bfc"}
Jul 14 13:28:22.684846 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:22.684771    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6rczr\""
Jul 14 13:28:22.692832 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:22.692767    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lkjjk"
Jul 14 13:28:23.456520 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:23.456480    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-42hkp" event={"ID":"d87134ea-5c27-47f7-a629-883bae028590","Type":"ContainerStarted","Data":"4a77399de44a295aa1fa810e8911947d8a4e3548c7fd732f2cdaf306594bd8ca"}
Jul 14 13:28:23.456978 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:23.456959    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-42hkp"
Jul 14 13:28:23.477211 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:23.477162    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-42hkp" podStartSLOduration=65.357851972 podStartE2EDuration="1m8.477143254s" totalImagesPullingTime="3.119291282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:27:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:28:23.476687239 +0000 UTC m=+68.803007162" watchObservedRunningTime="2026-07-14 13:28:23.477143254 +0000 UTC m=+68.803463184"
Jul 14 13:28:23.477580 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:23.477556    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-w4rvb" podStartSLOduration=67.349888846 podStartE2EDuration="1m8.477547034s" totalImagesPullingTime="1.127658188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:27:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:28:22.46890725 +0000 UTC m=+67.795227179" watchObservedRunningTime="2026-07-14 13:28:23.477547034 +0000 UTC m=+68.803866962"
Jul 14 13:28:23.482631 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:23.482617    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lkjjk"]
Jul 14 13:28:23.484932 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:28:23.484909    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb89a211f_60b9_4eab_84cb_7aad1dcd94b1.slice/crio-78677506f88bc052cf9868cec054326d016c98c6c7112d356ffdd47745a1adf2 WatchSource:0}: Error finding container 78677506f88bc052cf9868cec054326d016c98c6c7112d356ffdd47745a1adf2: Status 404 returned error can't find the container with id 78677506f88bc052cf9868cec054326d016c98c6c7112d356ffdd47745a1adf2
Jul 14 13:28:24.461965 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:24.461936    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lkjjk" event={"ID":"b89a211f-60b9-4eab-84cb-7aad1dcd94b1","Type":"ContainerStarted","Data":"78677506f88bc052cf9868cec054326d016c98c6c7112d356ffdd47745a1adf2"}
Jul 14 13:28:25.162537 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.162511    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-78b87b794c-w72v9"]
Jul 14 13:28:25.179950 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.179236    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-nqmzd"]
Jul 14 13:28:25.180135 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.180118    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78b87b794c-w72v9"
Jul 14 13:28:25.182982 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.182957    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 14 13:28:25.184284 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.184258    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 14 13:28:25.184407 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.184298    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 14 13:28:25.184407 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.184262    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 14 13:28:25.195360 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.195342    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-nqmzd"
Jul 14 13:28:25.198364 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.198325    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 14 13:28:25.198868 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.198847    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-78b87b794c-w72v9"]
Jul 14 13:28:25.198868 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.198861    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-7bvmp\""
Jul 14 13:28:25.199053 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.198890    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 14 13:28:25.199053 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.198901    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-nqmzd"]
Jul 14 13:28:25.199053 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.198926    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-584b976c99-8blm8"]
Jul 14 13:28:25.236205 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.236179    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-59f76b4df9-2glpk"]
Jul 14 13:28:25.248258 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.248234    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59f76b4df9-2glpk"
Jul 14 13:28:25.254549 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.254529    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91c843a4-a274-4a95-8833-4977544f0646-tmp\") pod \"klusterlet-addon-workmgr-78b87b794c-w72v9\" (UID: \"91c843a4-a274-4a95-8833-4977544f0646\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78b87b794c-w72v9"
Jul 14 13:28:25.254672 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.254585    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/421a6500-d253-4f95-beb4-db5438d099c9-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-nqmzd\" (UID: \"421a6500-d253-4f95-beb4-db5438d099c9\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-nqmzd"
Jul 14 13:28:25.254672 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.254645    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/91c843a4-a274-4a95-8833-4977544f0646-klusterlet-config\") pod \"klusterlet-addon-workmgr-78b87b794c-w72v9\" (UID: \"91c843a4-a274-4a95-8833-4977544f0646\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78b87b794c-w72v9"
Jul 14 13:28:25.254802 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.254683    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/421a6500-d253-4f95-beb4-db5438d099c9-nginx-conf\") pod \"networking-console-plugin-6cf856758c-nqmzd\" (UID: \"421a6500-d253-4f95-beb4-db5438d099c9\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-nqmzd"
Jul 14 13:28:25.254802 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.254732    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwcgz\" (UniqueName: \"kubernetes.io/projected/91c843a4-a274-4a95-8833-4977544f0646-kube-api-access-qwcgz\") pod \"klusterlet-addon-workmgr-78b87b794c-w72v9\" (UID: \"91c843a4-a274-4a95-8833-4977544f0646\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78b87b794c-w72v9"
Jul 14 13:28:25.254899 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.254821    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59f76b4df9-2glpk"]
Jul 14 13:28:25.355895 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.355865    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ca6401ee-66b4-449f-920d-74ef689c7429-trusted-ca\") pod \"image-registry-59f76b4df9-2glpk\" (UID: \"ca6401ee-66b4-449f-920d-74ef689c7429\") " pod="openshift-image-registry/image-registry-59f76b4df9-2glpk"
Jul 14 13:28:25.356040 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.355915    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwcgz\" (UniqueName: \"kubernetes.io/projected/91c843a4-a274-4a95-8833-4977544f0646-kube-api-access-qwcgz\") pod \"klusterlet-addon-workmgr-78b87b794c-w72v9\" (UID: \"91c843a4-a274-4a95-8833-4977544f0646\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78b87b794c-w72v9"
Jul 14 13:28:25.356040 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.355948    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/91c843a4-a274-4a95-8833-4977544f0646-klusterlet-config\") pod \"klusterlet-addon-workmgr-78b87b794c-w72v9\" (UID: \"91c843a4-a274-4a95-8833-4977544f0646\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78b87b794c-w72v9"
Jul 14 13:28:25.356040 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.355972    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/421a6500-d253-4f95-beb4-db5438d099c9-nginx-conf\") pod \"networking-console-plugin-6cf856758c-nqmzd\" (UID: \"421a6500-d253-4f95-beb4-db5438d099c9\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-nqmzd"
Jul 14 13:28:25.356040 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.356004    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ca6401ee-66b4-449f-920d-74ef689c7429-installation-pull-secrets\") pod \"image-registry-59f76b4df9-2glpk\" (UID: \"ca6401ee-66b4-449f-920d-74ef689c7429\") " pod="openshift-image-registry/image-registry-59f76b4df9-2glpk"
Jul 14 13:28:25.356252 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.356056    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ca6401ee-66b4-449f-920d-74ef689c7429-ca-trust-extracted\") pod \"image-registry-59f76b4df9-2glpk\" (UID: \"ca6401ee-66b4-449f-920d-74ef689c7429\") " pod="openshift-image-registry/image-registry-59f76b4df9-2glpk"
Jul 14 13:28:25.356252 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.356089    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ca6401ee-66b4-449f-920d-74ef689c7429-image-registry-private-configuration\") pod \"image-registry-59f76b4df9-2glpk\" (UID: \"ca6401ee-66b4-449f-920d-74ef689c7429\") " pod="openshift-image-registry/image-registry-59f76b4df9-2glpk"
Jul 14 13:28:25.356252 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.356118    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91c843a4-a274-4a95-8833-4977544f0646-tmp\") pod \"klusterlet-addon-workmgr-78b87b794c-w72v9\" (UID: \"91c843a4-a274-4a95-8833-4977544f0646\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78b87b794c-w72v9"
Jul 14 13:28:25.356252 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.356169    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ca6401ee-66b4-449f-920d-74ef689c7429-registry-certificates\") pod \"image-registry-59f76b4df9-2glpk\" (UID: \"ca6401ee-66b4-449f-920d-74ef689c7429\") " pod="openshift-image-registry/image-registry-59f76b4df9-2glpk"
Jul 14 13:28:25.356252 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.356192    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ca6401ee-66b4-449f-920d-74ef689c7429-registry-tls\") pod \"image-registry-59f76b4df9-2glpk\" (UID: \"ca6401ee-66b4-449f-920d-74ef689c7429\") " pod="openshift-image-registry/image-registry-59f76b4df9-2glpk"
Jul 14 13:28:25.356252 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.356228    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2xbr\" (UniqueName: \"kubernetes.io/projected/ca6401ee-66b4-449f-920d-74ef689c7429-kube-api-access-j2xbr\") pod \"image-registry-59f76b4df9-2glpk\" (UID: \"ca6401ee-66b4-449f-920d-74ef689c7429\") " pod="openshift-image-registry/image-registry-59f76b4df9-2glpk"
Jul 14 13:28:25.356529 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.356284    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/421a6500-d253-4f95-beb4-db5438d099c9-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-nqmzd\" (UID: \"421a6500-d253-4f95-beb4-db5438d099c9\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-nqmzd"
Jul 14 13:28:25.356529 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.356308    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca6401ee-66b4-449f-920d-74ef689c7429-bound-sa-token\") pod \"image-registry-59f76b4df9-2glpk\" (UID: \"ca6401ee-66b4-449f-920d-74ef689c7429\") " pod="openshift-image-registry/image-registry-59f76b4df9-2glpk"
Jul 14 13:28:25.356784 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.356760    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91c843a4-a274-4a95-8833-4977544f0646-tmp\") pod \"klusterlet-addon-workmgr-78b87b794c-w72v9\" (UID: \"91c843a4-a274-4a95-8833-4977544f0646\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78b87b794c-w72v9"
Jul 14 13:28:25.357285 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.357258    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/421a6500-d253-4f95-beb4-db5438d099c9-nginx-conf\") pod \"networking-console-plugin-6cf856758c-nqmzd\" (UID: \"421a6500-d253-4f95-beb4-db5438d099c9\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-nqmzd"
Jul 14 13:28:25.359292 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.359267    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/91c843a4-a274-4a95-8833-4977544f0646-klusterlet-config\") pod \"klusterlet-addon-workmgr-78b87b794c-w72v9\" (UID: \"91c843a4-a274-4a95-8833-4977544f0646\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78b87b794c-w72v9"
Jul 14 13:28:25.359570 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.359546    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/421a6500-d253-4f95-beb4-db5438d099c9-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-nqmzd\" (UID: \"421a6500-d253-4f95-beb4-db5438d099c9\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-nqmzd"
Jul 14 13:28:25.366113 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.366066    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwcgz\" (UniqueName: \"kubernetes.io/projected/91c843a4-a274-4a95-8833-4977544f0646-kube-api-access-qwcgz\") pod \"klusterlet-addon-workmgr-78b87b794c-w72v9\" (UID: \"91c843a4-a274-4a95-8833-4977544f0646\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78b87b794c-w72v9"
Jul 14 13:28:25.456849 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.456821    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ca6401ee-66b4-449f-920d-74ef689c7429-registry-certificates\") pod \"image-registry-59f76b4df9-2glpk\" (UID: \"ca6401ee-66b4-449f-920d-74ef689c7429\") " pod="openshift-image-registry/image-registry-59f76b4df9-2glpk"
Jul 14 13:28:25.456943 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.456861    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ca6401ee-66b4-449f-920d-74ef689c7429-registry-tls\") pod \"image-registry-59f76b4df9-2glpk\" (UID: \"ca6401ee-66b4-449f-920d-74ef689c7429\") " pod="openshift-image-registry/image-registry-59f76b4df9-2glpk"
Jul 14 13:28:25.456943 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.456893    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2xbr\" (UniqueName: \"kubernetes.io/projected/ca6401ee-66b4-449f-920d-74ef689c7429-kube-api-access-j2xbr\") pod \"image-registry-59f76b4df9-2glpk\" (UID: \"ca6401ee-66b4-449f-920d-74ef689c7429\") " pod="openshift-image-registry/image-registry-59f76b4df9-2glpk"
Jul 14 13:28:25.456943 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.456924    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca6401ee-66b4-449f-920d-74ef689c7429-bound-sa-token\") pod \"image-registry-59f76b4df9-2glpk\" (UID: \"ca6401ee-66b4-449f-920d-74ef689c7429\") " pod="openshift-image-registry/image-registry-59f76b4df9-2glpk"
Jul 14 13:28:25.457130 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.456956    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ca6401ee-66b4-449f-920d-74ef689c7429-trusted-ca\") pod \"image-registry-59f76b4df9-2glpk\" (UID: \"ca6401ee-66b4-449f-920d-74ef689c7429\") " pod="openshift-image-registry/image-registry-59f76b4df9-2glpk"
Jul 14 13:28:25.457130 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.456991    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ca6401ee-66b4-449f-920d-74ef689c7429-installation-pull-secrets\") pod \"image-registry-59f76b4df9-2glpk\" (UID: \"ca6401ee-66b4-449f-920d-74ef689c7429\") " pod="openshift-image-registry/image-registry-59f76b4df9-2glpk"
Jul 14 13:28:25.457130 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.457030    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ca6401ee-66b4-449f-920d-74ef689c7429-ca-trust-extracted\") pod \"image-registry-59f76b4df9-2glpk\" (UID: \"ca6401ee-66b4-449f-920d-74ef689c7429\") " pod="openshift-image-registry/image-registry-59f76b4df9-2glpk"
Jul 14 13:28:25.457130 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.457054    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ca6401ee-66b4-449f-920d-74ef689c7429-image-registry-private-configuration\") pod \"image-registry-59f76b4df9-2glpk\" (UID: \"ca6401ee-66b4-449f-920d-74ef689c7429\") " pod="openshift-image-registry/image-registry-59f76b4df9-2glpk"
Jul 14 13:28:25.457802 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.457771    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ca6401ee-66b4-449f-920d-74ef689c7429-registry-certificates\") pod \"image-registry-59f76b4df9-2glpk\" (UID: \"ca6401ee-66b4-449f-920d-74ef689c7429\") " pod="openshift-image-registry/image-registry-59f76b4df9-2glpk"
Jul 14 13:28:25.457996 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.457978    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ca6401ee-66b4-449f-920d-74ef689c7429-ca-trust-extracted\") pod \"image-registry-59f76b4df9-2glpk\" (UID: \"ca6401ee-66b4-449f-920d-74ef689c7429\") " pod="openshift-image-registry/image-registry-59f76b4df9-2glpk"
Jul 14 13:28:25.458320 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.458300    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ca6401ee-66b4-449f-920d-74ef689c7429-trusted-ca\") pod \"image-registry-59f76b4df9-2glpk\" (UID: \"ca6401ee-66b4-449f-920d-74ef689c7429\") " pod="openshift-image-registry/image-registry-59f76b4df9-2glpk"
Jul 14 13:28:25.459913 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.459893    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ca6401ee-66b4-449f-920d-74ef689c7429-registry-tls\") pod \"image-registry-59f76b4df9-2glpk\" (UID: \"ca6401ee-66b4-449f-920d-74ef689c7429\") " pod="openshift-image-registry/image-registry-59f76b4df9-2glpk"
Jul 14 13:28:25.460130 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.460109    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ca6401ee-66b4-449f-920d-74ef689c7429-image-registry-private-configuration\") pod \"image-registry-59f76b4df9-2glpk\" (UID: \"ca6401ee-66b4-449f-920d-74ef689c7429\") " pod="openshift-image-registry/image-registry-59f76b4df9-2glpk"
Jul 14 13:28:25.460773 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.460754    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ca6401ee-66b4-449f-920d-74ef689c7429-installation-pull-secrets\") pod \"image-registry-59f76b4df9-2glpk\" (UID: \"ca6401ee-66b4-449f-920d-74ef689c7429\") " pod="openshift-image-registry/image-registry-59f76b4df9-2glpk"
Jul 14 13:28:25.468836 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.468808    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-s7ft2" event={"ID":"b33bdbd5-63be-4828-a2d6-91d4bead7b80","Type":"ContainerStarted","Data":"3e7f37895a39206d383e3aa5cc23a3d397e327db1d0250c64865b0b692639762"}
Jul 14 13:28:25.470405 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.470381    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca6401ee-66b4-449f-920d-74ef689c7429-bound-sa-token\") pod \"image-registry-59f76b4df9-2glpk\" (UID: \"ca6401ee-66b4-449f-920d-74ef689c7429\") " pod="openshift-image-registry/image-registry-59f76b4df9-2glpk"
Jul 14 13:28:25.472159 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.472141    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2xbr\" (UniqueName: \"kubernetes.io/projected/ca6401ee-66b4-449f-920d-74ef689c7429-kube-api-access-j2xbr\") pod \"image-registry-59f76b4df9-2glpk\" (UID: \"ca6401ee-66b4-449f-920d-74ef689c7429\") " pod="openshift-image-registry/image-registry-59f76b4df9-2glpk"
Jul 14 13:28:25.491290 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.491267    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78b87b794c-w72v9"
Jul 14 13:28:25.504670 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.504648    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-nqmzd"
Jul 14 13:28:25.560863 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.560832    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59f76b4df9-2glpk"
Jul 14 13:28:25.677409 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.677340    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-78b87b794c-w72v9"]
Jul 14 13:28:25.681819 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:28:25.681795    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c843a4_a274_4a95_8833_4977544f0646.slice/crio-ae5e4ca1e5514f9b27cdd56baeb11bbbfa769703ca4535879e02ac05c3a3676c WatchSource:0}: Error finding container ae5e4ca1e5514f9b27cdd56baeb11bbbfa769703ca4535879e02ac05c3a3676c: Status 404 returned error can't find the container with id ae5e4ca1e5514f9b27cdd56baeb11bbbfa769703ca4535879e02ac05c3a3676c
Jul 14 13:28:25.703219 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.703143    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-nqmzd"]
Jul 14 13:28:25.707170 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:28:25.707135    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod421a6500_d253_4f95_beb4_db5438d099c9.slice/crio-d475175be6d4309aad67e37994ccf7070288022dfab6901ecbb8958e0931785a WatchSource:0}: Error finding container d475175be6d4309aad67e37994ccf7070288022dfab6901ecbb8958e0931785a: Status 404 returned error can't find the container with id d475175be6d4309aad67e37994ccf7070288022dfab6901ecbb8958e0931785a
Jul 14 13:28:25.754951 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:25.754926    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59f76b4df9-2glpk"]
Jul 14 13:28:25.759316 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:28:25.759275    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca6401ee_66b4_449f_920d_74ef689c7429.slice/crio-5ee797493e5f3a5dd6f18fcc762a004e66dc6e703ea40821ef4b4ee17068e9ec WatchSource:0}: Error finding container 5ee797493e5f3a5dd6f18fcc762a004e66dc6e703ea40821ef4b4ee17068e9ec: Status 404 returned error can't find the container with id 5ee797493e5f3a5dd6f18fcc762a004e66dc6e703ea40821ef4b4ee17068e9ec
Jul 14 13:28:26.476033 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:26.475665    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59f76b4df9-2glpk" event={"ID":"ca6401ee-66b4-449f-920d-74ef689c7429","Type":"ContainerStarted","Data":"69ed82b711cd290626044ef73d0bef4c8a5b49a778b9d39edeeb0de67b898e5e"}
Jul 14 13:28:26.476033 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:26.475737    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59f76b4df9-2glpk" event={"ID":"ca6401ee-66b4-449f-920d-74ef689c7429","Type":"ContainerStarted","Data":"5ee797493e5f3a5dd6f18fcc762a004e66dc6e703ea40821ef4b4ee17068e9ec"}
Jul 14 13:28:26.476033 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:26.475774    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-59f76b4df9-2glpk"
Jul 14 13:28:26.477218 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:26.477185    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-nqmzd" event={"ID":"421a6500-d253-4f95-beb4-db5438d099c9","Type":"ContainerStarted","Data":"d475175be6d4309aad67e37994ccf7070288022dfab6901ecbb8958e0931785a"}
Jul 14 13:28:26.478509 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:26.478468    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78b87b794c-w72v9" event={"ID":"91c843a4-a274-4a95-8833-4977544f0646","Type":"ContainerStarted","Data":"ae5e4ca1e5514f9b27cdd56baeb11bbbfa769703ca4535879e02ac05c3a3676c"}
Jul 14 13:28:26.522797 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:26.522749    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-s7ft2" podStartSLOduration=66.501501052 podStartE2EDuration="1m11.522734401s" totalImagesPullingTime="5.021233349s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 13:27:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:28:26.522240702 +0000 UTC m=+71.848560634" watchObservedRunningTime="2026-07-14 13:28:26.522734401 +0000 UTC m=+71.849054329"
Jul 14 13:28:26.523221 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:26.523196    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-59f76b4df9-2glpk" podStartSLOduration=1.523188521 podStartE2EDuration="1.523188521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:28:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:28:26.498745349 +0000 UTC m=+71.825065302" watchObservedRunningTime="2026-07-14 13:28:26.523188521 +0000 UTC m=+71.849508449"
Jul 14 13:28:27.486815 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:27.486191    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lkjjk" event={"ID":"b89a211f-60b9-4eab-84cb-7aad1dcd94b1","Type":"ContainerStarted","Data":"f32fd3c70f1838d6586137d77a9248a106852864bad739abfe0179d5a24eff37"}
Jul 14 13:28:27.486815 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:27.486247    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lkjjk" event={"ID":"b89a211f-60b9-4eab-84cb-7aad1dcd94b1","Type":"ContainerStarted","Data":"bc274af6b54945852c6701acf12a0930e224cbc0c9256cfadfe21fd55192db63"}
Jul 14 13:28:27.486815 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:27.486448    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lkjjk"
Jul 14 13:28:27.515262 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:27.515213    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lkjjk" podStartSLOduration=34.30914612 podStartE2EDuration="37.515192348s" totalImagesPullingTime="3.206046228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:27:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:28:27.512645462 +0000 UTC m=+72.838965436" watchObservedRunningTime="2026-07-14 13:28:27.515192348 +0000 UTC m=+72.841512278"
Jul 14 13:28:28.491896 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:28.491861    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-nqmzd" event={"ID":"421a6500-d253-4f95-beb4-db5438d099c9","Type":"ContainerStarted","Data":"3118d95e02341b1b182528757bf21e588fbb23e8222b3d03a077a0d142f68431"}
Jul 14 13:28:28.511765 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:28.511713    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-nqmzd" podStartSLOduration=1.642543417 podStartE2EDuration="3.511697182s" totalImagesPullingTime="1.869153765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:28:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:28:28.5105902 +0000 UTC m=+73.836910130" watchObservedRunningTime="2026-07-14 13:28:28.511697182 +0000 UTC m=+73.838017087"
Jul 14 13:28:29.246355 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:29.246325    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74cc4b7574-k9z2v"]
Jul 14 13:28:29.248425 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:29.248404    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74cc4b7574-k9z2v"
Jul 14 13:28:29.251047 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:29.251005    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 14 13:28:29.252164 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:29.252136    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 14 13:28:29.252262 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:29.252162    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 14 13:28:29.252431 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:29.252408    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 14 13:28:29.252431 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:29.252410    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 14 13:28:29.252580 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:29.252412    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 14 13:28:29.252580 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:29.252495    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 14 13:28:29.252746 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:29.252729    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-xz9wp\""
Jul 14 13:28:29.261837 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:29.261816    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74cc4b7574-k9z2v"]
Jul 14 13:28:29.290573 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:29.290551    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1138d77-ce78-41d4-9656-02843d9b24b3-console-config\") pod \"console-74cc4b7574-k9z2v\" (UID: \"a1138d77-ce78-41d4-9656-02843d9b24b3\") " pod="openshift-console/console-74cc4b7574-k9z2v"
Jul 14 13:28:29.290672 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:29.290587    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1138d77-ce78-41d4-9656-02843d9b24b3-console-oauth-config\") pod \"console-74cc4b7574-k9z2v\" (UID: \"a1138d77-ce78-41d4-9656-02843d9b24b3\") " pod="openshift-console/console-74cc4b7574-k9z2v"
Jul 14 13:28:29.290672 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:29.290617    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1138d77-ce78-41d4-9656-02843d9b24b3-oauth-serving-cert\") pod \"console-74cc4b7574-k9z2v\" (UID: \"a1138d77-ce78-41d4-9656-02843d9b24b3\") " pod="openshift-console/console-74cc4b7574-k9z2v"
Jul 14 13:28:29.290672 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:29.290649    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9q2r\" (UniqueName: \"kubernetes.io/projected/a1138d77-ce78-41d4-9656-02843d9b24b3-kube-api-access-r9q2r\") pod \"console-74cc4b7574-k9z2v\" (UID: \"a1138d77-ce78-41d4-9656-02843d9b24b3\") " pod="openshift-console/console-74cc4b7574-k9z2v"
Jul 14 13:28:29.290832 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:29.290675    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1138d77-ce78-41d4-9656-02843d9b24b3-service-ca\") pod \"console-74cc4b7574-k9z2v\" (UID: \"a1138d77-ce78-41d4-9656-02843d9b24b3\") " pod="openshift-console/console-74cc4b7574-k9z2v"
Jul 14 13:28:29.290832 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:29.290766    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1138d77-ce78-41d4-9656-02843d9b24b3-console-serving-cert\") pod \"console-74cc4b7574-k9z2v\" (UID: \"a1138d77-ce78-41d4-9656-02843d9b24b3\") " pod="openshift-console/console-74cc4b7574-k9z2v"
Jul 14 13:28:29.391560 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:29.391522    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1138d77-ce78-41d4-9656-02843d9b24b3-console-config\") pod \"console-74cc4b7574-k9z2v\" (UID: \"a1138d77-ce78-41d4-9656-02843d9b24b3\") " pod="openshift-console/console-74cc4b7574-k9z2v"
Jul 14 13:28:29.391698 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:29.391565    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1138d77-ce78-41d4-9656-02843d9b24b3-console-oauth-config\") pod \"console-74cc4b7574-k9z2v\" (UID: \"a1138d77-ce78-41d4-9656-02843d9b24b3\") " pod="openshift-console/console-74cc4b7574-k9z2v"
Jul 14 13:28:29.391698 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:29.391595    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1138d77-ce78-41d4-9656-02843d9b24b3-oauth-serving-cert\") pod \"console-74cc4b7574-k9z2v\" (UID: \"a1138d77-ce78-41d4-9656-02843d9b24b3\") " pod="openshift-console/console-74cc4b7574-k9z2v"
Jul 14 13:28:29.391698 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:29.391624    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9q2r\" (UniqueName: \"kubernetes.io/projected/a1138d77-ce78-41d4-9656-02843d9b24b3-kube-api-access-r9q2r\") pod \"console-74cc4b7574-k9z2v\" (UID: \"a1138d77-ce78-41d4-9656-02843d9b24b3\") " pod="openshift-console/console-74cc4b7574-k9z2v"
Jul 14 13:28:29.391698 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:29.391650    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1138d77-ce78-41d4-9656-02843d9b24b3-service-ca\") pod \"console-74cc4b7574-k9z2v\" (UID: \"a1138d77-ce78-41d4-9656-02843d9b24b3\") " pod="openshift-console/console-74cc4b7574-k9z2v"
Jul 14 13:28:29.391874 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:29.391727    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1138d77-ce78-41d4-9656-02843d9b24b3-console-serving-cert\") pod \"console-74cc4b7574-k9z2v\" (UID: \"a1138d77-ce78-41d4-9656-02843d9b24b3\") " pod="openshift-console/console-74cc4b7574-k9z2v"
Jul 14 13:28:29.392325 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:29.392295    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1138d77-ce78-41d4-9656-02843d9b24b3-console-config\") pod \"console-74cc4b7574-k9z2v\" (UID: \"a1138d77-ce78-41d4-9656-02843d9b24b3\") " pod="openshift-console/console-74cc4b7574-k9z2v"
Jul 14 13:28:29.392640 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:29.392611    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1138d77-ce78-41d4-9656-02843d9b24b3-oauth-serving-cert\") pod \"console-74cc4b7574-k9z2v\" (UID: \"a1138d77-ce78-41d4-9656-02843d9b24b3\") " pod="openshift-console/console-74cc4b7574-k9z2v"
Jul 14 13:28:29.392733 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:29.392622    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1138d77-ce78-41d4-9656-02843d9b24b3-service-ca\") pod \"console-74cc4b7574-k9z2v\" (UID: \"a1138d77-ce78-41d4-9656-02843d9b24b3\") " pod="openshift-console/console-74cc4b7574-k9z2v"
Jul 14 13:28:29.394388 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:29.394367    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1138d77-ce78-41d4-9656-02843d9b24b3-console-oauth-config\") pod \"console-74cc4b7574-k9z2v\" (UID: \"a1138d77-ce78-41d4-9656-02843d9b24b3\") " pod="openshift-console/console-74cc4b7574-k9z2v"
Jul 14 13:28:29.394504 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:29.394481    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1138d77-ce78-41d4-9656-02843d9b24b3-console-serving-cert\") pod \"console-74cc4b7574-k9z2v\" (UID: \"a1138d77-ce78-41d4-9656-02843d9b24b3\") " pod="openshift-console/console-74cc4b7574-k9z2v"
Jul 14 13:28:29.402205 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:29.402187    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9q2r\" (UniqueName: \"kubernetes.io/projected/a1138d77-ce78-41d4-9656-02843d9b24b3-kube-api-access-r9q2r\") pod \"console-74cc4b7574-k9z2v\" (UID: \"a1138d77-ce78-41d4-9656-02843d9b24b3\") " pod="openshift-console/console-74cc4b7574-k9z2v"
Jul 14 13:28:29.559231 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:29.559138    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74cc4b7574-k9z2v"
Jul 14 13:28:29.808502 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:29.808479    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74cc4b7574-k9z2v"]
Jul 14 13:28:29.811257 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:28:29.811224    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1138d77_ce78_41d4_9656_02843d9b24b3.slice/crio-2c31efcb51e27ea852d3863d2f299479751fa471604374140fdf2d53944964da WatchSource:0}: Error finding container 2c31efcb51e27ea852d3863d2f299479751fa471604374140fdf2d53944964da: Status 404 returned error can't find the container with id 2c31efcb51e27ea852d3863d2f299479751fa471604374140fdf2d53944964da
Jul 14 13:28:30.500200 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:30.500169    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74cc4b7574-k9z2v" event={"ID":"a1138d77-ce78-41d4-9656-02843d9b24b3","Type":"ContainerStarted","Data":"2c31efcb51e27ea852d3863d2f299479751fa471604374140fdf2d53944964da"}
Jul 14 13:28:30.501882 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:30.501830    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78b87b794c-w72v9" event={"ID":"91c843a4-a274-4a95-8833-4977544f0646","Type":"ContainerStarted","Data":"c19ef3a91f9e00910af419a8f43780faa4ef6c39c4cad0a03a6cd7d7d7965ce1"}
Jul 14 13:28:30.502065 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:30.502044    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78b87b794c-w72v9"
Jul 14 13:28:30.504050 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:30.504027    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78b87b794c-w72v9"
Jul 14 13:28:30.522373 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:30.522327    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78b87b794c-w72v9" podStartSLOduration=1.4767827 podStartE2EDuration="5.522307806s" totalImagesPullingTime="4.045525106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:28:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:28:30.521791913 +0000 UTC m=+75.848111842" watchObservedRunningTime="2026-07-14 13:28:30.522307806 +0000 UTC m=+75.848627734"
Jul 14 13:28:31.757042 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:31.757001    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-bshjk"]
Jul 14 13:28:31.760578 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:31.760557    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-bshjk"
Jul 14 13:28:31.763876 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:31.763729    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 14 13:28:31.764996 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:31.764927    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 14 13:28:31.764996 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:31.764964    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 14 13:28:31.764996 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:31.764999    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 14 13:28:31.765291 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:31.765281    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 14 13:28:31.765411 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:31.765394    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-bt4hj\""
Jul 14 13:28:31.774244 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:31.774223    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-bshjk"]
Jul 14 13:28:31.812461 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:31.812425    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec47e455-0f3c-4b63-bf5d-4481866117eb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-bshjk\" (UID: \"ec47e455-0f3c-4b63-bf5d-4481866117eb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bshjk"
Jul 14 13:28:31.812590 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:31.812470    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec47e455-0f3c-4b63-bf5d-4481866117eb-metrics-client-ca\") pod \"prometheus-operator-64785b6675-bshjk\" (UID: \"ec47e455-0f3c-4b63-bf5d-4481866117eb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bshjk"
Jul 14 13:28:31.812590 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:31.812495    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ec47e455-0f3c-4b63-bf5d-4481866117eb-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-bshjk\" (UID: \"ec47e455-0f3c-4b63-bf5d-4481866117eb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bshjk"
Jul 14 13:28:31.812685 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:31.812602    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5brdj\" (UniqueName: \"kubernetes.io/projected/ec47e455-0f3c-4b63-bf5d-4481866117eb-kube-api-access-5brdj\") pod \"prometheus-operator-64785b6675-bshjk\" (UID: \"ec47e455-0f3c-4b63-bf5d-4481866117eb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bshjk"
Jul 14 13:28:31.913719 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:31.913691    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec47e455-0f3c-4b63-bf5d-4481866117eb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-bshjk\" (UID: \"ec47e455-0f3c-4b63-bf5d-4481866117eb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bshjk"
Jul 14 13:28:31.913845 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:31.913730    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec47e455-0f3c-4b63-bf5d-4481866117eb-metrics-client-ca\") pod \"prometheus-operator-64785b6675-bshjk\" (UID: \"ec47e455-0f3c-4b63-bf5d-4481866117eb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bshjk"
Jul 14 13:28:31.913845 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:31.913800    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ec47e455-0f3c-4b63-bf5d-4481866117eb-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-bshjk\" (UID: \"ec47e455-0f3c-4b63-bf5d-4481866117eb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bshjk"
Jul 14 13:28:31.913926 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:31.913849    2569 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 14 13:28:31.913926 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:31.913878    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5brdj\" (UniqueName: \"kubernetes.io/projected/ec47e455-0f3c-4b63-bf5d-4481866117eb-kube-api-access-5brdj\") pod \"prometheus-operator-64785b6675-bshjk\" (UID: \"ec47e455-0f3c-4b63-bf5d-4481866117eb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bshjk"
Jul 14 13:28:31.913926 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:31.913916    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec47e455-0f3c-4b63-bf5d-4481866117eb-prometheus-operator-tls podName:ec47e455-0f3c-4b63-bf5d-4481866117eb nodeName:}" failed. No retries permitted until 2026-07-14 13:28:32.41389682 +0000 UTC m=+77.740216749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/ec47e455-0f3c-4b63-bf5d-4481866117eb-prometheus-operator-tls") pod "prometheus-operator-64785b6675-bshjk" (UID: "ec47e455-0f3c-4b63-bf5d-4481866117eb") : secret "prometheus-operator-tls" not found
Jul 14 13:28:31.914590 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:31.914563    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec47e455-0f3c-4b63-bf5d-4481866117eb-metrics-client-ca\") pod \"prometheus-operator-64785b6675-bshjk\" (UID: \"ec47e455-0f3c-4b63-bf5d-4481866117eb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bshjk"
Jul 14 13:28:31.916323 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:31.916303    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec47e455-0f3c-4b63-bf5d-4481866117eb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-bshjk\" (UID: \"ec47e455-0f3c-4b63-bf5d-4481866117eb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bshjk"
Jul 14 13:28:31.926424 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:31.926388    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5brdj\" (UniqueName: \"kubernetes.io/projected/ec47e455-0f3c-4b63-bf5d-4481866117eb-kube-api-access-5brdj\") pod \"prometheus-operator-64785b6675-bshjk\" (UID: \"ec47e455-0f3c-4b63-bf5d-4481866117eb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bshjk"
Jul 14 13:28:32.419265 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:32.419218    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ec47e455-0f3c-4b63-bf5d-4481866117eb-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-bshjk\" (UID: \"ec47e455-0f3c-4b63-bf5d-4481866117eb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bshjk"
Jul 14 13:28:32.421956 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:32.421928    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ec47e455-0f3c-4b63-bf5d-4481866117eb-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-bshjk\" (UID: \"ec47e455-0f3c-4b63-bf5d-4481866117eb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bshjk"
Jul 14 13:28:32.520057 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:32.520026    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xz2lp\" (UID: \"c650b1a4-6066-4d12-b4cb-0217c8f4ebfe\") " pod="openshift-insights/insights-runtime-extractor-xz2lp"
Jul 14 13:28:32.522606 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:32.522563    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c650b1a4-6066-4d12-b4cb-0217c8f4ebfe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xz2lp\" (UID: \"c650b1a4-6066-4d12-b4cb-0217c8f4ebfe\") " pod="openshift-insights/insights-runtime-extractor-xz2lp"
Jul 14 13:28:32.670922 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:32.670846    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-bshjk"
Jul 14 13:28:32.690668 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:32.690644    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tbwpd\""
Jul 14 13:28:32.699048 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:32.699005    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xz2lp"
Jul 14 13:28:32.919364 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:32.919343    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-bshjk"]
Jul 14 13:28:32.921120 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:28:32.921061    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec47e455_0f3c_4b63_bf5d_4481866117eb.slice/crio-70e8e5bd398cbd7d64131d8c46810f3216fee788bfc8d63cd6b6ba6a6cfcc65f WatchSource:0}: Error finding container 70e8e5bd398cbd7d64131d8c46810f3216fee788bfc8d63cd6b6ba6a6cfcc65f: Status 404 returned error can't find the container with id 70e8e5bd398cbd7d64131d8c46810f3216fee788bfc8d63cd6b6ba6a6cfcc65f
Jul 14 13:28:32.939649 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:32.938310    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xz2lp"]
Jul 14 13:28:33.517076 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:33.516983    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-bshjk" event={"ID":"ec47e455-0f3c-4b63-bf5d-4481866117eb","Type":"ContainerStarted","Data":"70e8e5bd398cbd7d64131d8c46810f3216fee788bfc8d63cd6b6ba6a6cfcc65f"}
Jul 14 13:28:33.520442 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:33.520414    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74cc4b7574-k9z2v" event={"ID":"a1138d77-ce78-41d4-9656-02843d9b24b3","Type":"ContainerStarted","Data":"6878ef37fe44c4bc7fc916b312967a9481c6eca5aaf1e999af3477198b85d643"}
Jul 14 13:28:33.523567 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:33.523534    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xz2lp" event={"ID":"c650b1a4-6066-4d12-b4cb-0217c8f4ebfe","Type":"ContainerStarted","Data":"5febc0ddf8546dcc00e1f063fc5a331ed88d00c31af9c7100cac547ae034e890"}
Jul 14 13:28:33.544975 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:33.544784    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74cc4b7574-k9z2v" podStartSLOduration=1.533004342 podStartE2EDuration="4.544764303s" totalImagesPullingTime="3.011759961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:28:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:28:33.543906565 +0000 UTC m=+78.870226496" watchObservedRunningTime="2026-07-14 13:28:33.544764303 +0000 UTC m=+78.871084232"
Jul 14 13:28:35.205429 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:35.205376    2569 patch_prober.go:28] interesting pod/image-registry-584b976c99-8blm8 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 14 13:28:35.205906 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:35.205440    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-584b976c99-8blm8" podUID="d137293e-b342-4b7e-938a-bc40f150ae6b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 14 13:28:35.534943 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:35.534903    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-bshjk" event={"ID":"ec47e455-0f3c-4b63-bf5d-4481866117eb","Type":"ContainerStarted","Data":"515c8662b5dc315e8a757685b2e728f78bfe29decc75a213e8bcca78dbe2c528"}
Jul 14 13:28:37.493788 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:37.493767    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lkjjk"
Jul 14 13:28:37.545330 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:37.545297    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-bshjk" event={"ID":"ec47e455-0f3c-4b63-bf5d-4481866117eb","Type":"ContainerStarted","Data":"45f7b741325397e950e92f5b5be6be15ef6a9feb98277be59979663b0482c161"}
Jul 14 13:28:37.547236 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:37.547185    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xz2lp" event={"ID":"c650b1a4-6066-4d12-b4cb-0217c8f4ebfe","Type":"ContainerStarted","Data":"76c1f5cde00cd3c510d36a3f649fe27e6aedec26bba0c97ebfa985b9edef690f"}
Jul 14 13:28:37.547340 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:37.547259    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xz2lp" event={"ID":"c650b1a4-6066-4d12-b4cb-0217c8f4ebfe","Type":"ContainerStarted","Data":"9d18ecd6e78f137252a6fc032158324f0669946eb9b8d699c863c482d62125b9"}
Jul 14 13:28:37.569706 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:37.569660    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-bshjk" podStartSLOduration=4.993227872 podStartE2EDuration="6.569647314s" totalImagesPullingTime="1.576419442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:28:37.569346526 +0000 UTC m=+82.895666455" watchObservedRunningTime="2026-07-14 13:28:37.569647314 +0000 UTC m=+82.895967220"
Jul 14 13:28:38.041668 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:38.041637    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8c6d96db7-ztdq9"]
Jul 14 13:28:38.044802 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:38.044781    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8c6d96db7-ztdq9"
Jul 14 13:28:38.056255 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:38.056232    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 14 13:28:38.057324 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:38.057303    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8c6d96db7-ztdq9"]
Jul 14 13:28:38.164985 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:38.164960    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/909c52b2-92e9-43e0-b2f7-1ba042a9311b-console-serving-cert\") pod \"console-8c6d96db7-ztdq9\" (UID: \"909c52b2-92e9-43e0-b2f7-1ba042a9311b\") " pod="openshift-console/console-8c6d96db7-ztdq9"
Jul 14 13:28:38.165159 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:38.164999    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/909c52b2-92e9-43e0-b2f7-1ba042a9311b-console-config\") pod \"console-8c6d96db7-ztdq9\" (UID: \"909c52b2-92e9-43e0-b2f7-1ba042a9311b\") " pod="openshift-console/console-8c6d96db7-ztdq9"
Jul 14 13:28:38.165159 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:38.165046    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/909c52b2-92e9-43e0-b2f7-1ba042a9311b-console-oauth-config\") pod \"console-8c6d96db7-ztdq9\" (UID: \"909c52b2-92e9-43e0-b2f7-1ba042a9311b\") " pod="openshift-console/console-8c6d96db7-ztdq9"
Jul 14 13:28:38.165159 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:38.165102    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/909c52b2-92e9-43e0-b2f7-1ba042a9311b-service-ca\") pod \"console-8c6d96db7-ztdq9\" (UID: \"909c52b2-92e9-43e0-b2f7-1ba042a9311b\") " pod="openshift-console/console-8c6d96db7-ztdq9"
Jul 14 13:28:38.165290 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:38.165161    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/909c52b2-92e9-43e0-b2f7-1ba042a9311b-trusted-ca-bundle\") pod \"console-8c6d96db7-ztdq9\" (UID: \"909c52b2-92e9-43e0-b2f7-1ba042a9311b\") " pod="openshift-console/console-8c6d96db7-ztdq9"
Jul 14 13:28:38.165290 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:38.165190    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/909c52b2-92e9-43e0-b2f7-1ba042a9311b-oauth-serving-cert\") pod \"console-8c6d96db7-ztdq9\" (UID: \"909c52b2-92e9-43e0-b2f7-1ba042a9311b\") " pod="openshift-console/console-8c6d96db7-ztdq9"
Jul 14 13:28:38.165290 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:38.165271    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfzzt\" (UniqueName: \"kubernetes.io/projected/909c52b2-92e9-43e0-b2f7-1ba042a9311b-kube-api-access-sfzzt\") pod \"console-8c6d96db7-ztdq9\" (UID: \"909c52b2-92e9-43e0-b2f7-1ba042a9311b\") " pod="openshift-console/console-8c6d96db7-ztdq9"
Jul 14 13:28:38.265607 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:38.265577    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfzzt\" (UniqueName: \"kubernetes.io/projected/909c52b2-92e9-43e0-b2f7-1ba042a9311b-kube-api-access-sfzzt\") pod \"console-8c6d96db7-ztdq9\" (UID: \"909c52b2-92e9-43e0-b2f7-1ba042a9311b\") " pod="openshift-console/console-8c6d96db7-ztdq9"
Jul 14 13:28:38.265767 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:38.265613    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/909c52b2-92e9-43e0-b2f7-1ba042a9311b-console-serving-cert\") pod \"console-8c6d96db7-ztdq9\" (UID: \"909c52b2-92e9-43e0-b2f7-1ba042a9311b\") " pod="openshift-console/console-8c6d96db7-ztdq9"
Jul 14 13:28:38.265767 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:38.265639    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/909c52b2-92e9-43e0-b2f7-1ba042a9311b-console-config\") pod \"console-8c6d96db7-ztdq9\" (UID: \"909c52b2-92e9-43e0-b2f7-1ba042a9311b\") " pod="openshift-console/console-8c6d96db7-ztdq9"
Jul 14 13:28:38.265767 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:38.265662    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/909c52b2-92e9-43e0-b2f7-1ba042a9311b-console-oauth-config\") pod \"console-8c6d96db7-ztdq9\" (UID: \"909c52b2-92e9-43e0-b2f7-1ba042a9311b\") " pod="openshift-console/console-8c6d96db7-ztdq9"
Jul 14 13:28:38.265767 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:38.265687    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/909c52b2-92e9-43e0-b2f7-1ba042a9311b-service-ca\") pod \"console-8c6d96db7-ztdq9\" (UID: \"909c52b2-92e9-43e0-b2f7-1ba042a9311b\") " pod="openshift-console/console-8c6d96db7-ztdq9"
Jul 14 13:28:38.265767 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:38.265716    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/909c52b2-92e9-43e0-b2f7-1ba042a9311b-trusted-ca-bundle\") pod \"console-8c6d96db7-ztdq9\" (UID: \"909c52b2-92e9-43e0-b2f7-1ba042a9311b\") " pod="openshift-console/console-8c6d96db7-ztdq9"
Jul 14 13:28:38.265767 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:38.265743    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/909c52b2-92e9-43e0-b2f7-1ba042a9311b-oauth-serving-cert\") pod \"console-8c6d96db7-ztdq9\" (UID: \"909c52b2-92e9-43e0-b2f7-1ba042a9311b\") " pod="openshift-console/console-8c6d96db7-ztdq9"
Jul 14 13:28:38.266499 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:38.266469    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/909c52b2-92e9-43e0-b2f7-1ba042a9311b-oauth-serving-cert\") pod \"console-8c6d96db7-ztdq9\" (UID: \"909c52b2-92e9-43e0-b2f7-1ba042a9311b\") " pod="openshift-console/console-8c6d96db7-ztdq9"
Jul 14 13:28:38.266615 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:38.266508    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/909c52b2-92e9-43e0-b2f7-1ba042a9311b-console-config\") pod \"console-8c6d96db7-ztdq9\" (UID: \"909c52b2-92e9-43e0-b2f7-1ba042a9311b\") " pod="openshift-console/console-8c6d96db7-ztdq9"
Jul 14 13:28:38.266615 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:38.266591    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/909c52b2-92e9-43e0-b2f7-1ba042a9311b-service-ca\") pod \"console-8c6d96db7-ztdq9\" (UID: \"909c52b2-92e9-43e0-b2f7-1ba042a9311b\") " pod="openshift-console/console-8c6d96db7-ztdq9"
Jul 14 13:28:38.266910 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:38.266885    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/909c52b2-92e9-43e0-b2f7-1ba042a9311b-trusted-ca-bundle\") pod \"console-8c6d96db7-ztdq9\" (UID: \"909c52b2-92e9-43e0-b2f7-1ba042a9311b\") " pod="openshift-console/console-8c6d96db7-ztdq9"
Jul 14 13:28:38.268376 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:38.268357    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/909c52b2-92e9-43e0-b2f7-1ba042a9311b-console-serving-cert\") pod \"console-8c6d96db7-ztdq9\" (UID: \"909c52b2-92e9-43e0-b2f7-1ba042a9311b\") " pod="openshift-console/console-8c6d96db7-ztdq9"
Jul 14 13:28:38.268534 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:38.268513    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/909c52b2-92e9-43e0-b2f7-1ba042a9311b-console-oauth-config\") pod \"console-8c6d96db7-ztdq9\" (UID: \"909c52b2-92e9-43e0-b2f7-1ba042a9311b\") " pod="openshift-console/console-8c6d96db7-ztdq9"
Jul 14 13:28:38.276523 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:38.276490    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfzzt\" (UniqueName: \"kubernetes.io/projected/909c52b2-92e9-43e0-b2f7-1ba042a9311b-kube-api-access-sfzzt\") pod \"console-8c6d96db7-ztdq9\" (UID: \"909c52b2-92e9-43e0-b2f7-1ba042a9311b\") " pod="openshift-console/console-8c6d96db7-ztdq9"
Jul 14 13:28:38.355468 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:38.355383    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8c6d96db7-ztdq9"
Jul 14 13:28:38.500610 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:38.500581    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8c6d96db7-ztdq9"]
Jul 14 13:28:38.502730 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:28:38.502702    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod909c52b2_92e9_43e0_b2f7_1ba042a9311b.slice/crio-a57d00a4ab592a4999d2f279bc8e28a30e8e2b06674e953d155a0d6d0d4d7283 WatchSource:0}: Error finding container a57d00a4ab592a4999d2f279bc8e28a30e8e2b06674e953d155a0d6d0d4d7283: Status 404 returned error can't find the container with id a57d00a4ab592a4999d2f279bc8e28a30e8e2b06674e953d155a0d6d0d4d7283
Jul 14 13:28:38.552382 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:38.552345    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8c6d96db7-ztdq9" event={"ID":"909c52b2-92e9-43e0-b2f7-1ba042a9311b","Type":"ContainerStarted","Data":"a57d00a4ab592a4999d2f279bc8e28a30e8e2b06674e953d155a0d6d0d4d7283"}
Jul 14 13:28:39.139153 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.139121    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-bfqm5"]
Jul 14 13:28:39.143840 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.143816    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-bfqm5"
Jul 14 13:28:39.147071 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.147028    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 14 13:28:39.147489 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.147468    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-2b8zg\""
Jul 14 13:28:39.147608 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.147556    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 14 13:28:39.148526 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.148505    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 14 13:28:39.157105 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.157082    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-bfqm5"]
Jul 14 13:28:39.158324 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.158209    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ltxjs"]
Jul 14 13:28:39.161538 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.161520    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ltxjs"
Jul 14 13:28:39.164996 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.164974    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 14 13:28:39.164996 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.164987    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 14 13:28:39.165172 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.164974    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 14 13:28:39.165722 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.165700    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zg27v\""
Jul 14 13:28:39.275432 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.275396    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/34510396-e510-4255-aab4-724af10b00a4-node-exporter-tls\") pod \"node-exporter-ltxjs\" (UID: \"34510396-e510-4255-aab4-724af10b00a4\") " pod="openshift-monitoring/node-exporter-ltxjs"
Jul 14 13:28:39.275598 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.275452    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34510396-e510-4255-aab4-724af10b00a4-metrics-client-ca\") pod \"node-exporter-ltxjs\" (UID: \"34510396-e510-4255-aab4-724af10b00a4\") " pod="openshift-monitoring/node-exporter-ltxjs"
Jul 14 13:28:39.275598 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.275479    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/34510396-e510-4255-aab4-724af10b00a4-root\") pod \"node-exporter-ltxjs\" (UID: \"34510396-e510-4255-aab4-724af10b00a4\") " pod="openshift-monitoring/node-exporter-ltxjs"
Jul 14 13:28:39.275598 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.275497    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fff04318-1a39-4b84-a5c5-d53129f9e071-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-bfqm5\" (UID: \"fff04318-1a39-4b84-a5c5-d53129f9e071\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bfqm5"
Jul 14 13:28:39.275598 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.275519    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/34510396-e510-4255-aab4-724af10b00a4-node-exporter-wtmp\") pod \"node-exporter-ltxjs\" (UID: \"34510396-e510-4255-aab4-724af10b00a4\") " pod="openshift-monitoring/node-exporter-ltxjs"
Jul 14 13:28:39.275598 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.275551    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fff04318-1a39-4b84-a5c5-d53129f9e071-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-bfqm5\" (UID: \"fff04318-1a39-4b84-a5c5-d53129f9e071\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bfqm5"
Jul 14 13:28:39.275839 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.275611    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34510396-e510-4255-aab4-724af10b00a4-sys\") pod \"node-exporter-ltxjs\" (UID: \"34510396-e510-4255-aab4-724af10b00a4\") " pod="openshift-monitoring/node-exporter-ltxjs"
Jul 14 13:28:39.275839 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.275642    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf6dx\" (UniqueName: \"kubernetes.io/projected/34510396-e510-4255-aab4-724af10b00a4-kube-api-access-kf6dx\") pod \"node-exporter-ltxjs\" (UID: \"34510396-e510-4255-aab4-724af10b00a4\") " pod="openshift-monitoring/node-exporter-ltxjs"
Jul 14 13:28:39.275839 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.275655    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fff04318-1a39-4b84-a5c5-d53129f9e071-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-bfqm5\" (UID: \"fff04318-1a39-4b84-a5c5-d53129f9e071\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bfqm5"
Jul 14 13:28:39.275839 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.275669    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8cs9\" (UniqueName: \"kubernetes.io/projected/fff04318-1a39-4b84-a5c5-d53129f9e071-kube-api-access-c8cs9\") pod \"kube-state-metrics-785778ff74-bfqm5\" (UID: \"fff04318-1a39-4b84-a5c5-d53129f9e071\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bfqm5"
Jul 14 13:28:39.275839 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.275717    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/34510396-e510-4255-aab4-724af10b00a4-node-exporter-textfile\") pod \"node-exporter-ltxjs\" (UID: \"34510396-e510-4255-aab4-724af10b00a4\") " pod="openshift-monitoring/node-exporter-ltxjs"
Jul 14 13:28:39.275839 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.275763    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fff04318-1a39-4b84-a5c5-d53129f9e071-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-bfqm5\" (UID: \"fff04318-1a39-4b84-a5c5-d53129f9e071\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bfqm5"
Jul 14 13:28:39.275839 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.275795    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/34510396-e510-4255-aab4-724af10b00a4-node-exporter-accelerators-collector-config\") pod \"node-exporter-ltxjs\" (UID: \"34510396-e510-4255-aab4-724af10b00a4\") " pod="openshift-monitoring/node-exporter-ltxjs"
Jul 14 13:28:39.276110 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.275892    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fff04318-1a39-4b84-a5c5-d53129f9e071-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-bfqm5\" (UID: \"fff04318-1a39-4b84-a5c5-d53129f9e071\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bfqm5"
Jul 14 13:28:39.276110 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.275943    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/34510396-e510-4255-aab4-724af10b00a4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ltxjs\" (UID: \"34510396-e510-4255-aab4-724af10b00a4\") " pod="openshift-monitoring/node-exporter-ltxjs"
Jul 14 13:28:39.377330 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.377300    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fff04318-1a39-4b84-a5c5-d53129f9e071-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-bfqm5\" (UID: \"fff04318-1a39-4b84-a5c5-d53129f9e071\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bfqm5"
Jul 14 13:28:39.377503 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.377345    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/34510396-e510-4255-aab4-724af10b00a4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ltxjs\" (UID: \"34510396-e510-4255-aab4-724af10b00a4\") " pod="openshift-monitoring/node-exporter-ltxjs"
Jul 14 13:28:39.377503 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.377380    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/34510396-e510-4255-aab4-724af10b00a4-node-exporter-tls\") pod \"node-exporter-ltxjs\" (UID: \"34510396-e510-4255-aab4-724af10b00a4\") " pod="openshift-monitoring/node-exporter-ltxjs"
Jul 14 13:28:39.377503 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.377407    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34510396-e510-4255-aab4-724af10b00a4-metrics-client-ca\") pod \"node-exporter-ltxjs\" (UID: \"34510396-e510-4255-aab4-724af10b00a4\") " pod="openshift-monitoring/node-exporter-ltxjs"
Jul 14 13:28:39.377503 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.377433    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/34510396-e510-4255-aab4-724af10b00a4-root\") pod \"node-exporter-ltxjs\" (UID: \"34510396-e510-4255-aab4-724af10b00a4\") " pod="openshift-monitoring/node-exporter-ltxjs"
Jul 14 13:28:39.377503 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.377456    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fff04318-1a39-4b84-a5c5-d53129f9e071-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-bfqm5\" (UID: \"fff04318-1a39-4b84-a5c5-d53129f9e071\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bfqm5"
Jul 14 13:28:39.377503 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.377487    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/34510396-e510-4255-aab4-724af10b00a4-node-exporter-wtmp\") pod \"node-exporter-ltxjs\" (UID: \"34510396-e510-4255-aab4-724af10b00a4\") " pod="openshift-monitoring/node-exporter-ltxjs"
Jul 14 13:28:39.377831 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.377519    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fff04318-1a39-4b84-a5c5-d53129f9e071-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-bfqm5\" (UID: \"fff04318-1a39-4b84-a5c5-d53129f9e071\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bfqm5"
Jul 14 13:28:39.377831 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.377546    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34510396-e510-4255-aab4-724af10b00a4-sys\") pod \"node-exporter-ltxjs\" (UID: \"34510396-e510-4255-aab4-724af10b00a4\") " pod="openshift-monitoring/node-exporter-ltxjs"
Jul 14 13:28:39.377831 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.377541    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/34510396-e510-4255-aab4-724af10b00a4-root\") pod \"node-exporter-ltxjs\" (UID: \"34510396-e510-4255-aab4-724af10b00a4\") " pod="openshift-monitoring/node-exporter-ltxjs"
Jul 14 13:28:39.377831 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.377604    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf6dx\" (UniqueName: \"kubernetes.io/projected/34510396-e510-4255-aab4-724af10b00a4-kube-api-access-kf6dx\") pod \"node-exporter-ltxjs\" (UID: \"34510396-e510-4255-aab4-724af10b00a4\") " pod="openshift-monitoring/node-exporter-ltxjs"
Jul 14 13:28:39.377831 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.377629    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fff04318-1a39-4b84-a5c5-d53129f9e071-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-bfqm5\" (UID: \"fff04318-1a39-4b84-a5c5-d53129f9e071\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bfqm5"
Jul 14 13:28:39.377831 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.377652    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8cs9\" (UniqueName: \"kubernetes.io/projected/fff04318-1a39-4b84-a5c5-d53129f9e071-kube-api-access-c8cs9\") pod \"kube-state-metrics-785778ff74-bfqm5\" (UID: \"fff04318-1a39-4b84-a5c5-d53129f9e071\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bfqm5"
Jul 14 13:28:39.377831 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.377676    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/34510396-e510-4255-aab4-724af10b00a4-node-exporter-textfile\") pod \"node-exporter-ltxjs\" (UID: \"34510396-e510-4255-aab4-724af10b00a4\") " pod="openshift-monitoring/node-exporter-ltxjs"
Jul 14 13:28:39.377831 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.377703    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fff04318-1a39-4b84-a5c5-d53129f9e071-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-bfqm5\" (UID: \"fff04318-1a39-4b84-a5c5-d53129f9e071\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bfqm5"
Jul 14 13:28:39.377831 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.377732    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/34510396-e510-4255-aab4-724af10b00a4-node-exporter-accelerators-collector-config\") pod \"node-exporter-ltxjs\" (UID: \"34510396-e510-4255-aab4-724af10b00a4\") " pod="openshift-monitoring/node-exporter-ltxjs"
Jul 14 13:28:39.378356 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.378101    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34510396-e510-4255-aab4-724af10b00a4-metrics-client-ca\") pod \"node-exporter-ltxjs\" (UID: \"34510396-e510-4255-aab4-724af10b00a4\") " pod="openshift-monitoring/node-exporter-ltxjs"
Jul 14 13:28:39.378356 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.378175    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34510396-e510-4255-aab4-724af10b00a4-sys\") pod \"node-exporter-ltxjs\" (UID: \"34510396-e510-4255-aab4-724af10b00a4\") " pod="openshift-monitoring/node-exporter-ltxjs"
Jul 14 13:28:39.378356 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.378253    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fff04318-1a39-4b84-a5c5-d53129f9e071-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-bfqm5\" (UID: \"fff04318-1a39-4b84-a5c5-d53129f9e071\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bfqm5"
Jul 14 13:28:39.378356 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.378274    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/34510396-e510-4255-aab4-724af10b00a4-node-exporter-accelerators-collector-config\") pod \"node-exporter-ltxjs\" (UID: \"34510396-e510-4255-aab4-724af10b00a4\") " pod="openshift-monitoring/node-exporter-ltxjs"
Jul 14 13:28:39.378356 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.378300    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/34510396-e510-4255-aab4-724af10b00a4-node-exporter-wtmp\") pod \"node-exporter-ltxjs\" (UID: \"34510396-e510-4255-aab4-724af10b00a4\") " pod="openshift-monitoring/node-exporter-ltxjs"
Jul 14 13:28:39.378605 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.378483    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/34510396-e510-4255-aab4-724af10b00a4-node-exporter-textfile\") pod \"node-exporter-ltxjs\" (UID: \"34510396-e510-4255-aab4-724af10b00a4\") " pod="openshift-monitoring/node-exporter-ltxjs"
Jul 14 13:28:39.378605 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.378531    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fff04318-1a39-4b84-a5c5-d53129f9e071-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-bfqm5\" (UID: \"fff04318-1a39-4b84-a5c5-d53129f9e071\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bfqm5"
Jul 14 13:28:39.378605 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.378573    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fff04318-1a39-4b84-a5c5-d53129f9e071-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-bfqm5\" (UID: \"fff04318-1a39-4b84-a5c5-d53129f9e071\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bfqm5"
Jul 14 13:28:39.380165 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.380141    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fff04318-1a39-4b84-a5c5-d53129f9e071-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-bfqm5\" (UID: \"fff04318-1a39-4b84-a5c5-d53129f9e071\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bfqm5"
Jul 14 13:28:39.380539 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.380515    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/34510396-e510-4255-aab4-724af10b00a4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ltxjs\" (UID: \"34510396-e510-4255-aab4-724af10b00a4\") " pod="openshift-monitoring/node-exporter-ltxjs"
Jul 14 13:28:39.380950 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.380929    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fff04318-1a39-4b84-a5c5-d53129f9e071-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-bfqm5\" (UID: \"fff04318-1a39-4b84-a5c5-d53129f9e071\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bfqm5"
Jul 14 13:28:39.380950 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.380944    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/34510396-e510-4255-aab4-724af10b00a4-node-exporter-tls\") pod \"node-exporter-ltxjs\" (UID: \"34510396-e510-4255-aab4-724af10b00a4\") " pod="openshift-monitoring/node-exporter-ltxjs"
Jul 14 13:28:39.388523 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.388480    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf6dx\" (UniqueName: \"kubernetes.io/projected/34510396-e510-4255-aab4-724af10b00a4-kube-api-access-kf6dx\") pod \"node-exporter-ltxjs\" (UID: \"34510396-e510-4255-aab4-724af10b00a4\") " pod="openshift-monitoring/node-exporter-ltxjs"
Jul 14 13:28:39.388980 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.388959    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8cs9\" (UniqueName: \"kubernetes.io/projected/fff04318-1a39-4b84-a5c5-d53129f9e071-kube-api-access-c8cs9\") pod \"kube-state-metrics-785778ff74-bfqm5\" (UID: \"fff04318-1a39-4b84-a5c5-d53129f9e071\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-bfqm5"
Jul 14 13:28:39.456558 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.456503    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-bfqm5"
Jul 14 13:28:39.471588 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.471563    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ltxjs"
Jul 14 13:28:39.559482 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.559455    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74cc4b7574-k9z2v"
Jul 14 13:28:39.559914 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.559740    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74cc4b7574-k9z2v"
Jul 14 13:28:39.559914 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.559796    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8c6d96db7-ztdq9" event={"ID":"909c52b2-92e9-43e0-b2f7-1ba042a9311b","Type":"ContainerStarted","Data":"3f66065637cf6bb0aa36b4902d9eb4af9d69b7a29382ee745e4937a76afbffd0"}
Jul 14 13:28:39.566487 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.566463    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74cc4b7574-k9z2v"
Jul 14 13:28:39.582699 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:39.582658    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8c6d96db7-ztdq9" podStartSLOduration=1.582642114 podStartE2EDuration="1.582642114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:28:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:28:39.581829225 +0000 UTC m=+84.908149168" watchObservedRunningTime="2026-07-14 13:28:39.582642114 +0000 UTC m=+84.908962098"
Jul 14 13:28:39.994234 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:28:39.994210    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34510396_e510_4255_aab4_724af10b00a4.slice/crio-f6ebd490e42a51bf32f47c2b101bf796b3fd630dbcbf59d0009f7c44820633e4 WatchSource:0}: Error finding container f6ebd490e42a51bf32f47c2b101bf796b3fd630dbcbf59d0009f7c44820633e4: Status 404 returned error can't find the container with id f6ebd490e42a51bf32f47c2b101bf796b3fd630dbcbf59d0009f7c44820633e4
Jul 14 13:28:40.117753 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:40.117733    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-bfqm5"]
Jul 14 13:28:40.120399 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:28:40.120378    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff04318_1a39_4b84_a5c5_d53129f9e071.slice/crio-da274bcd9ce8beeff4bcbe16335f4e0bf8e2a619f52ec998ee18c80dfdfecca9 WatchSource:0}: Error finding container da274bcd9ce8beeff4bcbe16335f4e0bf8e2a619f52ec998ee18c80dfdfecca9: Status 404 returned error can't find the container with id da274bcd9ce8beeff4bcbe16335f4e0bf8e2a619f52ec998ee18c80dfdfecca9
Jul 14 13:28:40.566715 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:40.566675    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xz2lp" event={"ID":"c650b1a4-6066-4d12-b4cb-0217c8f4ebfe","Type":"ContainerStarted","Data":"91bfe099ab6ffff704f9f9ea423c37571a9fb2d0995cefeba7c9a1acbdb4d3d8"}
Jul 14 13:28:40.568319 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:40.568288    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-bfqm5" event={"ID":"fff04318-1a39-4b84-a5c5-d53129f9e071","Type":"ContainerStarted","Data":"da274bcd9ce8beeff4bcbe16335f4e0bf8e2a619f52ec998ee18c80dfdfecca9"}
Jul 14 13:28:40.569792 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:40.569768    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ltxjs" event={"ID":"34510396-e510-4255-aab4-724af10b00a4","Type":"ContainerStarted","Data":"f6ebd490e42a51bf32f47c2b101bf796b3fd630dbcbf59d0009f7c44820633e4"}
Jul 14 13:28:40.574258 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:40.574231    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74cc4b7574-k9z2v"
Jul 14 13:28:40.586353 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:40.586274    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xz2lp" podStartSLOduration=33.591379913 podStartE2EDuration="40.586258872s" totalImagesPullingTime="6.994878959s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 13:28:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:28:40.58589565 +0000 UTC m=+85.912215579" watchObservedRunningTime="2026-07-14 13:28:40.586258872 +0000 UTC m=+85.912578799"
Jul 14 13:28:41.578105 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:41.578026    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-bfqm5" event={"ID":"fff04318-1a39-4b84-a5c5-d53129f9e071","Type":"ContainerStarted","Data":"37863db80e27e1d1ce67bf24b8d81a15fafae37859c0ccfc42b90d9e1c8181a0"}
Jul 14 13:28:41.580330 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:41.580290    2569 generic.go:356] "Generic (PLEG): container finished" podID="34510396-e510-4255-aab4-724af10b00a4" containerID="74711feea9160db8fe6afcd3cb544dfa141f1a6c3338f54d32ab6ee1ae4b3e4a" exitCode=0
Jul 14 13:28:41.580458 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:41.580389    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ltxjs" event={"ID":"34510396-e510-4255-aab4-724af10b00a4","Type":"ContainerDied","Data":"74711feea9160db8fe6afcd3cb544dfa141f1a6c3338f54d32ab6ee1ae4b3e4a"}
Jul 14 13:28:42.586916 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:42.586876    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-bfqm5" event={"ID":"fff04318-1a39-4b84-a5c5-d53129f9e071","Type":"ContainerStarted","Data":"1c4a5831f8b6735e58a8814a96bdd07b5c90a73a836536340b0b9254199305c8"}
Jul 14 13:28:42.587530 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:42.586989    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-bfqm5" event={"ID":"fff04318-1a39-4b84-a5c5-d53129f9e071","Type":"ContainerStarted","Data":"7606b2bae3dea2bd3a4cf7f31ba5ddb60dd88b6c0ebbc643de3d893f0dfdfc14"}
Jul 14 13:28:42.589168 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:42.589141    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ltxjs" event={"ID":"34510396-e510-4255-aab4-724af10b00a4","Type":"ContainerStarted","Data":"4c75c348f3304bbf45d263e9dfe592312135131cd65fa7b55c92255f15689bf3"}
Jul 14 13:28:42.589265 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:42.589187    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ltxjs" event={"ID":"34510396-e510-4255-aab4-724af10b00a4","Type":"ContainerStarted","Data":"a5852bf110208d2a198f11a2a30aa249b235ce150920b3aec4e4b27457d288df"}
Jul 14 13:28:42.609448 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:42.609403    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-bfqm5" podStartSLOduration=2.231688405 podStartE2EDuration="3.60938922s" totalImagesPullingTime="1.377700815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:28:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:28:42.608271262 +0000 UTC m=+87.934591191" watchObservedRunningTime="2026-07-14 13:28:42.60938922 +0000 UTC m=+87.935709148"
Jul 14 13:28:42.632655 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:42.632614    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ltxjs" podStartSLOduration=2.554232757 podStartE2EDuration="3.632600489s" totalImagesPullingTime="1.009738162s" totalInitContainerRuntime="68.62957ms" isStatefulPod=true podCreationTimestamp="2026-07-14 13:28:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:28:42.63140149 +0000 UTC m=+87.957721418" watchObservedRunningTime="2026-07-14 13:28:42.632600489 +0000 UTC m=+87.958920417"
Jul 14 13:28:43.915628 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:43.915596    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8c6d96db7-ztdq9"]
Jul 14 13:28:45.203810 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:45.203781    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-584b976c99-8blm8"
Jul 14 13:28:47.490343 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:47.490318    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-59f76b4df9-2glpk"
Jul 14 13:28:48.356251 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:48.356202    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8c6d96db7-ztdq9"
Jul 14 13:28:48.633271 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:48.633193    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-858f9df6bf-b69bj"]
Jul 14 13:28:48.637583 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:48.637552    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858f9df6bf-b69bj"
Jul 14 13:28:48.646854 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:48.646835    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-858f9df6bf-b69bj"]
Jul 14 13:28:48.756088 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:48.756055    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-oauth-serving-cert\") pod \"console-858f9df6bf-b69bj\" (UID: \"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6\") " pod="openshift-console/console-858f9df6bf-b69bj"
Jul 14 13:28:48.756228 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:48.756124    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-console-serving-cert\") pod \"console-858f9df6bf-b69bj\" (UID: \"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6\") " pod="openshift-console/console-858f9df6bf-b69bj"
Jul 14 13:28:48.756228 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:48.756149    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-console-config\") pod \"console-858f9df6bf-b69bj\" (UID: \"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6\") " pod="openshift-console/console-858f9df6bf-b69bj"
Jul 14 13:28:48.756228 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:48.756166    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck7k7\" (UniqueName: \"kubernetes.io/projected/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-kube-api-access-ck7k7\") pod \"console-858f9df6bf-b69bj\" (UID: \"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6\") " pod="openshift-console/console-858f9df6bf-b69bj"
Jul 14 13:28:48.756340 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:48.756240    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-console-oauth-config\") pod \"console-858f9df6bf-b69bj\" (UID: \"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6\") " pod="openshift-console/console-858f9df6bf-b69bj"
Jul 14 13:28:48.756340 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:48.756256    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-trusted-ca-bundle\") pod \"console-858f9df6bf-b69bj\" (UID: \"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6\") " pod="openshift-console/console-858f9df6bf-b69bj"
Jul 14 13:28:48.756340 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:48.756304    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-service-ca\") pod \"console-858f9df6bf-b69bj\" (UID: \"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6\") " pod="openshift-console/console-858f9df6bf-b69bj"
Jul 14 13:28:48.856908 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:48.856869    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-service-ca\") pod \"console-858f9df6bf-b69bj\" (UID: \"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6\") " pod="openshift-console/console-858f9df6bf-b69bj"
Jul 14 13:28:48.857090 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:48.856934    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-oauth-serving-cert\") pod \"console-858f9df6bf-b69bj\" (UID: \"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6\") " pod="openshift-console/console-858f9df6bf-b69bj"
Jul 14 13:28:48.857090 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:48.856963    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-console-serving-cert\") pod \"console-858f9df6bf-b69bj\" (UID: \"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6\") " pod="openshift-console/console-858f9df6bf-b69bj"
Jul 14 13:28:48.857090 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:48.856980    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-console-config\") pod \"console-858f9df6bf-b69bj\" (UID: \"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6\") " pod="openshift-console/console-858f9df6bf-b69bj"
Jul 14 13:28:48.857090 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:48.857000    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck7k7\" (UniqueName: \"kubernetes.io/projected/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-kube-api-access-ck7k7\") pod \"console-858f9df6bf-b69bj\" (UID: \"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6\") " pod="openshift-console/console-858f9df6bf-b69bj"
Jul 14 13:28:48.857090 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:48.857054    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-console-oauth-config\") pod \"console-858f9df6bf-b69bj\" (UID: \"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6\") " pod="openshift-console/console-858f9df6bf-b69bj"
Jul 14 13:28:48.857090 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:48.857077    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-trusted-ca-bundle\") pod \"console-858f9df6bf-b69bj\" (UID: \"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6\") " pod="openshift-console/console-858f9df6bf-b69bj"
Jul 14 13:28:48.857755 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:48.857725    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-oauth-serving-cert\") pod \"console-858f9df6bf-b69bj\" (UID: \"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6\") " pod="openshift-console/console-858f9df6bf-b69bj"
Jul 14 13:28:48.857900 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:48.857755    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-console-config\") pod \"console-858f9df6bf-b69bj\" (UID: \"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6\") " pod="openshift-console/console-858f9df6bf-b69bj"
Jul 14 13:28:48.857900 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:48.857725    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-service-ca\") pod \"console-858f9df6bf-b69bj\" (UID: \"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6\") " pod="openshift-console/console-858f9df6bf-b69bj"
Jul 14 13:28:48.858063 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:48.857917    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-trusted-ca-bundle\") pod \"console-858f9df6bf-b69bj\" (UID: \"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6\") " pod="openshift-console/console-858f9df6bf-b69bj"
Jul 14 13:28:48.859463 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:48.859445    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-console-serving-cert\") pod \"console-858f9df6bf-b69bj\" (UID: \"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6\") " pod="openshift-console/console-858f9df6bf-b69bj"
Jul 14 13:28:48.859611 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:48.859589    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-console-oauth-config\") pod \"console-858f9df6bf-b69bj\" (UID: \"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6\") " pod="openshift-console/console-858f9df6bf-b69bj"
Jul 14 13:28:48.866581 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:48.866562    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck7k7\" (UniqueName: \"kubernetes.io/projected/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-kube-api-access-ck7k7\") pod \"console-858f9df6bf-b69bj\" (UID: \"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6\") " pod="openshift-console/console-858f9df6bf-b69bj"
Jul 14 13:28:48.946734 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:48.946681    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858f9df6bf-b69bj"
Jul 14 13:28:49.072489 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:49.072466    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-858f9df6bf-b69bj"]
Jul 14 13:28:49.074932 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:28:49.074904    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc33b51_67c9_49d7_8d9d_3e3dc1cb12e6.slice/crio-29567fd363520148220d244c5348a49c5d0ab6d2b27663e07613bbd6c778b52e WatchSource:0}: Error finding container 29567fd363520148220d244c5348a49c5d0ab6d2b27663e07613bbd6c778b52e: Status 404 returned error can't find the container with id 29567fd363520148220d244c5348a49c5d0ab6d2b27663e07613bbd6c778b52e
Jul 14 13:28:49.616386 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:49.616347    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858f9df6bf-b69bj" event={"ID":"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6","Type":"ContainerStarted","Data":"7c933318498c24f531f945d4a194f00ff8f99bac7e0ddb76ec69fbdbc8d3c36d"}
Jul 14 13:28:49.616519 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:49.616404    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858f9df6bf-b69bj" event={"ID":"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6","Type":"ContainerStarted","Data":"29567fd363520148220d244c5348a49c5d0ab6d2b27663e07613bbd6c778b52e"}
Jul 14 13:28:49.639518 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:49.639472    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-858f9df6bf-b69bj" podStartSLOduration=1.639455389 podStartE2EDuration="1.639455389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:28:49.638459278 +0000 UTC m=+94.964779207" watchObservedRunningTime="2026-07-14 13:28:49.639455389 +0000 UTC m=+94.965775316"
Jul 14 13:28:50.218344 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:50.218288    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-584b976c99-8blm8" podUID="d137293e-b342-4b7e-938a-bc40f150ae6b" containerName="registry" containerID="cri-o://c1782b12cc6ed6d43659b9a5f324ed868158156191dade404083d7c1e3d974db" gracePeriod=30
Jul 14 13:28:50.460766 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:50.460744    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-584b976c99-8blm8"
Jul 14 13:28:50.570923 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:50.570895    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d137293e-b342-4b7e-938a-bc40f150ae6b-trusted-ca\") pod \"d137293e-b342-4b7e-938a-bc40f150ae6b\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") "
Jul 14 13:28:50.571066 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:50.570929    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d137293e-b342-4b7e-938a-bc40f150ae6b-ca-trust-extracted\") pod \"d137293e-b342-4b7e-938a-bc40f150ae6b\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") "
Jul 14 13:28:50.571066 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:50.570951    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5t5p\" (UniqueName: \"kubernetes.io/projected/d137293e-b342-4b7e-938a-bc40f150ae6b-kube-api-access-q5t5p\") pod \"d137293e-b342-4b7e-938a-bc40f150ae6b\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") "
Jul 14 13:28:50.571066 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:50.570974    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d137293e-b342-4b7e-938a-bc40f150ae6b-installation-pull-secrets\") pod \"d137293e-b342-4b7e-938a-bc40f150ae6b\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") "
Jul 14 13:28:50.571066 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:50.571060    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d137293e-b342-4b7e-938a-bc40f150ae6b-registry-tls\") pod \"d137293e-b342-4b7e-938a-bc40f150ae6b\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") "
Jul 14 13:28:50.571277 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:50.571088    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d137293e-b342-4b7e-938a-bc40f150ae6b-bound-sa-token\") pod \"d137293e-b342-4b7e-938a-bc40f150ae6b\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") "
Jul 14 13:28:50.571277 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:50.571134    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d137293e-b342-4b7e-938a-bc40f150ae6b-image-registry-private-configuration\") pod \"d137293e-b342-4b7e-938a-bc40f150ae6b\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") "
Jul 14 13:28:50.571277 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:50.571187    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d137293e-b342-4b7e-938a-bc40f150ae6b-registry-certificates\") pod \"d137293e-b342-4b7e-938a-bc40f150ae6b\" (UID: \"d137293e-b342-4b7e-938a-bc40f150ae6b\") "
Jul 14 13:28:50.571437 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:50.571379    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d137293e-b342-4b7e-938a-bc40f150ae6b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d137293e-b342-4b7e-938a-bc40f150ae6b" (UID: "d137293e-b342-4b7e-938a-bc40f150ae6b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 13:28:50.571501 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:50.571468    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d137293e-b342-4b7e-938a-bc40f150ae6b-trusted-ca\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:28:50.572133 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:50.572040    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d137293e-b342-4b7e-938a-bc40f150ae6b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d137293e-b342-4b7e-938a-bc40f150ae6b" (UID: "d137293e-b342-4b7e-938a-bc40f150ae6b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 13:28:50.573824 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:50.573796    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d137293e-b342-4b7e-938a-bc40f150ae6b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d137293e-b342-4b7e-938a-bc40f150ae6b" (UID: "d137293e-b342-4b7e-938a-bc40f150ae6b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 13:28:50.573973 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:50.573886    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d137293e-b342-4b7e-938a-bc40f150ae6b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d137293e-b342-4b7e-938a-bc40f150ae6b" (UID: "d137293e-b342-4b7e-938a-bc40f150ae6b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 13:28:50.573973 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:50.573957    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d137293e-b342-4b7e-938a-bc40f150ae6b-kube-api-access-q5t5p" (OuterVolumeSpecName: "kube-api-access-q5t5p") pod "d137293e-b342-4b7e-938a-bc40f150ae6b" (UID: "d137293e-b342-4b7e-938a-bc40f150ae6b"). InnerVolumeSpecName "kube-api-access-q5t5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 13:28:50.574119 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:50.574044    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d137293e-b342-4b7e-938a-bc40f150ae6b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d137293e-b342-4b7e-938a-bc40f150ae6b" (UID: "d137293e-b342-4b7e-938a-bc40f150ae6b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 13:28:50.574194 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:50.574178    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d137293e-b342-4b7e-938a-bc40f150ae6b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d137293e-b342-4b7e-938a-bc40f150ae6b" (UID: "d137293e-b342-4b7e-938a-bc40f150ae6b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 13:28:50.580184 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:50.580161    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d137293e-b342-4b7e-938a-bc40f150ae6b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d137293e-b342-4b7e-938a-bc40f150ae6b" (UID: "d137293e-b342-4b7e-938a-bc40f150ae6b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 13:28:50.621232 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:50.621210    2569 generic.go:356] "Generic (PLEG): container finished" podID="d137293e-b342-4b7e-938a-bc40f150ae6b" containerID="c1782b12cc6ed6d43659b9a5f324ed868158156191dade404083d7c1e3d974db" exitCode=0
Jul 14 13:28:50.621320 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:50.621252    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-584b976c99-8blm8"
Jul 14 13:28:50.621320 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:50.621299    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-584b976c99-8blm8" event={"ID":"d137293e-b342-4b7e-938a-bc40f150ae6b","Type":"ContainerDied","Data":"c1782b12cc6ed6d43659b9a5f324ed868158156191dade404083d7c1e3d974db"}
Jul 14 13:28:50.621419 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:50.621346    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-584b976c99-8blm8" event={"ID":"d137293e-b342-4b7e-938a-bc40f150ae6b","Type":"ContainerDied","Data":"8f3ed3a35445138efb7e0056c2e675a5688110cb8dc74c9e0afa3bb4656e1bc3"}
Jul 14 13:28:50.621419 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:50.621366    2569 scope.go:117] "RemoveContainer" containerID="c1782b12cc6ed6d43659b9a5f324ed868158156191dade404083d7c1e3d974db"
Jul 14 13:28:50.629342 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:50.629326    2569 scope.go:117] "RemoveContainer" containerID="c1782b12cc6ed6d43659b9a5f324ed868158156191dade404083d7c1e3d974db"
Jul 14 13:28:50.629577 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:28:50.629558    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1782b12cc6ed6d43659b9a5f324ed868158156191dade404083d7c1e3d974db\": container with ID starting with c1782b12cc6ed6d43659b9a5f324ed868158156191dade404083d7c1e3d974db not found: ID does not exist" containerID="c1782b12cc6ed6d43659b9a5f324ed868158156191dade404083d7c1e3d974db"
Jul 14 13:28:50.629626 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:50.629584    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1782b12cc6ed6d43659b9a5f324ed868158156191dade404083d7c1e3d974db"} err="failed to get container status \"c1782b12cc6ed6d43659b9a5f324ed868158156191dade404083d7c1e3d974db\": rpc error: code = NotFound desc = could not find container \"c1782b12cc6ed6d43659b9a5f324ed868158156191dade404083d7c1e3d974db\": container with ID starting with c1782b12cc6ed6d43659b9a5f324ed868158156191dade404083d7c1e3d974db not found: ID does not exist"
Jul 14 13:28:50.643543 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:50.643519    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-584b976c99-8blm8"]
Jul 14 13:28:50.650260 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:50.650241    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-584b976c99-8blm8"]
Jul 14 13:28:50.672145 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:50.672123    2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d137293e-b342-4b7e-938a-bc40f150ae6b-registry-tls\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:28:50.672145 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:50.672141    2569 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d137293e-b342-4b7e-938a-bc40f150ae6b-bound-sa-token\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:28:50.672260 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:50.672150    2569 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d137293e-b342-4b7e-938a-bc40f150ae6b-image-registry-private-configuration\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:28:50.672260 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:50.672160    2569 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d137293e-b342-4b7e-938a-bc40f150ae6b-registry-certificates\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:28:50.672260 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:50.672171    2569 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d137293e-b342-4b7e-938a-bc40f150ae6b-ca-trust-extracted\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:28:50.672260 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:50.672179    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5t5p\" (UniqueName: \"kubernetes.io/projected/d137293e-b342-4b7e-938a-bc40f150ae6b-kube-api-access-q5t5p\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:28:50.672260 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:50.672187    2569 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d137293e-b342-4b7e-938a-bc40f150ae6b-installation-pull-secrets\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:28:51.191517 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:51.191487    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d137293e-b342-4b7e-938a-bc40f150ae6b" path="/var/lib/kubelet/pods/d137293e-b342-4b7e-938a-bc40f150ae6b/volumes"
Jul 14 13:28:55.471249 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:55.471219    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-42hkp"
Jul 14 13:28:58.947231 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:58.947198    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-858f9df6bf-b69bj"
Jul 14 13:28:58.947702 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:58.947276    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-858f9df6bf-b69bj"
Jul 14 13:28:58.951962 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:58.951942    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-858f9df6bf-b69bj"
Jul 14 13:28:59.658464 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:59.658435    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-858f9df6bf-b69bj"
Jul 14 13:28:59.713328 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:28:59.713298    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74cc4b7574-k9z2v"]
Jul 14 13:29:08.936140 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:08.936097    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8c6d96db7-ztdq9" podUID="909c52b2-92e9-43e0-b2f7-1ba042a9311b" containerName="console" containerID="cri-o://3f66065637cf6bb0aa36b4902d9eb4af9d69b7a29382ee745e4937a76afbffd0" gracePeriod=15
Jul 14 13:29:09.693475 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:09.693447    2569 generic.go:356] "Generic (PLEG): container finished" podID="909c52b2-92e9-43e0-b2f7-1ba042a9311b" containerID="3f66065637cf6bb0aa36b4902d9eb4af9d69b7a29382ee745e4937a76afbffd0" exitCode=0
Jul 14 13:29:09.693584 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:09.693520    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8c6d96db7-ztdq9" event={"ID":"909c52b2-92e9-43e0-b2f7-1ba042a9311b","Type":"ContainerDied","Data":"3f66065637cf6bb0aa36b4902d9eb4af9d69b7a29382ee745e4937a76afbffd0"}
Jul 14 13:29:09.693625 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:09.693594    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8c6d96db7-ztdq9" event={"ID":"909c52b2-92e9-43e0-b2f7-1ba042a9311b","Type":"ContainerDied","Data":"a57d00a4ab592a4999d2f279bc8e28a30e8e2b06674e953d155a0d6d0d4d7283"}
Jul 14 13:29:09.693625 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:09.693610    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a57d00a4ab592a4999d2f279bc8e28a30e8e2b06674e953d155a0d6d0d4d7283"
Jul 14 13:29:09.710967 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:09.710948    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8c6d96db7-ztdq9"
Jul 14 13:29:09.809574 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:09.809547    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/909c52b2-92e9-43e0-b2f7-1ba042a9311b-console-oauth-config\") pod \"909c52b2-92e9-43e0-b2f7-1ba042a9311b\" (UID: \"909c52b2-92e9-43e0-b2f7-1ba042a9311b\") "
Jul 14 13:29:09.809686 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:09.809586    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/909c52b2-92e9-43e0-b2f7-1ba042a9311b-service-ca\") pod \"909c52b2-92e9-43e0-b2f7-1ba042a9311b\" (UID: \"909c52b2-92e9-43e0-b2f7-1ba042a9311b\") "
Jul 14 13:29:09.809686 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:09.809636    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfzzt\" (UniqueName: \"kubernetes.io/projected/909c52b2-92e9-43e0-b2f7-1ba042a9311b-kube-api-access-sfzzt\") pod \"909c52b2-92e9-43e0-b2f7-1ba042a9311b\" (UID: \"909c52b2-92e9-43e0-b2f7-1ba042a9311b\") "
Jul 14 13:29:09.809807 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:09.809699    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/909c52b2-92e9-43e0-b2f7-1ba042a9311b-console-config\") pod \"909c52b2-92e9-43e0-b2f7-1ba042a9311b\" (UID: \"909c52b2-92e9-43e0-b2f7-1ba042a9311b\") "
Jul 14 13:29:09.809807 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:09.809727    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/909c52b2-92e9-43e0-b2f7-1ba042a9311b-console-serving-cert\") pod \"909c52b2-92e9-43e0-b2f7-1ba042a9311b\" (UID: \"909c52b2-92e9-43e0-b2f7-1ba042a9311b\") "
Jul 14 13:29:09.809807 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:09.809766    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/909c52b2-92e9-43e0-b2f7-1ba042a9311b-oauth-serving-cert\") pod \"909c52b2-92e9-43e0-b2f7-1ba042a9311b\" (UID: \"909c52b2-92e9-43e0-b2f7-1ba042a9311b\") "
Jul 14 13:29:09.809948 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:09.809821    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/909c52b2-92e9-43e0-b2f7-1ba042a9311b-trusted-ca-bundle\") pod \"909c52b2-92e9-43e0-b2f7-1ba042a9311b\" (UID: \"909c52b2-92e9-43e0-b2f7-1ba042a9311b\") "
Jul 14 13:29:09.810125 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:09.810098    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/909c52b2-92e9-43e0-b2f7-1ba042a9311b-service-ca" (OuterVolumeSpecName: "service-ca") pod "909c52b2-92e9-43e0-b2f7-1ba042a9311b" (UID: "909c52b2-92e9-43e0-b2f7-1ba042a9311b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 13:29:09.810285 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:09.810128    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/909c52b2-92e9-43e0-b2f7-1ba042a9311b-console-config" (OuterVolumeSpecName: "console-config") pod "909c52b2-92e9-43e0-b2f7-1ba042a9311b" (UID: "909c52b2-92e9-43e0-b2f7-1ba042a9311b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 13:29:09.810285 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:09.810177    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/909c52b2-92e9-43e0-b2f7-1ba042a9311b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "909c52b2-92e9-43e0-b2f7-1ba042a9311b" (UID: "909c52b2-92e9-43e0-b2f7-1ba042a9311b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 13:29:09.810454 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:09.810418    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/909c52b2-92e9-43e0-b2f7-1ba042a9311b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "909c52b2-92e9-43e0-b2f7-1ba042a9311b" (UID: "909c52b2-92e9-43e0-b2f7-1ba042a9311b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 13:29:09.811922 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:09.811893    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/909c52b2-92e9-43e0-b2f7-1ba042a9311b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "909c52b2-92e9-43e0-b2f7-1ba042a9311b" (UID: "909c52b2-92e9-43e0-b2f7-1ba042a9311b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 13:29:09.812059 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:09.811977    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/909c52b2-92e9-43e0-b2f7-1ba042a9311b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "909c52b2-92e9-43e0-b2f7-1ba042a9311b" (UID: "909c52b2-92e9-43e0-b2f7-1ba042a9311b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 13:29:09.812113 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:09.812057    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/909c52b2-92e9-43e0-b2f7-1ba042a9311b-kube-api-access-sfzzt" (OuterVolumeSpecName: "kube-api-access-sfzzt") pod "909c52b2-92e9-43e0-b2f7-1ba042a9311b" (UID: "909c52b2-92e9-43e0-b2f7-1ba042a9311b"). InnerVolumeSpecName "kube-api-access-sfzzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 13:29:09.910563 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:09.910535    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfzzt\" (UniqueName: \"kubernetes.io/projected/909c52b2-92e9-43e0-b2f7-1ba042a9311b-kube-api-access-sfzzt\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:29:09.910563 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:09.910560    2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/909c52b2-92e9-43e0-b2f7-1ba042a9311b-console-config\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:29:09.910713 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:09.910570    2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/909c52b2-92e9-43e0-b2f7-1ba042a9311b-console-serving-cert\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:29:09.910713 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:09.910578    2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/909c52b2-92e9-43e0-b2f7-1ba042a9311b-oauth-serving-cert\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:29:09.910713 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:09.910587    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/909c52b2-92e9-43e0-b2f7-1ba042a9311b-trusted-ca-bundle\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:29:09.910713 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:09.910595    2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/909c52b2-92e9-43e0-b2f7-1ba042a9311b-console-oauth-config\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:29:09.910713 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:09.910603    2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/909c52b2-92e9-43e0-b2f7-1ba042a9311b-service-ca\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:29:10.697364 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:10.697334    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8c6d96db7-ztdq9"
Jul 14 13:29:10.719150 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:10.719127    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8c6d96db7-ztdq9"]
Jul 14 13:29:10.724437 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:10.724415    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8c6d96db7-ztdq9"]
Jul 14 13:29:11.192158 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:11.192125    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="909c52b2-92e9-43e0-b2f7-1ba042a9311b" path="/var/lib/kubelet/pods/909c52b2-92e9-43e0-b2f7-1ba042a9311b/volumes"
Jul 14 13:29:24.732976 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:24.732912    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-74cc4b7574-k9z2v" podUID="a1138d77-ce78-41d4-9656-02843d9b24b3" containerName="console" containerID="cri-o://6878ef37fe44c4bc7fc916b312967a9481c6eca5aaf1e999af3477198b85d643" gracePeriod=15
Jul 14 13:29:24.970973 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:24.970949    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74cc4b7574-k9z2v"
Jul 14 13:29:25.010763 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:25.010705    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1138d77-ce78-41d4-9656-02843d9b24b3-console-oauth-config\") pod \"a1138d77-ce78-41d4-9656-02843d9b24b3\" (UID: \"a1138d77-ce78-41d4-9656-02843d9b24b3\") "
Jul 14 13:29:25.010866 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:25.010818    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1138d77-ce78-41d4-9656-02843d9b24b3-oauth-serving-cert\") pod \"a1138d77-ce78-41d4-9656-02843d9b24b3\" (UID: \"a1138d77-ce78-41d4-9656-02843d9b24b3\") "
Jul 14 13:29:25.010866 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:25.010848    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1138d77-ce78-41d4-9656-02843d9b24b3-console-config\") pod \"a1138d77-ce78-41d4-9656-02843d9b24b3\" (UID: \"a1138d77-ce78-41d4-9656-02843d9b24b3\") "
Jul 14 13:29:25.010948 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:25.010886    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9q2r\" (UniqueName: \"kubernetes.io/projected/a1138d77-ce78-41d4-9656-02843d9b24b3-kube-api-access-r9q2r\") pod \"a1138d77-ce78-41d4-9656-02843d9b24b3\" (UID: \"a1138d77-ce78-41d4-9656-02843d9b24b3\") "
Jul 14 13:29:25.011100 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:25.011075    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1138d77-ce78-41d4-9656-02843d9b24b3-service-ca\") pod \"a1138d77-ce78-41d4-9656-02843d9b24b3\" (UID: \"a1138d77-ce78-41d4-9656-02843d9b24b3\") "
Jul 14 13:29:25.011190 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:25.011121    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1138d77-ce78-41d4-9656-02843d9b24b3-console-serving-cert\") pod \"a1138d77-ce78-41d4-9656-02843d9b24b3\" (UID: \"a1138d77-ce78-41d4-9656-02843d9b24b3\") "
Jul 14 13:29:25.011248 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:25.011215    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1138d77-ce78-41d4-9656-02843d9b24b3-console-config" (OuterVolumeSpecName: "console-config") pod "a1138d77-ce78-41d4-9656-02843d9b24b3" (UID: "a1138d77-ce78-41d4-9656-02843d9b24b3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 13:29:25.011248 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:25.011222    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1138d77-ce78-41d4-9656-02843d9b24b3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a1138d77-ce78-41d4-9656-02843d9b24b3" (UID: "a1138d77-ce78-41d4-9656-02843d9b24b3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 13:29:25.011405 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:25.011388    2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1138d77-ce78-41d4-9656-02843d9b24b3-oauth-serving-cert\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:29:25.011441 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:25.011412    2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1138d77-ce78-41d4-9656-02843d9b24b3-console-config\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:29:25.011551 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:25.011526    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1138d77-ce78-41d4-9656-02843d9b24b3-service-ca" (OuterVolumeSpecName: "service-ca") pod "a1138d77-ce78-41d4-9656-02843d9b24b3" (UID: "a1138d77-ce78-41d4-9656-02843d9b24b3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 13:29:25.012952 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:25.012923    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1138d77-ce78-41d4-9656-02843d9b24b3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a1138d77-ce78-41d4-9656-02843d9b24b3" (UID: "a1138d77-ce78-41d4-9656-02843d9b24b3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 13:29:25.013142 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:25.013119    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1138d77-ce78-41d4-9656-02843d9b24b3-kube-api-access-r9q2r" (OuterVolumeSpecName: "kube-api-access-r9q2r") pod "a1138d77-ce78-41d4-9656-02843d9b24b3" (UID: "a1138d77-ce78-41d4-9656-02843d9b24b3"). InnerVolumeSpecName "kube-api-access-r9q2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 13:29:25.013207 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:25.013184    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1138d77-ce78-41d4-9656-02843d9b24b3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a1138d77-ce78-41d4-9656-02843d9b24b3" (UID: "a1138d77-ce78-41d4-9656-02843d9b24b3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 13:29:25.112248 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:25.112222    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9q2r\" (UniqueName: \"kubernetes.io/projected/a1138d77-ce78-41d4-9656-02843d9b24b3-kube-api-access-r9q2r\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:29:25.112248 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:25.112245    2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1138d77-ce78-41d4-9656-02843d9b24b3-service-ca\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:29:25.112380 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:25.112257    2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1138d77-ce78-41d4-9656-02843d9b24b3-console-serving-cert\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:29:25.112380 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:25.112269    2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1138d77-ce78-41d4-9656-02843d9b24b3-console-oauth-config\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:29:25.759041 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:25.758988    2569 generic.go:356] "Generic (PLEG): container finished" podID="a1138d77-ce78-41d4-9656-02843d9b24b3" containerID="6878ef37fe44c4bc7fc916b312967a9481c6eca5aaf1e999af3477198b85d643" exitCode=0
Jul 14 13:29:25.759547 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:25.759064    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74cc4b7574-k9z2v"
Jul 14 13:29:25.759547 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:25.759066    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74cc4b7574-k9z2v" event={"ID":"a1138d77-ce78-41d4-9656-02843d9b24b3","Type":"ContainerDied","Data":"6878ef37fe44c4bc7fc916b312967a9481c6eca5aaf1e999af3477198b85d643"}
Jul 14 13:29:25.759547 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:25.759149    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74cc4b7574-k9z2v" event={"ID":"a1138d77-ce78-41d4-9656-02843d9b24b3","Type":"ContainerDied","Data":"2c31efcb51e27ea852d3863d2f299479751fa471604374140fdf2d53944964da"}
Jul 14 13:29:25.759547 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:25.759184    2569 scope.go:117] "RemoveContainer" containerID="6878ef37fe44c4bc7fc916b312967a9481c6eca5aaf1e999af3477198b85d643"
Jul 14 13:29:25.766911 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:25.766892    2569 scope.go:117] "RemoveContainer" containerID="6878ef37fe44c4bc7fc916b312967a9481c6eca5aaf1e999af3477198b85d643"
Jul 14 13:29:25.767160 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:29:25.767141    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6878ef37fe44c4bc7fc916b312967a9481c6eca5aaf1e999af3477198b85d643\": container with ID starting with 6878ef37fe44c4bc7fc916b312967a9481c6eca5aaf1e999af3477198b85d643 not found: ID does not exist" containerID="6878ef37fe44c4bc7fc916b312967a9481c6eca5aaf1e999af3477198b85d643"
Jul 14 13:29:25.767217 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:25.767168    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6878ef37fe44c4bc7fc916b312967a9481c6eca5aaf1e999af3477198b85d643"} err="failed to get container status \"6878ef37fe44c4bc7fc916b312967a9481c6eca5aaf1e999af3477198b85d643\": rpc error: code = NotFound desc = could not find container \"6878ef37fe44c4bc7fc916b312967a9481c6eca5aaf1e999af3477198b85d643\": container with ID starting with 6878ef37fe44c4bc7fc916b312967a9481c6eca5aaf1e999af3477198b85d643 not found: ID does not exist"
Jul 14 13:29:25.778302 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:25.778280    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74cc4b7574-k9z2v"]
Jul 14 13:29:25.782904 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:25.782885    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74cc4b7574-k9z2v"]
Jul 14 13:29:27.190956 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:29:27.190916    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1138d77-ce78-41d4-9656-02843d9b24b3" path="/var/lib/kubelet/pods/a1138d77-ce78-41d4-9656-02843d9b24b3/volumes"
Jul 14 13:30:08.146584 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.146513    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-546d6554f6-6xfzk"]
Jul 14 13:30:08.147082 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.146790    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d137293e-b342-4b7e-938a-bc40f150ae6b" containerName="registry"
Jul 14 13:30:08.147082 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.146804    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d137293e-b342-4b7e-938a-bc40f150ae6b" containerName="registry"
Jul 14 13:30:08.147082 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.146812    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1138d77-ce78-41d4-9656-02843d9b24b3" containerName="console"
Jul 14 13:30:08.147082 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.146817    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1138d77-ce78-41d4-9656-02843d9b24b3" containerName="console"
Jul 14 13:30:08.147082 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.146825    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="909c52b2-92e9-43e0-b2f7-1ba042a9311b" containerName="console"
Jul 14 13:30:08.147082 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.146831    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="909c52b2-92e9-43e0-b2f7-1ba042a9311b" containerName="console"
Jul 14 13:30:08.147082 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.146871    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d137293e-b342-4b7e-938a-bc40f150ae6b" containerName="registry"
Jul 14 13:30:08.147082 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.146879    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1138d77-ce78-41d4-9656-02843d9b24b3" containerName="console"
Jul 14 13:30:08.147082 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.146886    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="909c52b2-92e9-43e0-b2f7-1ba042a9311b" containerName="console"
Jul 14 13:30:08.148595 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.148572    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546d6554f6-6xfzk"
Jul 14 13:30:08.165033 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.164996    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-546d6554f6-6xfzk"]
Jul 14 13:30:08.196977 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.196952    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/751bad59-841d-4ec7-a16d-90b66fdb38c6-service-ca\") pod \"console-546d6554f6-6xfzk\" (UID: \"751bad59-841d-4ec7-a16d-90b66fdb38c6\") " pod="openshift-console/console-546d6554f6-6xfzk"
Jul 14 13:30:08.197094 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.196982    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/751bad59-841d-4ec7-a16d-90b66fdb38c6-trusted-ca-bundle\") pod \"console-546d6554f6-6xfzk\" (UID: \"751bad59-841d-4ec7-a16d-90b66fdb38c6\") " pod="openshift-console/console-546d6554f6-6xfzk"
Jul 14 13:30:08.197094 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.197054    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/751bad59-841d-4ec7-a16d-90b66fdb38c6-console-oauth-config\") pod \"console-546d6554f6-6xfzk\" (UID: \"751bad59-841d-4ec7-a16d-90b66fdb38c6\") " pod="openshift-console/console-546d6554f6-6xfzk"
Jul 14 13:30:08.197094 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.197088    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/751bad59-841d-4ec7-a16d-90b66fdb38c6-console-serving-cert\") pod \"console-546d6554f6-6xfzk\" (UID: \"751bad59-841d-4ec7-a16d-90b66fdb38c6\") " pod="openshift-console/console-546d6554f6-6xfzk"
Jul 14 13:30:08.197207 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.197122    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/751bad59-841d-4ec7-a16d-90b66fdb38c6-console-config\") pod \"console-546d6554f6-6xfzk\" (UID: \"751bad59-841d-4ec7-a16d-90b66fdb38c6\") " pod="openshift-console/console-546d6554f6-6xfzk"
Jul 14 13:30:08.197207 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.197153    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/751bad59-841d-4ec7-a16d-90b66fdb38c6-oauth-serving-cert\") pod \"console-546d6554f6-6xfzk\" (UID: \"751bad59-841d-4ec7-a16d-90b66fdb38c6\") " pod="openshift-console/console-546d6554f6-6xfzk"
Jul 14 13:30:08.197207 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.197173    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzkks\" (UniqueName: \"kubernetes.io/projected/751bad59-841d-4ec7-a16d-90b66fdb38c6-kube-api-access-bzkks\") pod \"console-546d6554f6-6xfzk\" (UID: \"751bad59-841d-4ec7-a16d-90b66fdb38c6\") " pod="openshift-console/console-546d6554f6-6xfzk"
Jul 14 13:30:08.298308 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.298282    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/751bad59-841d-4ec7-a16d-90b66fdb38c6-oauth-serving-cert\") pod \"console-546d6554f6-6xfzk\" (UID: \"751bad59-841d-4ec7-a16d-90b66fdb38c6\") " pod="openshift-console/console-546d6554f6-6xfzk"
Jul 14 13:30:08.298308 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.298309    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzkks\" (UniqueName: \"kubernetes.io/projected/751bad59-841d-4ec7-a16d-90b66fdb38c6-kube-api-access-bzkks\") pod \"console-546d6554f6-6xfzk\" (UID: \"751bad59-841d-4ec7-a16d-90b66fdb38c6\") " pod="openshift-console/console-546d6554f6-6xfzk"
Jul 14 13:30:08.298449 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.298332    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/751bad59-841d-4ec7-a16d-90b66fdb38c6-service-ca\") pod \"console-546d6554f6-6xfzk\" (UID: \"751bad59-841d-4ec7-a16d-90b66fdb38c6\") " pod="openshift-console/console-546d6554f6-6xfzk"
Jul 14 13:30:08.298449 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.298349    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/751bad59-841d-4ec7-a16d-90b66fdb38c6-trusted-ca-bundle\") pod \"console-546d6554f6-6xfzk\" (UID: \"751bad59-841d-4ec7-a16d-90b66fdb38c6\") " pod="openshift-console/console-546d6554f6-6xfzk"
Jul 14 13:30:08.298449 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.298384    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/751bad59-841d-4ec7-a16d-90b66fdb38c6-console-oauth-config\") pod \"console-546d6554f6-6xfzk\" (UID: \"751bad59-841d-4ec7-a16d-90b66fdb38c6\") " pod="openshift-console/console-546d6554f6-6xfzk"
Jul 14 13:30:08.298449 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.298424    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/751bad59-841d-4ec7-a16d-90b66fdb38c6-console-serving-cert\") pod \"console-546d6554f6-6xfzk\" (UID: \"751bad59-841d-4ec7-a16d-90b66fdb38c6\") " pod="openshift-console/console-546d6554f6-6xfzk"
Jul 14 13:30:08.298634 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.298472    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/751bad59-841d-4ec7-a16d-90b66fdb38c6-console-config\") pod \"console-546d6554f6-6xfzk\" (UID: \"751bad59-841d-4ec7-a16d-90b66fdb38c6\") " pod="openshift-console/console-546d6554f6-6xfzk"
Jul 14 13:30:08.299085 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.299057    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/751bad59-841d-4ec7-a16d-90b66fdb38c6-service-ca\") pod \"console-546d6554f6-6xfzk\" (UID: \"751bad59-841d-4ec7-a16d-90b66fdb38c6\") " pod="openshift-console/console-546d6554f6-6xfzk"
Jul 14 13:30:08.299186 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.299083    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/751bad59-841d-4ec7-a16d-90b66fdb38c6-oauth-serving-cert\") pod \"console-546d6554f6-6xfzk\" (UID: \"751bad59-841d-4ec7-a16d-90b66fdb38c6\") " pod="openshift-console/console-546d6554f6-6xfzk"
Jul 14 13:30:08.299240 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.299206    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/751bad59-841d-4ec7-a16d-90b66fdb38c6-console-config\") pod \"console-546d6554f6-6xfzk\" (UID: \"751bad59-841d-4ec7-a16d-90b66fdb38c6\") " pod="openshift-console/console-546d6554f6-6xfzk"
Jul 14 13:30:08.299294 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.299270    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/751bad59-841d-4ec7-a16d-90b66fdb38c6-trusted-ca-bundle\") pod \"console-546d6554f6-6xfzk\" (UID: \"751bad59-841d-4ec7-a16d-90b66fdb38c6\") " pod="openshift-console/console-546d6554f6-6xfzk"
Jul 14 13:30:08.300939 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.300911    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/751bad59-841d-4ec7-a16d-90b66fdb38c6-console-serving-cert\") pod \"console-546d6554f6-6xfzk\" (UID: \"751bad59-841d-4ec7-a16d-90b66fdb38c6\") " pod="openshift-console/console-546d6554f6-6xfzk"
Jul 14 13:30:08.301055 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.300936    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/751bad59-841d-4ec7-a16d-90b66fdb38c6-console-oauth-config\") pod \"console-546d6554f6-6xfzk\" (UID: \"751bad59-841d-4ec7-a16d-90b66fdb38c6\") " pod="openshift-console/console-546d6554f6-6xfzk"
Jul 14 13:30:08.309261 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.309235    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzkks\" (UniqueName: \"kubernetes.io/projected/751bad59-841d-4ec7-a16d-90b66fdb38c6-kube-api-access-bzkks\") pod \"console-546d6554f6-6xfzk\" (UID: \"751bad59-841d-4ec7-a16d-90b66fdb38c6\") " pod="openshift-console/console-546d6554f6-6xfzk"
Jul 14 13:30:08.456930 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.456874    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546d6554f6-6xfzk"
Jul 14 13:30:08.581565 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.581534    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-546d6554f6-6xfzk"]
Jul 14 13:30:08.584001 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:30:08.583976    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod751bad59_841d_4ec7_a16d_90b66fdb38c6.slice/crio-5bfb45c55beafdd959668a70346b6b3c42d08c882197f7a748a4bcd5d565a227 WatchSource:0}: Error finding container 5bfb45c55beafdd959668a70346b6b3c42d08c882197f7a748a4bcd5d565a227: Status 404 returned error can't find the container with id 5bfb45c55beafdd959668a70346b6b3c42d08c882197f7a748a4bcd5d565a227
Jul 14 13:30:08.913937 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.913908    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546d6554f6-6xfzk" event={"ID":"751bad59-841d-4ec7-a16d-90b66fdb38c6","Type":"ContainerStarted","Data":"4cebd9b47428e8f60f81702f7586629523d9512a2e2049f8953dd79e4d660f72"}
Jul 14 13:30:08.914107 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.913952    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546d6554f6-6xfzk" event={"ID":"751bad59-841d-4ec7-a16d-90b66fdb38c6","Type":"ContainerStarted","Data":"5bfb45c55beafdd959668a70346b6b3c42d08c882197f7a748a4bcd5d565a227"}
Jul 14 13:30:08.937656 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:08.937611    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-546d6554f6-6xfzk" podStartSLOduration=0.937595807 podStartE2EDuration="937.595807ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:30:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:30:08.936739155 +0000 UTC m=+174.263059085" watchObservedRunningTime="2026-07-14 13:30:08.937595807 +0000 UTC m=+174.263915735"
Jul 14 13:30:18.457303 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:18.457259    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-546d6554f6-6xfzk"
Jul 14 13:30:18.457730 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:18.457344    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-546d6554f6-6xfzk"
Jul 14 13:30:18.462003 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:18.461983    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-546d6554f6-6xfzk"
Jul 14 13:30:18.952671 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:18.952644    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-546d6554f6-6xfzk"
Jul 14 13:30:19.008907 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:19.008880    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-858f9df6bf-b69bj"]
Jul 14 13:30:44.029187 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:44.029122    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-858f9df6bf-b69bj" podUID="9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6" containerName="console" containerID="cri-o://7c933318498c24f531f945d4a194f00ff8f99bac7e0ddb76ec69fbdbc8d3c36d" gracePeriod=15
Jul 14 13:30:44.269910 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:44.269888    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858f9df6bf-b69bj"
Jul 14 13:30:44.447630 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:44.447562    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-trusted-ca-bundle\") pod \"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6\" (UID: \"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6\") "
Jul 14 13:30:44.447630 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:44.447604    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-oauth-serving-cert\") pod \"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6\" (UID: \"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6\") "
Jul 14 13:30:44.447630 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:44.447627    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-console-oauth-config\") pod \"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6\" (UID: \"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6\") "
Jul 14 13:30:44.447861 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:44.447660    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-console-config\") pod \"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6\" (UID: \"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6\") "
Jul 14 13:30:44.447861 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:44.447691    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-service-ca\") pod \"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6\" (UID: \"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6\") "
Jul 14 13:30:44.447861 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:44.447729    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-console-serving-cert\") pod \"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6\" (UID: \"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6\") "
Jul 14 13:30:44.447861 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:44.447757    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ck7k7\" (UniqueName: \"kubernetes.io/projected/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-kube-api-access-ck7k7\") pod \"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6\" (UID: \"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6\") "
Jul 14 13:30:44.448097 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:44.447924    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6" (UID: "9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 13:30:44.448097 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:44.448045    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6" (UID: "9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 13:30:44.448200 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:44.448168    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-service-ca" (OuterVolumeSpecName: "service-ca") pod "9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6" (UID: "9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 13:30:44.448292 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:44.448262    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-console-config" (OuterVolumeSpecName: "console-config") pod "9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6" (UID: "9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 13:30:44.449774 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:44.449741    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-kube-api-access-ck7k7" (OuterVolumeSpecName: "kube-api-access-ck7k7") pod "9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6" (UID: "9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6"). InnerVolumeSpecName "kube-api-access-ck7k7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 13:30:44.449878 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:44.449826    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6" (UID: "9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 13:30:44.449933 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:44.449869    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6" (UID: "9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 13:30:44.548785 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:44.548759    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-trusted-ca-bundle\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:30:44.548785 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:44.548781    2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-oauth-serving-cert\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:30:44.548898 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:44.548790    2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-console-oauth-config\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:30:44.548898 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:44.548799    2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-console-config\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:30:44.548898 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:44.548807    2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-service-ca\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:30:44.548898 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:44.548815    2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-console-serving-cert\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:30:44.548898 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:44.548823    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck7k7\" (UniqueName: \"kubernetes.io/projected/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6-kube-api-access-ck7k7\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:30:45.039372 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:45.039338    2569 generic.go:356] "Generic (PLEG): container finished" podID="9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6" containerID="7c933318498c24f531f945d4a194f00ff8f99bac7e0ddb76ec69fbdbc8d3c36d" exitCode=0
Jul 14 13:30:45.039762 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:45.039393    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858f9df6bf-b69bj"
Jul 14 13:30:45.039762 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:45.039410    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858f9df6bf-b69bj" event={"ID":"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6","Type":"ContainerDied","Data":"7c933318498c24f531f945d4a194f00ff8f99bac7e0ddb76ec69fbdbc8d3c36d"}
Jul 14 13:30:45.039762 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:45.039472    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858f9df6bf-b69bj" event={"ID":"9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6","Type":"ContainerDied","Data":"29567fd363520148220d244c5348a49c5d0ab6d2b27663e07613bbd6c778b52e"}
Jul 14 13:30:45.039762 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:45.039490    2569 scope.go:117] "RemoveContainer" containerID="7c933318498c24f531f945d4a194f00ff8f99bac7e0ddb76ec69fbdbc8d3c36d"
Jul 14 13:30:45.047768 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:45.047749    2569 scope.go:117] "RemoveContainer" containerID="7c933318498c24f531f945d4a194f00ff8f99bac7e0ddb76ec69fbdbc8d3c36d"
Jul 14 13:30:45.048047 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:30:45.048005    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c933318498c24f531f945d4a194f00ff8f99bac7e0ddb76ec69fbdbc8d3c36d\": container with ID starting with 7c933318498c24f531f945d4a194f00ff8f99bac7e0ddb76ec69fbdbc8d3c36d not found: ID does not exist" containerID="7c933318498c24f531f945d4a194f00ff8f99bac7e0ddb76ec69fbdbc8d3c36d"
Jul 14 13:30:45.048102 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:45.048055    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c933318498c24f531f945d4a194f00ff8f99bac7e0ddb76ec69fbdbc8d3c36d"} err="failed to get container status \"7c933318498c24f531f945d4a194f00ff8f99bac7e0ddb76ec69fbdbc8d3c36d\": rpc error: code = NotFound desc = could not find container \"7c933318498c24f531f945d4a194f00ff8f99bac7e0ddb76ec69fbdbc8d3c36d\": container with ID starting with 7c933318498c24f531f945d4a194f00ff8f99bac7e0ddb76ec69fbdbc8d3c36d not found: ID does not exist"
Jul 14 13:30:45.062320 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:45.062286    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-858f9df6bf-b69bj"]
Jul 14 13:30:45.067549 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:45.067532    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-858f9df6bf-b69bj"]
Jul 14 13:30:45.191080 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:45.191056    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6" path="/var/lib/kubelet/pods/9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6/volumes"
Jul 14 13:30:58.296457 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:58.296424    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpmrp"]
Jul 14 13:30:58.296850 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:58.296685    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6" containerName="console"
Jul 14 13:30:58.296850 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:58.296696    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6" containerName="console"
Jul 14 13:30:58.296850 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:58.296748    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fc33b51-67c9-49d7-8d9d-3e3dc1cb12e6" containerName="console"
Jul 14 13:30:58.299586 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:58.299568    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpmrp"
Jul 14 13:30:58.303062 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:58.303042    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 14 13:30:58.304133 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:58.304112    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-drmvc\""
Jul 14 13:30:58.304226 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:58.304178    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 14 13:30:58.310789 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:58.310768    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpmrp"]
Jul 14 13:30:58.434036 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:58.433993    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdwd4\" (UniqueName: \"kubernetes.io/projected/343c5c07-0a5d-42b8-9e89-c3de9bcad186-kube-api-access-jdwd4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpmrp\" (UID: \"343c5c07-0a5d-42b8-9e89-c3de9bcad186\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpmrp"
Jul 14 13:30:58.434168 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:58.434043    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/343c5c07-0a5d-42b8-9e89-c3de9bcad186-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpmrp\" (UID: \"343c5c07-0a5d-42b8-9e89-c3de9bcad186\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpmrp"
Jul 14 13:30:58.434168 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:58.434151    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/343c5c07-0a5d-42b8-9e89-c3de9bcad186-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpmrp\" (UID: \"343c5c07-0a5d-42b8-9e89-c3de9bcad186\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpmrp"
Jul 14 13:30:58.534887 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:58.534863    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/343c5c07-0a5d-42b8-9e89-c3de9bcad186-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpmrp\" (UID: \"343c5c07-0a5d-42b8-9e89-c3de9bcad186\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpmrp"
Jul 14 13:30:58.534991 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:58.534908    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdwd4\" (UniqueName: \"kubernetes.io/projected/343c5c07-0a5d-42b8-9e89-c3de9bcad186-kube-api-access-jdwd4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpmrp\" (UID: \"343c5c07-0a5d-42b8-9e89-c3de9bcad186\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpmrp"
Jul 14 13:30:58.534991 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:58.534928    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/343c5c07-0a5d-42b8-9e89-c3de9bcad186-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpmrp\" (UID: \"343c5c07-0a5d-42b8-9e89-c3de9bcad186\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpmrp"
Jul 14 13:30:58.535314 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:58.535295    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/343c5c07-0a5d-42b8-9e89-c3de9bcad186-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpmrp\" (UID: \"343c5c07-0a5d-42b8-9e89-c3de9bcad186\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpmrp"
Jul 14 13:30:58.535350 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:58.535301    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/343c5c07-0a5d-42b8-9e89-c3de9bcad186-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpmrp\" (UID: \"343c5c07-0a5d-42b8-9e89-c3de9bcad186\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpmrp"
Jul 14 13:30:58.544263 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:58.544245    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdwd4\" (UniqueName: \"kubernetes.io/projected/343c5c07-0a5d-42b8-9e89-c3de9bcad186-kube-api-access-jdwd4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpmrp\" (UID: \"343c5c07-0a5d-42b8-9e89-c3de9bcad186\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpmrp"
Jul 14 13:30:58.608504 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:58.608454    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpmrp"
Jul 14 13:30:58.730279 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:58.730248    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpmrp"]
Jul 14 13:30:58.733541 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:30:58.733508    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod343c5c07_0a5d_42b8_9e89_c3de9bcad186.slice/crio-dd06355396661686a1b19675ae9c2280d66f3b2f2ef6fecaa271cd05047072b1 WatchSource:0}: Error finding container dd06355396661686a1b19675ae9c2280d66f3b2f2ef6fecaa271cd05047072b1: Status 404 returned error can't find the container with id dd06355396661686a1b19675ae9c2280d66f3b2f2ef6fecaa271cd05047072b1
Jul 14 13:30:59.087717 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:30:59.087685    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpmrp" event={"ID":"343c5c07-0a5d-42b8-9e89-c3de9bcad186","Type":"ContainerStarted","Data":"dd06355396661686a1b19675ae9c2280d66f3b2f2ef6fecaa271cd05047072b1"}
Jul 14 13:31:06.113355 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:06.113323    2569 generic.go:356] "Generic (PLEG): container finished" podID="343c5c07-0a5d-42b8-9e89-c3de9bcad186" containerID="d1016ca364d47e27285afe42f0af4fe6119a3bf8cbd0cf13dcb9adb50f634d8c" exitCode=0
Jul 14 13:31:06.113655 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:06.113409    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpmrp" event={"ID":"343c5c07-0a5d-42b8-9e89-c3de9bcad186","Type":"ContainerDied","Data":"d1016ca364d47e27285afe42f0af4fe6119a3bf8cbd0cf13dcb9adb50f634d8c"}
Jul 14 13:31:08.122589 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:08.122554    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpmrp" event={"ID":"343c5c07-0a5d-42b8-9e89-c3de9bcad186","Type":"ContainerStarted","Data":"55cf56939140677479d0976fb172df5cb068e80873f95cb4d0a02bbd802dd6c0"}
Jul 14 13:31:09.128751 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:09.128721    2569 generic.go:356] "Generic (PLEG): container finished" podID="343c5c07-0a5d-42b8-9e89-c3de9bcad186" containerID="55cf56939140677479d0976fb172df5cb068e80873f95cb4d0a02bbd802dd6c0" exitCode=0
Jul 14 13:31:09.129155 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:09.128765    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpmrp" event={"ID":"343c5c07-0a5d-42b8-9e89-c3de9bcad186","Type":"ContainerDied","Data":"55cf56939140677479d0976fb172df5cb068e80873f95cb4d0a02bbd802dd6c0"}
Jul 14 13:31:18.164296 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:18.164261    2569 generic.go:356] "Generic (PLEG): container finished" podID="343c5c07-0a5d-42b8-9e89-c3de9bcad186" containerID="e44f3bd0f66597f9b59cda59f58726bf854181f5bd3f5af9f984baaa8142c971" exitCode=0
Jul 14 13:31:18.164677 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:18.164343    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpmrp" event={"ID":"343c5c07-0a5d-42b8-9e89-c3de9bcad186","Type":"ContainerDied","Data":"e44f3bd0f66597f9b59cda59f58726bf854181f5bd3f5af9f984baaa8142c971"}
Jul 14 13:31:19.295197 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:19.295176    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpmrp"
Jul 14 13:31:19.391923 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:19.391894    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/343c5c07-0a5d-42b8-9e89-c3de9bcad186-bundle\") pod \"343c5c07-0a5d-42b8-9e89-c3de9bcad186\" (UID: \"343c5c07-0a5d-42b8-9e89-c3de9bcad186\") "
Jul 14 13:31:19.392083 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:19.391941    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdwd4\" (UniqueName: \"kubernetes.io/projected/343c5c07-0a5d-42b8-9e89-c3de9bcad186-kube-api-access-jdwd4\") pod \"343c5c07-0a5d-42b8-9e89-c3de9bcad186\" (UID: \"343c5c07-0a5d-42b8-9e89-c3de9bcad186\") "
Jul 14 13:31:19.392083 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:19.391957    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/343c5c07-0a5d-42b8-9e89-c3de9bcad186-util\") pod \"343c5c07-0a5d-42b8-9e89-c3de9bcad186\" (UID: \"343c5c07-0a5d-42b8-9e89-c3de9bcad186\") "
Jul 14 13:31:19.392572 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:19.392547    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/343c5c07-0a5d-42b8-9e89-c3de9bcad186-bundle" (OuterVolumeSpecName: "bundle") pod "343c5c07-0a5d-42b8-9e89-c3de9bcad186" (UID: "343c5c07-0a5d-42b8-9e89-c3de9bcad186"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 13:31:19.393995 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:19.393971    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/343c5c07-0a5d-42b8-9e89-c3de9bcad186-kube-api-access-jdwd4" (OuterVolumeSpecName: "kube-api-access-jdwd4") pod "343c5c07-0a5d-42b8-9e89-c3de9bcad186" (UID: "343c5c07-0a5d-42b8-9e89-c3de9bcad186"). InnerVolumeSpecName "kube-api-access-jdwd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 13:31:19.396261 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:19.396226    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/343c5c07-0a5d-42b8-9e89-c3de9bcad186-util" (OuterVolumeSpecName: "util") pod "343c5c07-0a5d-42b8-9e89-c3de9bcad186" (UID: "343c5c07-0a5d-42b8-9e89-c3de9bcad186"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 13:31:19.492594 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:19.492574    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdwd4\" (UniqueName: \"kubernetes.io/projected/343c5c07-0a5d-42b8-9e89-c3de9bcad186-kube-api-access-jdwd4\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:31:19.492687 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:19.492597    2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/343c5c07-0a5d-42b8-9e89-c3de9bcad186-util\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:31:19.492687 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:19.492612    2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/343c5c07-0a5d-42b8-9e89-c3de9bcad186-bundle\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:31:20.172989 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:20.172952    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpmrp" event={"ID":"343c5c07-0a5d-42b8-9e89-c3de9bcad186","Type":"ContainerDied","Data":"dd06355396661686a1b19675ae9c2280d66f3b2f2ef6fecaa271cd05047072b1"}
Jul 14 13:31:20.172989 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:20.172974    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bpmrp"
Jul 14 13:31:20.173185 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:20.173003    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd06355396661686a1b19675ae9c2280d66f3b2f2ef6fecaa271cd05047072b1"
Jul 14 13:31:26.379141 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:26.379110    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjlkd"]
Jul 14 13:31:26.379511 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:26.379356    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="343c5c07-0a5d-42b8-9e89-c3de9bcad186" containerName="pull"
Jul 14 13:31:26.379511 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:26.379366    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="343c5c07-0a5d-42b8-9e89-c3de9bcad186" containerName="pull"
Jul 14 13:31:26.379511 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:26.379375    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="343c5c07-0a5d-42b8-9e89-c3de9bcad186" containerName="util"
Jul 14 13:31:26.379511 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:26.379380    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="343c5c07-0a5d-42b8-9e89-c3de9bcad186" containerName="util"
Jul 14 13:31:26.379511 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:26.379386    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="343c5c07-0a5d-42b8-9e89-c3de9bcad186" containerName="extract"
Jul 14 13:31:26.379511 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:26.379393    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="343c5c07-0a5d-42b8-9e89-c3de9bcad186" containerName="extract"
Jul 14 13:31:26.379511 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:26.379433    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="343c5c07-0a5d-42b8-9e89-c3de9bcad186" containerName="extract"
Jul 14 13:31:26.382769 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:26.382753    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjlkd"
Jul 14 13:31:26.385867 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:26.385843    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 14 13:31:26.385987 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:26.385910    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 14 13:31:26.385987 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:26.385843    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-hvn6p\""
Jul 14 13:31:26.395438 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:26.395418    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjlkd"]
Jul 14 13:31:26.443099 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:26.443067    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d64b26f-0603-407d-b314-bcc935803db3-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-cjlkd\" (UID: \"3d64b26f-0603-407d-b314-bcc935803db3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjlkd"
Jul 14 13:31:26.443201 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:26.443142    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbzpn\" (UniqueName: \"kubernetes.io/projected/3d64b26f-0603-407d-b314-bcc935803db3-kube-api-access-pbzpn\") pod \"cert-manager-operator-controller-manager-55c444bf86-cjlkd\" (UID: \"3d64b26f-0603-407d-b314-bcc935803db3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjlkd"
Jul 14 13:31:26.543613 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:26.543571    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbzpn\" (UniqueName: \"kubernetes.io/projected/3d64b26f-0603-407d-b314-bcc935803db3-kube-api-access-pbzpn\") pod \"cert-manager-operator-controller-manager-55c444bf86-cjlkd\" (UID: \"3d64b26f-0603-407d-b314-bcc935803db3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjlkd"
Jul 14 13:31:26.543716 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:26.543643    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d64b26f-0603-407d-b314-bcc935803db3-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-cjlkd\" (UID: \"3d64b26f-0603-407d-b314-bcc935803db3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjlkd"
Jul 14 13:31:26.543964 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:26.543946    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d64b26f-0603-407d-b314-bcc935803db3-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-cjlkd\" (UID: \"3d64b26f-0603-407d-b314-bcc935803db3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjlkd"
Jul 14 13:31:26.561801 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:26.561776    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbzpn\" (UniqueName: \"kubernetes.io/projected/3d64b26f-0603-407d-b314-bcc935803db3-kube-api-access-pbzpn\") pod \"cert-manager-operator-controller-manager-55c444bf86-cjlkd\" (UID: \"3d64b26f-0603-407d-b314-bcc935803db3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjlkd"
Jul 14 13:31:26.691843 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:26.691760    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjlkd"
Jul 14 13:31:26.832128 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:26.832092    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjlkd"]
Jul 14 13:31:26.835977 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:31:26.835947    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d64b26f_0603_407d_b314_bcc935803db3.slice/crio-be02161a2ec4bc3439ac323619120720c1fad1af166b16309b19d9884dc3c55c WatchSource:0}: Error finding container be02161a2ec4bc3439ac323619120720c1fad1af166b16309b19d9884dc3c55c: Status 404 returned error can't find the container with id be02161a2ec4bc3439ac323619120720c1fad1af166b16309b19d9884dc3c55c
Jul 14 13:31:27.197819 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:27.197792    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjlkd" event={"ID":"3d64b26f-0603-407d-b314-bcc935803db3","Type":"ContainerStarted","Data":"be02161a2ec4bc3439ac323619120720c1fad1af166b16309b19d9884dc3c55c"}
Jul 14 13:31:29.212202 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:29.212154    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjlkd" event={"ID":"3d64b26f-0603-407d-b314-bcc935803db3","Type":"ContainerStarted","Data":"470a286c9276fb0a29f41ae977b1d6f88bf140e536f964daa9e378aa2ee79fef"}
Jul 14 13:31:29.239324 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:29.239277    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjlkd" podStartSLOduration=1.7090694659999999 podStartE2EDuration="3.23926078s" totalImagesPullingTime="1.530191314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:31:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:31:29.236780829 +0000 UTC m=+254.563100774" watchObservedRunningTime="2026-07-14 13:31:29.23926078 +0000 UTC m=+254.565580707"
Jul 14 13:31:30.504099 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:30.504067    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47fjf"]
Jul 14 13:31:30.506345 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:30.506328    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47fjf"
Jul 14 13:31:30.510203 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:30.510178    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 14 13:31:30.510333 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:30.510216    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-drmvc\""
Jul 14 13:31:30.511402 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:30.511377    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 14 13:31:30.523292 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:30.523270    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47fjf"]
Jul 14 13:31:30.575261 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:30.575231    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d00cdaa1-93e9-4bfd-bab3-db6562e431d9-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47fjf\" (UID: \"d00cdaa1-93e9-4bfd-bab3-db6562e431d9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47fjf"
Jul 14 13:31:30.575374 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:30.575273    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d00cdaa1-93e9-4bfd-bab3-db6562e431d9-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47fjf\" (UID: \"d00cdaa1-93e9-4bfd-bab3-db6562e431d9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47fjf"
Jul 14 13:31:30.575374 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:30.575299    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnp89\" (UniqueName: \"kubernetes.io/projected/d00cdaa1-93e9-4bfd-bab3-db6562e431d9-kube-api-access-pnp89\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47fjf\" (UID: \"d00cdaa1-93e9-4bfd-bab3-db6562e431d9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47fjf"
Jul 14 13:31:30.676241 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:30.676212    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnp89\" (UniqueName: \"kubernetes.io/projected/d00cdaa1-93e9-4bfd-bab3-db6562e431d9-kube-api-access-pnp89\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47fjf\" (UID: \"d00cdaa1-93e9-4bfd-bab3-db6562e431d9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47fjf"
Jul 14 13:31:30.676347 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:30.676261    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d00cdaa1-93e9-4bfd-bab3-db6562e431d9-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47fjf\" (UID: \"d00cdaa1-93e9-4bfd-bab3-db6562e431d9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47fjf"
Jul 14 13:31:30.676347 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:30.676289    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d00cdaa1-93e9-4bfd-bab3-db6562e431d9-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47fjf\" (UID: \"d00cdaa1-93e9-4bfd-bab3-db6562e431d9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47fjf"
Jul 14 13:31:30.676600 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:30.676586    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d00cdaa1-93e9-4bfd-bab3-db6562e431d9-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47fjf\" (UID: \"d00cdaa1-93e9-4bfd-bab3-db6562e431d9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47fjf"
Jul 14 13:31:30.676651 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:30.676621    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d00cdaa1-93e9-4bfd-bab3-db6562e431d9-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47fjf\" (UID: \"d00cdaa1-93e9-4bfd-bab3-db6562e431d9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47fjf"
Jul 14 13:31:30.686672 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:30.686645    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnp89\" (UniqueName: \"kubernetes.io/projected/d00cdaa1-93e9-4bfd-bab3-db6562e431d9-kube-api-access-pnp89\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47fjf\" (UID: \"d00cdaa1-93e9-4bfd-bab3-db6562e431d9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47fjf"
Jul 14 13:31:30.814542 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:30.814473    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47fjf"
Jul 14 13:31:30.937020 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:30.936981    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47fjf"]
Jul 14 13:31:30.938755 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:31:30.938732    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd00cdaa1_93e9_4bfd_bab3_db6562e431d9.slice/crio-af6f710117e7a49678afe187330cd80e222284ba89eb3ee309f2ed5a95e50c28 WatchSource:0}: Error finding container af6f710117e7a49678afe187330cd80e222284ba89eb3ee309f2ed5a95e50c28: Status 404 returned error can't find the container with id af6f710117e7a49678afe187330cd80e222284ba89eb3ee309f2ed5a95e50c28
Jul 14 13:31:31.220557 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:31.220484    2569 generic.go:356] "Generic (PLEG): container finished" podID="d00cdaa1-93e9-4bfd-bab3-db6562e431d9" containerID="0261f9b34ed4b1aaa6f35a565d87bce2cb4e0aca96324405396c41056d45ea95" exitCode=0
Jul 14 13:31:31.220557 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:31.220548    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47fjf" event={"ID":"d00cdaa1-93e9-4bfd-bab3-db6562e431d9","Type":"ContainerDied","Data":"0261f9b34ed4b1aaa6f35a565d87bce2cb4e0aca96324405396c41056d45ea95"}
Jul 14 13:31:31.220697 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:31.220589    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47fjf" event={"ID":"d00cdaa1-93e9-4bfd-bab3-db6562e431d9","Type":"ContainerStarted","Data":"af6f710117e7a49678afe187330cd80e222284ba89eb3ee309f2ed5a95e50c28"}
Jul 14 13:31:32.067388 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:32.067355    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zdhxk"]
Jul 14 13:31:32.069714 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:32.069688    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zdhxk"
Jul 14 13:31:32.073900 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:32.073876    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 14 13:31:32.074033 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:32.073935    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 14 13:31:32.075165 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:32.075147    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-76m25\""
Jul 14 13:31:32.087833 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:32.087813    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zdhxk"]
Jul 14 13:31:32.187891 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:32.187854    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7rh2\" (UniqueName: \"kubernetes.io/projected/f2bc0eb5-76c4-416c-b604-b4b99d5a74cc-kube-api-access-c7rh2\") pod \"cert-manager-cainjector-559cdd8c87-zdhxk\" (UID: \"f2bc0eb5-76c4-416c-b604-b4b99d5a74cc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zdhxk"
Jul 14 13:31:32.188057 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:32.187926    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f2bc0eb5-76c4-416c-b604-b4b99d5a74cc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zdhxk\" (UID: \"f2bc0eb5-76c4-416c-b604-b4b99d5a74cc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zdhxk"
Jul 14 13:31:32.288636 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:32.288604    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7rh2\" (UniqueName: \"kubernetes.io/projected/f2bc0eb5-76c4-416c-b604-b4b99d5a74cc-kube-api-access-c7rh2\") pod \"cert-manager-cainjector-559cdd8c87-zdhxk\" (UID: \"f2bc0eb5-76c4-416c-b604-b4b99d5a74cc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zdhxk"
Jul 14 13:31:32.288794 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:32.288717    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f2bc0eb5-76c4-416c-b604-b4b99d5a74cc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zdhxk\" (UID: \"f2bc0eb5-76c4-416c-b604-b4b99d5a74cc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zdhxk"
Jul 14 13:31:32.299893 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:32.299867    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7rh2\" (UniqueName: \"kubernetes.io/projected/f2bc0eb5-76c4-416c-b604-b4b99d5a74cc-kube-api-access-c7rh2\") pod \"cert-manager-cainjector-559cdd8c87-zdhxk\" (UID: \"f2bc0eb5-76c4-416c-b604-b4b99d5a74cc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zdhxk"
Jul 14 13:31:32.300747 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:32.300723    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f2bc0eb5-76c4-416c-b604-b4b99d5a74cc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zdhxk\" (UID: \"f2bc0eb5-76c4-416c-b604-b4b99d5a74cc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zdhxk"
Jul 14 13:31:32.391639 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:32.391571    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zdhxk"
Jul 14 13:31:32.537348 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:32.537322    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zdhxk"]
Jul 14 13:31:32.538386 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:31:32.538359    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2bc0eb5_76c4_416c_b604_b4b99d5a74cc.slice/crio-c30b91f1f70ce65a7032f2d267bb233d48b861e2673da1a9f5c1a7d0c2b0585b WatchSource:0}: Error finding container c30b91f1f70ce65a7032f2d267bb233d48b861e2673da1a9f5c1a7d0c2b0585b: Status 404 returned error can't find the container with id c30b91f1f70ce65a7032f2d267bb233d48b861e2673da1a9f5c1a7d0c2b0585b
Jul 14 13:31:33.232156 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:33.232120    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zdhxk" event={"ID":"f2bc0eb5-76c4-416c-b604-b4b99d5a74cc","Type":"ContainerStarted","Data":"c30b91f1f70ce65a7032f2d267bb233d48b861e2673da1a9f5c1a7d0c2b0585b"}
Jul 14 13:31:34.238582 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:34.238544    2569 generic.go:356] "Generic (PLEG): container finished" podID="d00cdaa1-93e9-4bfd-bab3-db6562e431d9" containerID="ee0a49cc55741acf884129829d1dad5aef3baa6ce93b202544a5897763afbfd7" exitCode=0
Jul 14 13:31:34.238961 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:34.238587    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47fjf" event={"ID":"d00cdaa1-93e9-4bfd-bab3-db6562e431d9","Type":"ContainerDied","Data":"ee0a49cc55741acf884129829d1dad5aef3baa6ce93b202544a5897763afbfd7"}
Jul 14 13:31:35.245060 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:35.245001    2569 generic.go:356] "Generic (PLEG): container finished" podID="d00cdaa1-93e9-4bfd-bab3-db6562e431d9" containerID="deb929624daded5bed5ad1a8db8b58028bbf999db2e6cc4cd75182500fa06cce" exitCode=0
Jul 14 13:31:35.245449 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:35.245105    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47fjf" event={"ID":"d00cdaa1-93e9-4bfd-bab3-db6562e431d9","Type":"ContainerDied","Data":"deb929624daded5bed5ad1a8db8b58028bbf999db2e6cc4cd75182500fa06cce"}
Jul 14 13:31:36.249971 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:36.249936    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zdhxk" event={"ID":"f2bc0eb5-76c4-416c-b604-b4b99d5a74cc","Type":"ContainerStarted","Data":"1b77b4198cca3728d7774ec3d11ed87307edf9b450976f688ea39fb279e88536"}
Jul 14 13:31:36.269182 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:36.269131    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zdhxk" podStartSLOduration=1.470534556 podStartE2EDuration="4.269114665s" totalImagesPullingTime="2.798580109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:31:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:31:36.267880074 +0000 UTC m=+261.594200002" watchObservedRunningTime="2026-07-14 13:31:36.269114665 +0000 UTC m=+261.595434593"
Jul 14 13:31:36.374502 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:36.374480    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47fjf"
Jul 14 13:31:36.527067 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:36.526983    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnp89\" (UniqueName: \"kubernetes.io/projected/d00cdaa1-93e9-4bfd-bab3-db6562e431d9-kube-api-access-pnp89\") pod \"d00cdaa1-93e9-4bfd-bab3-db6562e431d9\" (UID: \"d00cdaa1-93e9-4bfd-bab3-db6562e431d9\") "
Jul 14 13:31:36.527067 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:36.527042    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d00cdaa1-93e9-4bfd-bab3-db6562e431d9-bundle\") pod \"d00cdaa1-93e9-4bfd-bab3-db6562e431d9\" (UID: \"d00cdaa1-93e9-4bfd-bab3-db6562e431d9\") "
Jul 14 13:31:36.527223 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:36.527106    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d00cdaa1-93e9-4bfd-bab3-db6562e431d9-util\") pod \"d00cdaa1-93e9-4bfd-bab3-db6562e431d9\" (UID: \"d00cdaa1-93e9-4bfd-bab3-db6562e431d9\") "
Jul 14 13:31:36.527464 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:36.527439    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00cdaa1-93e9-4bfd-bab3-db6562e431d9-bundle" (OuterVolumeSpecName: "bundle") pod "d00cdaa1-93e9-4bfd-bab3-db6562e431d9" (UID: "d00cdaa1-93e9-4bfd-bab3-db6562e431d9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 13:31:36.529107 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:36.529083    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d00cdaa1-93e9-4bfd-bab3-db6562e431d9-kube-api-access-pnp89" (OuterVolumeSpecName: "kube-api-access-pnp89") pod "d00cdaa1-93e9-4bfd-bab3-db6562e431d9" (UID: "d00cdaa1-93e9-4bfd-bab3-db6562e431d9"). InnerVolumeSpecName "kube-api-access-pnp89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 13:31:36.628114 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:36.628088    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnp89\" (UniqueName: \"kubernetes.io/projected/d00cdaa1-93e9-4bfd-bab3-db6562e431d9-kube-api-access-pnp89\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:31:36.628114 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:36.628113    2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d00cdaa1-93e9-4bfd-bab3-db6562e431d9-bundle\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:31:36.737756 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:36.737704    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00cdaa1-93e9-4bfd-bab3-db6562e431d9-util" (OuterVolumeSpecName: "util") pod "d00cdaa1-93e9-4bfd-bab3-db6562e431d9" (UID: "d00cdaa1-93e9-4bfd-bab3-db6562e431d9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 13:31:36.830296 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:36.830239    2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d00cdaa1-93e9-4bfd-bab3-db6562e431d9-util\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:31:37.255867 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:37.255845    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47fjf"
Jul 14 13:31:37.256295 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:37.255877    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f47fjf" event={"ID":"d00cdaa1-93e9-4bfd-bab3-db6562e431d9","Type":"ContainerDied","Data":"af6f710117e7a49678afe187330cd80e222284ba89eb3ee309f2ed5a95e50c28"}
Jul 14 13:31:37.256295 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:37.255920    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af6f710117e7a49678afe187330cd80e222284ba89eb3ee309f2ed5a95e50c28"
Jul 14 13:31:48.584732 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:48.584701    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznxx4"]
Jul 14 13:31:48.585233 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:48.585089    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d00cdaa1-93e9-4bfd-bab3-db6562e431d9" containerName="pull"
Jul 14 13:31:48.585233 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:48.585107    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d00cdaa1-93e9-4bfd-bab3-db6562e431d9" containerName="pull"
Jul 14 13:31:48.585233 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:48.585125    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d00cdaa1-93e9-4bfd-bab3-db6562e431d9" containerName="util"
Jul 14 13:31:48.585233 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:48.585133    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d00cdaa1-93e9-4bfd-bab3-db6562e431d9" containerName="util"
Jul 14 13:31:48.585233 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:48.585150    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d00cdaa1-93e9-4bfd-bab3-db6562e431d9" containerName="extract"
Jul 14 13:31:48.585233 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:48.585158    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d00cdaa1-93e9-4bfd-bab3-db6562e431d9" containerName="extract"
Jul 14 13:31:48.585233 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:48.585231    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d00cdaa1-93e9-4bfd-bab3-db6562e431d9" containerName="extract"
Jul 14 13:31:48.587424 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:48.587403    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznxx4"
Jul 14 13:31:48.590935 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:48.590916    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-drmvc\""
Jul 14 13:31:48.591192 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:48.591170    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 14 13:31:48.592155 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:48.592140    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 14 13:31:48.597787 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:48.597762    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznxx4"]
Jul 14 13:31:48.713019 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:48.712992    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40df30ab-86bf-43b1-ac06-b4a53cb86daa-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznxx4\" (UID: \"40df30ab-86bf-43b1-ac06-b4a53cb86daa\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznxx4"
Jul 14 13:31:48.713146 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:48.713029    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7xhr\" (UniqueName: \"kubernetes.io/projected/40df30ab-86bf-43b1-ac06-b4a53cb86daa-kube-api-access-r7xhr\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznxx4\" (UID: \"40df30ab-86bf-43b1-ac06-b4a53cb86daa\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznxx4"
Jul 14 13:31:48.713146 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:48.713124    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40df30ab-86bf-43b1-ac06-b4a53cb86daa-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznxx4\" (UID: \"40df30ab-86bf-43b1-ac06-b4a53cb86daa\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznxx4"
Jul 14 13:31:48.814337 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:48.814300    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40df30ab-86bf-43b1-ac06-b4a53cb86daa-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznxx4\" (UID: \"40df30ab-86bf-43b1-ac06-b4a53cb86daa\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznxx4"
Jul 14 13:31:48.814465 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:48.814353    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40df30ab-86bf-43b1-ac06-b4a53cb86daa-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznxx4\" (UID: \"40df30ab-86bf-43b1-ac06-b4a53cb86daa\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznxx4"
Jul 14 13:31:48.814465 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:48.814374    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7xhr\" (UniqueName: \"kubernetes.io/projected/40df30ab-86bf-43b1-ac06-b4a53cb86daa-kube-api-access-r7xhr\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznxx4\" (UID: \"40df30ab-86bf-43b1-ac06-b4a53cb86daa\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznxx4"
Jul 14 13:31:48.814672 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:48.814651    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40df30ab-86bf-43b1-ac06-b4a53cb86daa-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznxx4\" (UID: \"40df30ab-86bf-43b1-ac06-b4a53cb86daa\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznxx4"
Jul 14 13:31:48.814719 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:48.814694    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40df30ab-86bf-43b1-ac06-b4a53cb86daa-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznxx4\" (UID: \"40df30ab-86bf-43b1-ac06-b4a53cb86daa\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznxx4"
Jul 14 13:31:48.826431 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:48.825064    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7xhr\" (UniqueName: \"kubernetes.io/projected/40df30ab-86bf-43b1-ac06-b4a53cb86daa-kube-api-access-r7xhr\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznxx4\" (UID: \"40df30ab-86bf-43b1-ac06-b4a53cb86daa\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznxx4"
Jul 14 13:31:48.896053 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:48.895986    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznxx4"
Jul 14 13:31:49.024162 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:49.024137    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznxx4"]
Jul 14 13:31:49.025697 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:31:49.025666    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40df30ab_86bf_43b1_ac06_b4a53cb86daa.slice/crio-52eb3f0fc488477f578c65a59899b2f6e0cf644dfd22bb13e0966bb7d78370f7 WatchSource:0}: Error finding container 52eb3f0fc488477f578c65a59899b2f6e0cf644dfd22bb13e0966bb7d78370f7: Status 404 returned error can't find the container with id 52eb3f0fc488477f578c65a59899b2f6e0cf644dfd22bb13e0966bb7d78370f7
Jul 14 13:31:49.301805 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:49.301774    2569 generic.go:356] "Generic (PLEG): container finished" podID="40df30ab-86bf-43b1-ac06-b4a53cb86daa" containerID="5833e59af285b9838733c8a6c7fdb7a73cca819d66ccf739b8aa5d2a124ee567" exitCode=0
Jul 14 13:31:49.301929 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:49.301839    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznxx4" event={"ID":"40df30ab-86bf-43b1-ac06-b4a53cb86daa","Type":"ContainerDied","Data":"5833e59af285b9838733c8a6c7fdb7a73cca819d66ccf739b8aa5d2a124ee567"}
Jul 14 13:31:49.301929 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:49.301881    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznxx4" event={"ID":"40df30ab-86bf-43b1-ac06-b4a53cb86daa","Type":"ContainerStarted","Data":"52eb3f0fc488477f578c65a59899b2f6e0cf644dfd22bb13e0966bb7d78370f7"}
Jul 14 13:31:49.594708 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:49.594631    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-wc858"]
Jul 14 13:31:49.596685 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:49.596666    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-wc858"
Jul 14 13:31:49.599112 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:49.599089    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-wpbgz\""
Jul 14 13:31:49.609044 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:49.609022    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-wc858"]
Jul 14 13:31:49.721829 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:49.721804    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ffa22ad-71f0-4bf3-bc16-64d469fc2b37-bound-sa-token\") pod \"cert-manager-55b4d954d5-wc858\" (UID: \"2ffa22ad-71f0-4bf3-bc16-64d469fc2b37\") " pod="cert-manager/cert-manager-55b4d954d5-wc858"
Jul 14 13:31:49.721941 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:49.721835    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvbzq\" (UniqueName: \"kubernetes.io/projected/2ffa22ad-71f0-4bf3-bc16-64d469fc2b37-kube-api-access-lvbzq\") pod \"cert-manager-55b4d954d5-wc858\" (UID: \"2ffa22ad-71f0-4bf3-bc16-64d469fc2b37\") " pod="cert-manager/cert-manager-55b4d954d5-wc858"
Jul 14 13:31:49.823120 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:49.823087    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ffa22ad-71f0-4bf3-bc16-64d469fc2b37-bound-sa-token\") pod \"cert-manager-55b4d954d5-wc858\" (UID: \"2ffa22ad-71f0-4bf3-bc16-64d469fc2b37\") " pod="cert-manager/cert-manager-55b4d954d5-wc858"
Jul 14 13:31:49.823275 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:49.823131    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvbzq\" (UniqueName: \"kubernetes.io/projected/2ffa22ad-71f0-4bf3-bc16-64d469fc2b37-kube-api-access-lvbzq\") pod \"cert-manager-55b4d954d5-wc858\" (UID: \"2ffa22ad-71f0-4bf3-bc16-64d469fc2b37\") " pod="cert-manager/cert-manager-55b4d954d5-wc858"
Jul 14 13:31:49.833891 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:49.833863    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ffa22ad-71f0-4bf3-bc16-64d469fc2b37-bound-sa-token\") pod \"cert-manager-55b4d954d5-wc858\" (UID: \"2ffa22ad-71f0-4bf3-bc16-64d469fc2b37\") " pod="cert-manager/cert-manager-55b4d954d5-wc858"
Jul 14 13:31:49.834266 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:49.834241    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvbzq\" (UniqueName: \"kubernetes.io/projected/2ffa22ad-71f0-4bf3-bc16-64d469fc2b37-kube-api-access-lvbzq\") pod \"cert-manager-55b4d954d5-wc858\" (UID: \"2ffa22ad-71f0-4bf3-bc16-64d469fc2b37\") " pod="cert-manager/cert-manager-55b4d954d5-wc858"
Jul 14 13:31:49.905633 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:49.905584    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-wc858"
Jul 14 13:31:50.061373 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:50.061348    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-wc858"]
Jul 14 13:31:50.079725 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:31:50.079701    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ffa22ad_71f0_4bf3_bc16_64d469fc2b37.slice/crio-1b785a5a7fce094d25936d7acd331ba18f88e59cb0921578f53e78da055f97d5 WatchSource:0}: Error finding container 1b785a5a7fce094d25936d7acd331ba18f88e59cb0921578f53e78da055f97d5: Status 404 returned error can't find the container with id 1b785a5a7fce094d25936d7acd331ba18f88e59cb0921578f53e78da055f97d5
Jul 14 13:31:50.307483 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:50.307450    2569 generic.go:356] "Generic (PLEG): container finished" podID="40df30ab-86bf-43b1-ac06-b4a53cb86daa" containerID="669960896df60a3d7fd65620a7cbe0ced365a7a2e1e44af0ba6daf19f68382be" exitCode=0
Jul 14 13:31:50.307648 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:50.307533    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznxx4" event={"ID":"40df30ab-86bf-43b1-ac06-b4a53cb86daa","Type":"ContainerDied","Data":"669960896df60a3d7fd65620a7cbe0ced365a7a2e1e44af0ba6daf19f68382be"}
Jul 14 13:31:50.309065 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:50.309039    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-wc858" event={"ID":"2ffa22ad-71f0-4bf3-bc16-64d469fc2b37","Type":"ContainerStarted","Data":"76de095f38b8242c099fe1d58c5283d2ae5da9e7ef15f67678babd73e23d706a"}
Jul 14 13:31:50.309176 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:50.309072    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-wc858" event={"ID":"2ffa22ad-71f0-4bf3-bc16-64d469fc2b37","Type":"ContainerStarted","Data":"1b785a5a7fce094d25936d7acd331ba18f88e59cb0921578f53e78da055f97d5"}
Jul 14 13:31:50.346081 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:50.346042    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-wc858" podStartSLOduration=1.3460289890000001 podStartE2EDuration="1.346028989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:31:50.344803763 +0000 UTC m=+275.671123683" watchObservedRunningTime="2026-07-14 13:31:50.346028989 +0000 UTC m=+275.672348917"
Jul 14 13:31:51.314908 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:51.314876    2569 generic.go:356] "Generic (PLEG): container finished" podID="40df30ab-86bf-43b1-ac06-b4a53cb86daa" containerID="3dac87b2351597ecc377989e32fdd1629cedd9c8bab901c3d38767eb31d76697" exitCode=0
Jul 14 13:31:51.315261 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:51.314939    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznxx4" event={"ID":"40df30ab-86bf-43b1-ac06-b4a53cb86daa","Type":"ContainerDied","Data":"3dac87b2351597ecc377989e32fdd1629cedd9c8bab901c3d38767eb31d76697"}
Jul 14 13:31:52.462542 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:52.462522    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznxx4"
Jul 14 13:31:52.543474 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:52.543450    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40df30ab-86bf-43b1-ac06-b4a53cb86daa-bundle\") pod \"40df30ab-86bf-43b1-ac06-b4a53cb86daa\" (UID: \"40df30ab-86bf-43b1-ac06-b4a53cb86daa\") "
Jul 14 13:31:52.543576 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:52.543497    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7xhr\" (UniqueName: \"kubernetes.io/projected/40df30ab-86bf-43b1-ac06-b4a53cb86daa-kube-api-access-r7xhr\") pod \"40df30ab-86bf-43b1-ac06-b4a53cb86daa\" (UID: \"40df30ab-86bf-43b1-ac06-b4a53cb86daa\") "
Jul 14 13:31:52.543576 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:52.543523    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40df30ab-86bf-43b1-ac06-b4a53cb86daa-util\") pod \"40df30ab-86bf-43b1-ac06-b4a53cb86daa\" (UID: \"40df30ab-86bf-43b1-ac06-b4a53cb86daa\") "
Jul 14 13:31:52.544253 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:52.544228    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40df30ab-86bf-43b1-ac06-b4a53cb86daa-bundle" (OuterVolumeSpecName: "bundle") pod "40df30ab-86bf-43b1-ac06-b4a53cb86daa" (UID: "40df30ab-86bf-43b1-ac06-b4a53cb86daa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 13:31:52.545555 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:52.545529    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40df30ab-86bf-43b1-ac06-b4a53cb86daa-kube-api-access-r7xhr" (OuterVolumeSpecName: "kube-api-access-r7xhr") pod "40df30ab-86bf-43b1-ac06-b4a53cb86daa" (UID: "40df30ab-86bf-43b1-ac06-b4a53cb86daa"). InnerVolumeSpecName "kube-api-access-r7xhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 13:31:52.549282 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:52.549260    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40df30ab-86bf-43b1-ac06-b4a53cb86daa-util" (OuterVolumeSpecName: "util") pod "40df30ab-86bf-43b1-ac06-b4a53cb86daa" (UID: "40df30ab-86bf-43b1-ac06-b4a53cb86daa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 13:31:52.643980 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:52.643929    2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40df30ab-86bf-43b1-ac06-b4a53cb86daa-bundle\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:31:52.643980 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:52.643949    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7xhr\" (UniqueName: \"kubernetes.io/projected/40df30ab-86bf-43b1-ac06-b4a53cb86daa-kube-api-access-r7xhr\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:31:52.643980 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:52.643959    2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40df30ab-86bf-43b1-ac06-b4a53cb86daa-util\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:31:53.324530 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:53.324497    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznxx4" event={"ID":"40df30ab-86bf-43b1-ac06-b4a53cb86daa","Type":"ContainerDied","Data":"52eb3f0fc488477f578c65a59899b2f6e0cf644dfd22bb13e0966bb7d78370f7"}
Jul 14 13:31:53.324530 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:53.324516    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afznxx4"
Jul 14 13:31:53.324681 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:31:53.324548    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52eb3f0fc488477f578c65a59899b2f6e0cf644dfd22bb13e0966bb7d78370f7"
Jul 14 13:32:01.413824 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:01.413792    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9464f545f-d9n84"]
Jul 14 13:32:01.414263 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:01.414073    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40df30ab-86bf-43b1-ac06-b4a53cb86daa" containerName="util"
Jul 14 13:32:01.414263 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:01.414083    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="40df30ab-86bf-43b1-ac06-b4a53cb86daa" containerName="util"
Jul 14 13:32:01.414263 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:01.414093    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40df30ab-86bf-43b1-ac06-b4a53cb86daa" containerName="extract"
Jul 14 13:32:01.414263 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:01.414099    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="40df30ab-86bf-43b1-ac06-b4a53cb86daa" containerName="extract"
Jul 14 13:32:01.414263 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:01.414114    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40df30ab-86bf-43b1-ac06-b4a53cb86daa" containerName="pull"
Jul 14 13:32:01.414263 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:01.414119    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="40df30ab-86bf-43b1-ac06-b4a53cb86daa" containerName="pull"
Jul 14 13:32:01.414263 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:01.414161    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="40df30ab-86bf-43b1-ac06-b4a53cb86daa" containerName="extract"
Jul 14 13:32:01.416834 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:01.416816    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9464f545f-d9n84"
Jul 14 13:32:01.420460 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:01.420438    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 14 13:32:01.420761 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:01.420744    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 14 13:32:01.420761 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:01.420759    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 14 13:32:01.420890 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:01.420759    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-b48w9\""
Jul 14 13:32:01.420890 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:01.420811    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 14 13:32:01.438882 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:01.438857    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9464f545f-d9n84"]
Jul 14 13:32:01.509352 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:01.509328    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b4b8e64-9a5b-4839-a34c-edf6e5742d03-tmp\") pod \"opendatahub-operator-controller-manager-9464f545f-d9n84\" (UID: \"9b4b8e64-9a5b-4839-a34c-edf6e5742d03\") " pod="opendatahub/opendatahub-operator-controller-manager-9464f545f-d9n84"
Jul 14 13:32:01.509483 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:01.509365    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llrdx\" (UniqueName: \"kubernetes.io/projected/9b4b8e64-9a5b-4839-a34c-edf6e5742d03-kube-api-access-llrdx\") pod \"opendatahub-operator-controller-manager-9464f545f-d9n84\" (UID: \"9b4b8e64-9a5b-4839-a34c-edf6e5742d03\") " pod="opendatahub/opendatahub-operator-controller-manager-9464f545f-d9n84"
Jul 14 13:32:01.509483 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:01.509446    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9b4b8e64-9a5b-4839-a34c-edf6e5742d03-manifests\") pod \"opendatahub-operator-controller-manager-9464f545f-d9n84\" (UID: \"9b4b8e64-9a5b-4839-a34c-edf6e5742d03\") " pod="opendatahub/opendatahub-operator-controller-manager-9464f545f-d9n84"
Jul 14 13:32:01.509483 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:01.509478    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b4b8e64-9a5b-4839-a34c-edf6e5742d03-webhook-cert\") pod \"opendatahub-operator-controller-manager-9464f545f-d9n84\" (UID: \"9b4b8e64-9a5b-4839-a34c-edf6e5742d03\") " pod="opendatahub/opendatahub-operator-controller-manager-9464f545f-d9n84"
Jul 14 13:32:01.509640 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:01.509521    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9b4b8e64-9a5b-4839-a34c-edf6e5742d03-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9464f545f-d9n84\" (UID: \"9b4b8e64-9a5b-4839-a34c-edf6e5742d03\") " pod="opendatahub/opendatahub-operator-controller-manager-9464f545f-d9n84"
Jul 14 13:32:01.610592 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:01.610569    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b4b8e64-9a5b-4839-a34c-edf6e5742d03-tmp\") pod \"opendatahub-operator-controller-manager-9464f545f-d9n84\" (UID: \"9b4b8e64-9a5b-4839-a34c-edf6e5742d03\") " pod="opendatahub/opendatahub-operator-controller-manager-9464f545f-d9n84"
Jul 14 13:32:01.610708 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:01.610599    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llrdx\" (UniqueName: \"kubernetes.io/projected/9b4b8e64-9a5b-4839-a34c-edf6e5742d03-kube-api-access-llrdx\") pod \"opendatahub-operator-controller-manager-9464f545f-d9n84\" (UID: \"9b4b8e64-9a5b-4839-a34c-edf6e5742d03\") " pod="opendatahub/opendatahub-operator-controller-manager-9464f545f-d9n84"
Jul 14 13:32:01.610708 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:01.610627    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9b4b8e64-9a5b-4839-a34c-edf6e5742d03-manifests\") pod \"opendatahub-operator-controller-manager-9464f545f-d9n84\" (UID: \"9b4b8e64-9a5b-4839-a34c-edf6e5742d03\") " pod="opendatahub/opendatahub-operator-controller-manager-9464f545f-d9n84"
Jul 14 13:32:01.610708 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:01.610647    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b4b8e64-9a5b-4839-a34c-edf6e5742d03-webhook-cert\") pod \"opendatahub-operator-controller-manager-9464f545f-d9n84\" (UID: \"9b4b8e64-9a5b-4839-a34c-edf6e5742d03\") " pod="opendatahub/opendatahub-operator-controller-manager-9464f545f-d9n84"
Jul 14 13:32:01.610708 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:01.610682    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9b4b8e64-9a5b-4839-a34c-edf6e5742d03-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9464f545f-d9n84\" (UID: \"9b4b8e64-9a5b-4839-a34c-edf6e5742d03\") " pod="opendatahub/opendatahub-operator-controller-manager-9464f545f-d9n84"
Jul 14 13:32:01.611124 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:01.611097    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9b4b8e64-9a5b-4839-a34c-edf6e5742d03-manifests\") pod \"opendatahub-operator-controller-manager-9464f545f-d9n84\" (UID: \"9b4b8e64-9a5b-4839-a34c-edf6e5742d03\") " pod="opendatahub/opendatahub-operator-controller-manager-9464f545f-d9n84"
Jul 14 13:32:01.612775 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:01.612750    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b4b8e64-9a5b-4839-a34c-edf6e5742d03-tmp\") pod \"opendatahub-operator-controller-manager-9464f545f-d9n84\" (UID: \"9b4b8e64-9a5b-4839-a34c-edf6e5742d03\") " pod="opendatahub/opendatahub-operator-controller-manager-9464f545f-d9n84"
Jul 14 13:32:01.613126 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:01.613105    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9b4b8e64-9a5b-4839-a34c-edf6e5742d03-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9464f545f-d9n84\" (UID: \"9b4b8e64-9a5b-4839-a34c-edf6e5742d03\") " pod="opendatahub/opendatahub-operator-controller-manager-9464f545f-d9n84"
Jul 14 13:32:01.613177 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:01.613146    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b4b8e64-9a5b-4839-a34c-edf6e5742d03-webhook-cert\") pod \"opendatahub-operator-controller-manager-9464f545f-d9n84\" (UID: \"9b4b8e64-9a5b-4839-a34c-edf6e5742d03\") " pod="opendatahub/opendatahub-operator-controller-manager-9464f545f-d9n84"
Jul 14 13:32:01.621993 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:01.621967    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llrdx\" (UniqueName: \"kubernetes.io/projected/9b4b8e64-9a5b-4839-a34c-edf6e5742d03-kube-api-access-llrdx\") pod \"opendatahub-operator-controller-manager-9464f545f-d9n84\" (UID: \"9b4b8e64-9a5b-4839-a34c-edf6e5742d03\") " pod="opendatahub/opendatahub-operator-controller-manager-9464f545f-d9n84"
Jul 14 13:32:01.727531 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:01.727480    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9464f545f-d9n84"
Jul 14 13:32:01.862617 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:01.862591    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9464f545f-d9n84"]
Jul 14 13:32:01.866431 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:32:01.866400    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b4b8e64_9a5b_4839_a34c_edf6e5742d03.slice/crio-4b3dd484a3d63edef551ddba654db845273287c314ae243e97b96ba002a3e92c WatchSource:0}: Error finding container 4b3dd484a3d63edef551ddba654db845273287c314ae243e97b96ba002a3e92c: Status 404 returned error can't find the container with id 4b3dd484a3d63edef551ddba654db845273287c314ae243e97b96ba002a3e92c
Jul 14 13:32:02.363563 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:02.363533    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9464f545f-d9n84" event={"ID":"9b4b8e64-9a5b-4839-a34c-edf6e5742d03","Type":"ContainerStarted","Data":"4b3dd484a3d63edef551ddba654db845273287c314ae243e97b96ba002a3e92c"}
Jul 14 13:32:05.377000 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:05.376928    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9464f545f-d9n84" event={"ID":"9b4b8e64-9a5b-4839-a34c-edf6e5742d03","Type":"ContainerStarted","Data":"c3b4631ab5162bd83d9cd2c5b9f9a7899371fd0d86fa89ad88846e0b5535b07e"}
Jul 14 13:32:06.384185 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:06.384155    2569 generic.go:356] "Generic (PLEG): container finished" podID="9b4b8e64-9a5b-4839-a34c-edf6e5742d03" containerID="c3b4631ab5162bd83d9cd2c5b9f9a7899371fd0d86fa89ad88846e0b5535b07e" exitCode=0
Jul 14 13:32:06.384508 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:06.384239    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9464f545f-d9n84" event={"ID":"9b4b8e64-9a5b-4839-a34c-edf6e5742d03","Type":"ContainerDied","Data":"c3b4631ab5162bd83d9cd2c5b9f9a7899371fd0d86fa89ad88846e0b5535b07e"}
Jul 14 13:32:07.390316 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:07.390287    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9464f545f-d9n84" event={"ID":"9b4b8e64-9a5b-4839-a34c-edf6e5742d03","Type":"ContainerStarted","Data":"fb44e96228e2fba56dae3cf29bf7792b2f6181a658561dd708d1872c766db12e"}
Jul 14 13:32:07.390683 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:07.390434    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-9464f545f-d9n84"
Jul 14 13:32:07.426786 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:07.426734    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-9464f545f-d9n84" podStartSLOduration=1.828146381 podStartE2EDuration="6.426717706s" totalImagesPullingTime="3.542497554s" totalInitContainerRuntime="1.056073771s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:32:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:32:07.426280514 +0000 UTC m=+292.752600444" watchObservedRunningTime="2026-07-14 13:32:07.426717706 +0000 UTC m=+292.753037634"
Jul 14 13:32:15.025332 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:15.025300    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vq8tk_e33a489f-7d68-4e6c-9ffe-e7252049e15d/ovn-acl-logging/0.log"
Jul 14 13:32:15.025784 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:15.025465    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vq8tk_e33a489f-7d68-4e6c-9ffe-e7252049e15d/ovn-acl-logging/0.log"
Jul 14 13:32:15.028160 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:15.028143    2569 kubelet.go:1648] "Image garbage collection succeeded"
Jul 14 13:32:18.397079 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:18.397044    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-9464f545f-d9n84"
Jul 14 13:32:21.272806 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.272771    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-fc4b75864-z5cd5"]
Jul 14 13:32:21.276232 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.276209    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-fc4b75864-z5cd5"
Jul 14 13:32:21.280355 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.280332    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-7zflm\""
Jul 14 13:32:21.280553 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.280403    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 14 13:32:21.280553 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.280421    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 14 13:32:21.280553 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.280438    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 14 13:32:21.280723 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.280571    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 14 13:32:21.280723 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.280612    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 14 13:32:21.302909 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.302879    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-fc4b75864-z5cd5"]
Jul 14 13:32:21.354287 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.354254    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5e95c6d4-7c7a-4414-bed9-e57c5e5d8e68-manager-config\") pod \"lws-controller-manager-fc4b75864-z5cd5\" (UID: \"5e95c6d4-7c7a-4414-bed9-e57c5e5d8e68\") " pod="openshift-lws-operator/lws-controller-manager-fc4b75864-z5cd5"
Jul 14 13:32:21.354417 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.354304    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e95c6d4-7c7a-4414-bed9-e57c5e5d8e68-cert\") pod \"lws-controller-manager-fc4b75864-z5cd5\" (UID: \"5e95c6d4-7c7a-4414-bed9-e57c5e5d8e68\") " pod="openshift-lws-operator/lws-controller-manager-fc4b75864-z5cd5"
Jul 14 13:32:21.354417 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.354332    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5e95c6d4-7c7a-4414-bed9-e57c5e5d8e68-metrics-cert\") pod \"lws-controller-manager-fc4b75864-z5cd5\" (UID: \"5e95c6d4-7c7a-4414-bed9-e57c5e5d8e68\") " pod="openshift-lws-operator/lws-controller-manager-fc4b75864-z5cd5"
Jul 14 13:32:21.354417 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.354362    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zmnl\" (UniqueName: \"kubernetes.io/projected/5e95c6d4-7c7a-4414-bed9-e57c5e5d8e68-kube-api-access-6zmnl\") pod \"lws-controller-manager-fc4b75864-z5cd5\" (UID: \"5e95c6d4-7c7a-4414-bed9-e57c5e5d8e68\") " pod="openshift-lws-operator/lws-controller-manager-fc4b75864-z5cd5"
Jul 14 13:32:21.455497 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.455466    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5e95c6d4-7c7a-4414-bed9-e57c5e5d8e68-manager-config\") pod \"lws-controller-manager-fc4b75864-z5cd5\" (UID: \"5e95c6d4-7c7a-4414-bed9-e57c5e5d8e68\") " pod="openshift-lws-operator/lws-controller-manager-fc4b75864-z5cd5"
Jul 14 13:32:21.455656 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.455507    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e95c6d4-7c7a-4414-bed9-e57c5e5d8e68-cert\") pod \"lws-controller-manager-fc4b75864-z5cd5\" (UID: \"5e95c6d4-7c7a-4414-bed9-e57c5e5d8e68\") " pod="openshift-lws-operator/lws-controller-manager-fc4b75864-z5cd5"
Jul 14 13:32:21.455656 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.455541    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5e95c6d4-7c7a-4414-bed9-e57c5e5d8e68-metrics-cert\") pod \"lws-controller-manager-fc4b75864-z5cd5\" (UID: \"5e95c6d4-7c7a-4414-bed9-e57c5e5d8e68\") " pod="openshift-lws-operator/lws-controller-manager-fc4b75864-z5cd5"
Jul 14 13:32:21.455656 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.455569    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zmnl\" (UniqueName: \"kubernetes.io/projected/5e95c6d4-7c7a-4414-bed9-e57c5e5d8e68-kube-api-access-6zmnl\") pod \"lws-controller-manager-fc4b75864-z5cd5\" (UID: \"5e95c6d4-7c7a-4414-bed9-e57c5e5d8e68\") " pod="openshift-lws-operator/lws-controller-manager-fc4b75864-z5cd5"
Jul 14 13:32:21.456294 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.456271    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5e95c6d4-7c7a-4414-bed9-e57c5e5d8e68-manager-config\") pod \"lws-controller-manager-fc4b75864-z5cd5\" (UID: \"5e95c6d4-7c7a-4414-bed9-e57c5e5d8e68\") " pod="openshift-lws-operator/lws-controller-manager-fc4b75864-z5cd5"
Jul 14 13:32:21.458064 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.458045    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e95c6d4-7c7a-4414-bed9-e57c5e5d8e68-cert\") pod \"lws-controller-manager-fc4b75864-z5cd5\" (UID: \"5e95c6d4-7c7a-4414-bed9-e57c5e5d8e68\") " pod="openshift-lws-operator/lws-controller-manager-fc4b75864-z5cd5"
Jul 14 13:32:21.458176 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.458097    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5e95c6d4-7c7a-4414-bed9-e57c5e5d8e68-metrics-cert\") pod \"lws-controller-manager-fc4b75864-z5cd5\" (UID: \"5e95c6d4-7c7a-4414-bed9-e57c5e5d8e68\") " pod="openshift-lws-operator/lws-controller-manager-fc4b75864-z5cd5"
Jul 14 13:32:21.472838 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.472816    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zmnl\" (UniqueName: \"kubernetes.io/projected/5e95c6d4-7c7a-4414-bed9-e57c5e5d8e68-kube-api-access-6zmnl\") pod \"lws-controller-manager-fc4b75864-z5cd5\" (UID: \"5e95c6d4-7c7a-4414-bed9-e57c5e5d8e68\") " pod="openshift-lws-operator/lws-controller-manager-fc4b75864-z5cd5"
Jul 14 13:32:21.586595 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.586524    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-fc4b75864-z5cd5"
Jul 14 13:32:21.636742 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.636257    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-594bccb575-kj7t8"]
Jul 14 13:32:21.640269 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.640245    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-594bccb575-kj7t8"
Jul 14 13:32:21.655025 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.653681    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 14 13:32:21.655025 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.653998    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 14 13:32:21.655025 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.654226    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xtqjz\""
Jul 14 13:32:21.655025 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.654422    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 14 13:32:21.655288 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.655094    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 14 13:32:21.657411 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.657378    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f9fa9ff-066a-4e48-a0c2-b7087967960b-tls-certs\") pod \"kube-auth-proxy-594bccb575-kj7t8\" (UID: \"1f9fa9ff-066a-4e48-a0c2-b7087967960b\") " pod="openshift-ingress/kube-auth-proxy-594bccb575-kj7t8"
Jul 14 13:32:21.657585 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.657459    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncs89\" (UniqueName: \"kubernetes.io/projected/1f9fa9ff-066a-4e48-a0c2-b7087967960b-kube-api-access-ncs89\") pod \"kube-auth-proxy-594bccb575-kj7t8\" (UID: \"1f9fa9ff-066a-4e48-a0c2-b7087967960b\") " pod="openshift-ingress/kube-auth-proxy-594bccb575-kj7t8"
Jul 14 13:32:21.657585 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.657530    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f9fa9ff-066a-4e48-a0c2-b7087967960b-tmp\") pod \"kube-auth-proxy-594bccb575-kj7t8\" (UID: \"1f9fa9ff-066a-4e48-a0c2-b7087967960b\") " pod="openshift-ingress/kube-auth-proxy-594bccb575-kj7t8"
Jul 14 13:32:21.666253 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.666229    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-594bccb575-kj7t8"]
Jul 14 13:32:21.758961 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.758922    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncs89\" (UniqueName: \"kubernetes.io/projected/1f9fa9ff-066a-4e48-a0c2-b7087967960b-kube-api-access-ncs89\") pod \"kube-auth-proxy-594bccb575-kj7t8\" (UID: \"1f9fa9ff-066a-4e48-a0c2-b7087967960b\") " pod="openshift-ingress/kube-auth-proxy-594bccb575-kj7t8"
Jul 14 13:32:21.759156 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.758990    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f9fa9ff-066a-4e48-a0c2-b7087967960b-tmp\") pod \"kube-auth-proxy-594bccb575-kj7t8\" (UID: \"1f9fa9ff-066a-4e48-a0c2-b7087967960b\") " pod="openshift-ingress/kube-auth-proxy-594bccb575-kj7t8"
Jul 14 13:32:21.759156 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.759050    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f9fa9ff-066a-4e48-a0c2-b7087967960b-tls-certs\") pod \"kube-auth-proxy-594bccb575-kj7t8\" (UID: \"1f9fa9ff-066a-4e48-a0c2-b7087967960b\") " pod="openshift-ingress/kube-auth-proxy-594bccb575-kj7t8"
Jul 14 13:32:21.761330 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.761307    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f9fa9ff-066a-4e48-a0c2-b7087967960b-tmp\") pod \"kube-auth-proxy-594bccb575-kj7t8\" (UID: \"1f9fa9ff-066a-4e48-a0c2-b7087967960b\") " pod="openshift-ingress/kube-auth-proxy-594bccb575-kj7t8"
Jul 14 13:32:21.761487 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.761470    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f9fa9ff-066a-4e48-a0c2-b7087967960b-tls-certs\") pod \"kube-auth-proxy-594bccb575-kj7t8\" (UID: \"1f9fa9ff-066a-4e48-a0c2-b7087967960b\") " pod="openshift-ingress/kube-auth-proxy-594bccb575-kj7t8"
Jul 14 13:32:21.778389 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.778368    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-fc4b75864-z5cd5"]
Jul 14 13:32:21.781002 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:32:21.780976    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e95c6d4_7c7a_4414_bed9_e57c5e5d8e68.slice/crio-c76438be81bd13b6138c58e1cc8c1f435045fd9a1e0a1081ed8705a47fa82e14 WatchSource:0}: Error finding container c76438be81bd13b6138c58e1cc8c1f435045fd9a1e0a1081ed8705a47fa82e14: Status 404 returned error can't find the container with id c76438be81bd13b6138c58e1cc8c1f435045fd9a1e0a1081ed8705a47fa82e14
Jul 14 13:32:21.783001 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.782986    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 13:32:21.823254 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.823236    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncs89\" (UniqueName: \"kubernetes.io/projected/1f9fa9ff-066a-4e48-a0c2-b7087967960b-kube-api-access-ncs89\") pod \"kube-auth-proxy-594bccb575-kj7t8\" (UID: \"1f9fa9ff-066a-4e48-a0c2-b7087967960b\") " pod="openshift-ingress/kube-auth-proxy-594bccb575-kj7t8"
Jul 14 13:32:21.955167 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:21.955088    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-594bccb575-kj7t8"
Jul 14 13:32:22.108825 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:32:22.108359    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f9fa9ff_066a_4e48_a0c2_b7087967960b.slice/crio-04fb327a69c5401ffd1cea811ed5f93438a2133b25ac21182b7de35cc8a1e697 WatchSource:0}: Error finding container 04fb327a69c5401ffd1cea811ed5f93438a2133b25ac21182b7de35cc8a1e697: Status 404 returned error can't find the container with id 04fb327a69c5401ffd1cea811ed5f93438a2133b25ac21182b7de35cc8a1e697
Jul 14 13:32:22.112454 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:22.112429    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-594bccb575-kj7t8"]
Jul 14 13:32:22.455593 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:22.455498    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-594bccb575-kj7t8" event={"ID":"1f9fa9ff-066a-4e48-a0c2-b7087967960b","Type":"ContainerStarted","Data":"04fb327a69c5401ffd1cea811ed5f93438a2133b25ac21182b7de35cc8a1e697"}
Jul 14 13:32:22.457517 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:22.457484    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-fc4b75864-z5cd5" event={"ID":"5e95c6d4-7c7a-4414-bed9-e57c5e5d8e68","Type":"ContainerStarted","Data":"c76438be81bd13b6138c58e1cc8c1f435045fd9a1e0a1081ed8705a47fa82e14"}
Jul 14 13:32:23.463712 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:23.463630    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-fc4b75864-z5cd5" event={"ID":"5e95c6d4-7c7a-4414-bed9-e57c5e5d8e68","Type":"ContainerStarted","Data":"b2af28f8d35f89fd37cd1d5bb53c9ede51c5a556fa38d914e8d91ed961ad7a5d"}
Jul 14 13:32:23.464090 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:23.463771    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-fc4b75864-z5cd5"
Jul 14 13:32:23.497271 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:23.497224    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-fc4b75864-z5cd5" podStartSLOduration=1.229564767 podStartE2EDuration="2.497201948s" totalImagesPullingTime="1.267637181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:32:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:32:23.496117511 +0000 UTC m=+308.822437438" watchObservedRunningTime="2026-07-14 13:32:23.497201948 +0000 UTC m=+308.823521872"
Jul 14 13:32:25.475192 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:25.475148    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-594bccb575-kj7t8" event={"ID":"1f9fa9ff-066a-4e48-a0c2-b7087967960b","Type":"ContainerStarted","Data":"d9866a9ed22891c287e7145fb5d09f1d9f329b37a9ea23b1611231dae86b8154"}
Jul 14 13:32:25.496677 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:25.496623    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-594bccb575-kj7t8" podStartSLOduration=1.354117997 podStartE2EDuration="4.496607643s" totalImagesPullingTime="3.142489646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:32:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:32:25.496303536 +0000 UTC m=+310.822623477" watchObservedRunningTime="2026-07-14 13:32:25.496607643 +0000 UTC m=+310.822927571"
Jul 14 13:32:34.471390 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:32:34.471362    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-fc4b75864-z5cd5"
Jul 14 13:33:37.788001 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:37.787930    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v"]
Jul 14 13:33:37.791266 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:37.791248    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v"
Jul 14 13:33:37.794175 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:37.794158    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 14 13:33:37.795630 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:37.795613    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 14 13:33:37.795732 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:37.795664    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fspqb\""
Jul 14 13:33:37.802677 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:37.802647    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v"]
Jul 14 13:33:37.833873 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:37.833845    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvnmx\" (UniqueName: \"kubernetes.io/projected/e0ff5e6d-1a86-4426-92a1-6765774cfe35-kube-api-access-gvnmx\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v\" (UID: \"e0ff5e6d-1a86-4426-92a1-6765774cfe35\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v"
Jul 14 13:33:37.833964 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:37.833878    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0ff5e6d-1a86-4426-92a1-6765774cfe35-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v\" (UID: \"e0ff5e6d-1a86-4426-92a1-6765774cfe35\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v"
Jul 14 13:33:37.833964 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:37.833904    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0ff5e6d-1a86-4426-92a1-6765774cfe35-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v\" (UID: \"e0ff5e6d-1a86-4426-92a1-6765774cfe35\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v"
Jul 14 13:33:37.935126 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:37.935100    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvnmx\" (UniqueName: \"kubernetes.io/projected/e0ff5e6d-1a86-4426-92a1-6765774cfe35-kube-api-access-gvnmx\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v\" (UID: \"e0ff5e6d-1a86-4426-92a1-6765774cfe35\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v"
Jul 14 13:33:37.935212 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:37.935131    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0ff5e6d-1a86-4426-92a1-6765774cfe35-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v\" (UID: \"e0ff5e6d-1a86-4426-92a1-6765774cfe35\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v"
Jul 14 13:33:37.935212 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:37.935157    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0ff5e6d-1a86-4426-92a1-6765774cfe35-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v\" (UID: \"e0ff5e6d-1a86-4426-92a1-6765774cfe35\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v"
Jul 14 13:33:37.935557 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:37.935537    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0ff5e6d-1a86-4426-92a1-6765774cfe35-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v\" (UID: \"e0ff5e6d-1a86-4426-92a1-6765774cfe35\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v"
Jul 14 13:33:37.935557 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:37.935551    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0ff5e6d-1a86-4426-92a1-6765774cfe35-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v\" (UID: \"e0ff5e6d-1a86-4426-92a1-6765774cfe35\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v"
Jul 14 13:33:37.945254 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:37.945228    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvnmx\" (UniqueName: \"kubernetes.io/projected/e0ff5e6d-1a86-4426-92a1-6765774cfe35-kube-api-access-gvnmx\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v\" (UID: \"e0ff5e6d-1a86-4426-92a1-6765774cfe35\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v"
Jul 14 13:33:38.100556 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:38.100504    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v"
Jul 14 13:33:38.227682 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:38.227660    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v"]
Jul 14 13:33:38.229724 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:33:38.229699    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ff5e6d_1a86_4426_92a1_6765774cfe35.slice/crio-81bc94b835a3cf7575a9f8dbbf4d5e1c66cc391c7beac43d3159e8a1bf5bb2e4 WatchSource:0}: Error finding container 81bc94b835a3cf7575a9f8dbbf4d5e1c66cc391c7beac43d3159e8a1bf5bb2e4: Status 404 returned error can't find the container with id 81bc94b835a3cf7575a9f8dbbf4d5e1c66cc391c7beac43d3159e8a1bf5bb2e4
Jul 14 13:33:38.379725 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:38.379670    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf"]
Jul 14 13:33:38.382859 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:38.382843    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf"
Jul 14 13:33:38.394344 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:38.394323    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf"]
Jul 14 13:33:38.439252 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:38.439225    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvvds\" (UniqueName: \"kubernetes.io/projected/c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55-kube-api-access-lvvds\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf\" (UID: \"c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf"
Jul 14 13:33:38.439357 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:38.439271    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf\" (UID: \"c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf"
Jul 14 13:33:38.439357 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:38.439306    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf\" (UID: \"c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf"
Jul 14 13:33:38.540175 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:38.540137    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf\" (UID: \"c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf"
Jul 14 13:33:38.540286 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:38.540189    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf\" (UID: \"c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf"
Jul 14 13:33:38.540286 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:38.540259    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvvds\" (UniqueName: \"kubernetes.io/projected/c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55-kube-api-access-lvvds\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf\" (UID: \"c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf"
Jul 14 13:33:38.540572 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:38.540554    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf\" (UID: \"c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf"
Jul 14 13:33:38.540649 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:38.540565    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf\" (UID: \"c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf"
Jul 14 13:33:38.550170 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:38.550149    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvvds\" (UniqueName: \"kubernetes.io/projected/c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55-kube-api-access-lvvds\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf\" (UID: \"c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf"
Jul 14 13:33:38.693497 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:38.693439    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf"
Jul 14 13:33:38.783111 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:38.783000    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582"]
Jul 14 13:33:38.786619 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:38.786599    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582"
Jul 14 13:33:38.792253 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:38.792031    2569 generic.go:356] "Generic (PLEG): container finished" podID="e0ff5e6d-1a86-4426-92a1-6765774cfe35" containerID="bf8af31c43347063eb7c748fa3368dd7726900b6765c670e25d9d441e5e3aeb1" exitCode=0
Jul 14 13:33:38.792253 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:38.792068    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v" event={"ID":"e0ff5e6d-1a86-4426-92a1-6765774cfe35","Type":"ContainerDied","Data":"bf8af31c43347063eb7c748fa3368dd7726900b6765c670e25d9d441e5e3aeb1"}
Jul 14 13:33:38.792253 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:38.792117    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v" event={"ID":"e0ff5e6d-1a86-4426-92a1-6765774cfe35","Type":"ContainerStarted","Data":"81bc94b835a3cf7575a9f8dbbf4d5e1c66cc391c7beac43d3159e8a1bf5bb2e4"}
Jul 14 13:33:38.798823 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:38.798797    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582"]
Jul 14 13:33:38.822047 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:38.822003    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf"]
Jul 14 13:33:38.824703 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:33:38.824675    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d0134f_2bbc_40a0_9c11_bfc9d0d65a55.slice/crio-53cc235285f0320136717aa12e2c126177f600d73ec04c0bd8fbf0a0892e4a72 WatchSource:0}: Error finding container 53cc235285f0320136717aa12e2c126177f600d73ec04c0bd8fbf0a0892e4a72: Status 404 returned error can't find the container with id 53cc235285f0320136717aa12e2c126177f600d73ec04c0bd8fbf0a0892e4a72
Jul 14 13:33:38.842822 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:38.842794    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9488dcf-d5f5-472c-a31b-30ab21f755a3-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582\" (UID: \"b9488dcf-d5f5-472c-a31b-30ab21f755a3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582"
Jul 14 13:33:38.842930 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:38.842914    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9488dcf-d5f5-472c-a31b-30ab21f755a3-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582\" (UID: \"b9488dcf-d5f5-472c-a31b-30ab21f755a3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582"
Jul 14 13:33:38.842981 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:38.842945    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77zjg\" (UniqueName: \"kubernetes.io/projected/b9488dcf-d5f5-472c-a31b-30ab21f755a3-kube-api-access-77zjg\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582\" (UID: \"b9488dcf-d5f5-472c-a31b-30ab21f755a3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582"
Jul 14 13:33:38.943995 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:38.943931    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9488dcf-d5f5-472c-a31b-30ab21f755a3-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582\" (UID: \"b9488dcf-d5f5-472c-a31b-30ab21f755a3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582"
Jul 14 13:33:38.943995 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:38.943963    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9488dcf-d5f5-472c-a31b-30ab21f755a3-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582\" (UID: \"b9488dcf-d5f5-472c-a31b-30ab21f755a3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582"
Jul 14 13:33:38.943995 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:38.943984    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77zjg\" (UniqueName: \"kubernetes.io/projected/b9488dcf-d5f5-472c-a31b-30ab21f755a3-kube-api-access-77zjg\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582\" (UID: \"b9488dcf-d5f5-472c-a31b-30ab21f755a3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582"
Jul 14 13:33:38.944513 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:38.944490    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9488dcf-d5f5-472c-a31b-30ab21f755a3-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582\" (UID: \"b9488dcf-d5f5-472c-a31b-30ab21f755a3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582"
Jul 14 13:33:38.944513 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:38.944506    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9488dcf-d5f5-472c-a31b-30ab21f755a3-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582\" (UID: \"b9488dcf-d5f5-472c-a31b-30ab21f755a3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582"
Jul 14 13:33:38.959742 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:38.959721    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77zjg\" (UniqueName: \"kubernetes.io/projected/b9488dcf-d5f5-472c-a31b-30ab21f755a3-kube-api-access-77zjg\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582\" (UID: \"b9488dcf-d5f5-472c-a31b-30ab21f755a3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582"
Jul 14 13:33:39.100115 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:39.100090    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582"
Jul 14 13:33:39.196442 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:39.196378    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp"]
Jul 14 13:33:39.200087 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:39.200065    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp"
Jul 14 13:33:39.211105 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:39.211071    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp"]
Jul 14 13:33:39.236107 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:39.236080    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582"]
Jul 14 13:33:39.239530 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:33:39.239503    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9488dcf_d5f5_472c_a31b_30ab21f755a3.slice/crio-16b01d45c4cfb6a698db064208e03108a3f84bcdb229aaac999e12d487322775 WatchSource:0}: Error finding container 16b01d45c4cfb6a698db064208e03108a3f84bcdb229aaac999e12d487322775: Status 404 returned error can't find the container with id 16b01d45c4cfb6a698db064208e03108a3f84bcdb229aaac999e12d487322775
Jul 14 13:33:39.246798 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:39.246778    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/688ce8c2-6ba4-450e-bd7c-e5a025b403a5-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp\" (UID: \"688ce8c2-6ba4-450e-bd7c-e5a025b403a5\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp"
Jul 14 13:33:39.246871 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:39.246817    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/688ce8c2-6ba4-450e-bd7c-e5a025b403a5-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp\" (UID: \"688ce8c2-6ba4-450e-bd7c-e5a025b403a5\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp"
Jul 14 13:33:39.246871 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:39.246835    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr2nk\" (UniqueName: \"kubernetes.io/projected/688ce8c2-6ba4-450e-bd7c-e5a025b403a5-kube-api-access-qr2nk\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp\" (UID: \"688ce8c2-6ba4-450e-bd7c-e5a025b403a5\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp"
Jul 14 13:33:39.348037 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:39.347996    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/688ce8c2-6ba4-450e-bd7c-e5a025b403a5-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp\" (UID: \"688ce8c2-6ba4-450e-bd7c-e5a025b403a5\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp"
Jul 14 13:33:39.348127 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:39.348066    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/688ce8c2-6ba4-450e-bd7c-e5a025b403a5-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp\" (UID: \"688ce8c2-6ba4-450e-bd7c-e5a025b403a5\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp"
Jul 14 13:33:39.348127 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:39.348093    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr2nk\" (UniqueName: \"kubernetes.io/projected/688ce8c2-6ba4-450e-bd7c-e5a025b403a5-kube-api-access-qr2nk\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp\" (UID: \"688ce8c2-6ba4-450e-bd7c-e5a025b403a5\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp"
Jul 14 13:33:39.348377 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:39.348357    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/688ce8c2-6ba4-450e-bd7c-e5a025b403a5-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp\" (UID: \"688ce8c2-6ba4-450e-bd7c-e5a025b403a5\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp"
Jul 14 13:33:39.348426 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:39.348364    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/688ce8c2-6ba4-450e-bd7c-e5a025b403a5-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp\" (UID: \"688ce8c2-6ba4-450e-bd7c-e5a025b403a5\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp"
Jul 14 13:33:39.358608 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:39.358588    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr2nk\" (UniqueName: \"kubernetes.io/projected/688ce8c2-6ba4-450e-bd7c-e5a025b403a5-kube-api-access-qr2nk\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp\" (UID: \"688ce8c2-6ba4-450e-bd7c-e5a025b403a5\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp"
Jul 14 13:33:39.511816 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:39.511797    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp"
Jul 14 13:33:39.685892 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:39.685863    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp"]
Jul 14 13:33:39.686379 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:33:39.686354    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod688ce8c2_6ba4_450e_bd7c_e5a025b403a5.slice/crio-886e62eb60b9cf307f214c5b6ac9af8bd75797809f7cdf6d86ec2b6aedc3fdfc WatchSource:0}: Error finding container 886e62eb60b9cf307f214c5b6ac9af8bd75797809f7cdf6d86ec2b6aedc3fdfc: Status 404 returned error can't find the container with id 886e62eb60b9cf307f214c5b6ac9af8bd75797809f7cdf6d86ec2b6aedc3fdfc
Jul 14 13:33:39.799266 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:39.799235    2569 generic.go:356] "Generic (PLEG): container finished" podID="b9488dcf-d5f5-472c-a31b-30ab21f755a3" containerID="2d00f5b96eca9b2111ed1104fa26d43648e383ec96ca3678c41f592df685e0a3" exitCode=0
Jul 14 13:33:39.799643 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:39.799317    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582" event={"ID":"b9488dcf-d5f5-472c-a31b-30ab21f755a3","Type":"ContainerDied","Data":"2d00f5b96eca9b2111ed1104fa26d43648e383ec96ca3678c41f592df685e0a3"}
Jul 14 13:33:39.799643 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:39.799369    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582" event={"ID":"b9488dcf-d5f5-472c-a31b-30ab21f755a3","Type":"ContainerStarted","Data":"16b01d45c4cfb6a698db064208e03108a3f84bcdb229aaac999e12d487322775"}
Jul 14 13:33:39.801137 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:39.801110    2569 generic.go:356] "Generic (PLEG): container finished" podID="c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55" containerID="5f21c786276102a08c5297199eb078871d5ec7b362c7e8e3f18d5a9adb547ff4" exitCode=0
Jul 14 13:33:39.801267 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:39.801185    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf" event={"ID":"c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55","Type":"ContainerDied","Data":"5f21c786276102a08c5297199eb078871d5ec7b362c7e8e3f18d5a9adb547ff4"}
Jul 14 13:33:39.801267 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:39.801238    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf" event={"ID":"c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55","Type":"ContainerStarted","Data":"53cc235285f0320136717aa12e2c126177f600d73ec04c0bd8fbf0a0892e4a72"}
Jul 14 13:33:39.803378 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:39.803353    2569 generic.go:356] "Generic (PLEG): container finished" podID="e0ff5e6d-1a86-4426-92a1-6765774cfe35" containerID="c4f9ba16aa2db57790e6b9f72a69642362fb7267a9a565f5ad4e91ad1994037b" exitCode=0
Jul 14 13:33:39.803482 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:39.803457    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v" event={"ID":"e0ff5e6d-1a86-4426-92a1-6765774cfe35","Type":"ContainerDied","Data":"c4f9ba16aa2db57790e6b9f72a69642362fb7267a9a565f5ad4e91ad1994037b"}
Jul 14 13:33:39.805195 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:39.805165    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp" event={"ID":"688ce8c2-6ba4-450e-bd7c-e5a025b403a5","Type":"ContainerStarted","Data":"f4331312c9d88a51d0ac16d2f7342109b210e611bfbea94dd81f74f1a810e85b"}
Jul 14 13:33:39.805284 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:39.805203    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp" event={"ID":"688ce8c2-6ba4-450e-bd7c-e5a025b403a5","Type":"ContainerStarted","Data":"886e62eb60b9cf307f214c5b6ac9af8bd75797809f7cdf6d86ec2b6aedc3fdfc"}
Jul 14 13:33:40.811446 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:40.811419    2569 generic.go:356] "Generic (PLEG): container finished" podID="b9488dcf-d5f5-472c-a31b-30ab21f755a3" containerID="eb17abee9a5d6bdc9297a3c97d3e1aef351e86ad055ef3a21f201baa9adc172d" exitCode=0
Jul 14 13:33:40.811790 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:40.811519    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582" event={"ID":"b9488dcf-d5f5-472c-a31b-30ab21f755a3","Type":"ContainerDied","Data":"eb17abee9a5d6bdc9297a3c97d3e1aef351e86ad055ef3a21f201baa9adc172d"}
Jul 14 13:33:40.813525 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:40.813499    2569 generic.go:356] "Generic (PLEG): container finished" podID="c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55" containerID="20d8bb4c229ac1490eb9f326ae1e7bbd13bbefcc79469abd8e432dd1499d140f" exitCode=0
Jul 14 13:33:40.813623 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:40.813531    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf" event={"ID":"c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55","Type":"ContainerDied","Data":"20d8bb4c229ac1490eb9f326ae1e7bbd13bbefcc79469abd8e432dd1499d140f"}
Jul 14 13:33:40.815796 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:40.815774    2569 generic.go:356] "Generic (PLEG): container finished" podID="e0ff5e6d-1a86-4426-92a1-6765774cfe35" containerID="8a2a2b6958841209a87471011efdb410fee8f3957473ba99b058e60d25dd1bab" exitCode=0
Jul 14 13:33:40.815903 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:40.815845    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v" event={"ID":"e0ff5e6d-1a86-4426-92a1-6765774cfe35","Type":"ContainerDied","Data":"8a2a2b6958841209a87471011efdb410fee8f3957473ba99b058e60d25dd1bab"}
Jul 14 13:33:40.817424 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:40.817406    2569 generic.go:356] "Generic (PLEG): container finished" podID="688ce8c2-6ba4-450e-bd7c-e5a025b403a5" containerID="f4331312c9d88a51d0ac16d2f7342109b210e611bfbea94dd81f74f1a810e85b" exitCode=0
Jul 14 13:33:40.817531 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:40.817448    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp" event={"ID":"688ce8c2-6ba4-450e-bd7c-e5a025b403a5","Type":"ContainerDied","Data":"f4331312c9d88a51d0ac16d2f7342109b210e611bfbea94dd81f74f1a810e85b"}
Jul 14 13:33:41.823577 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:41.823546    2569 generic.go:356] "Generic (PLEG): container finished" podID="c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55" containerID="99cf9c6e63f060b83a9c6a250f8724dea756d7fea3fdbc1986ae757596c51317" exitCode=0
Jul 14 13:33:41.823895 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:41.823630    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf" event={"ID":"c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55","Type":"ContainerDied","Data":"99cf9c6e63f060b83a9c6a250f8724dea756d7fea3fdbc1986ae757596c51317"}
Jul 14 13:33:41.825344 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:41.825325    2569 generic.go:356] "Generic (PLEG): container finished" podID="688ce8c2-6ba4-450e-bd7c-e5a025b403a5" containerID="be89edf90e5907cb5592f60556efefbfe0ca5c6851560f5b35e4cc111549a2d0" exitCode=0
Jul 14 13:33:41.825450 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:41.825403    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp" event={"ID":"688ce8c2-6ba4-450e-bd7c-e5a025b403a5","Type":"ContainerDied","Data":"be89edf90e5907cb5592f60556efefbfe0ca5c6851560f5b35e4cc111549a2d0"}
Jul 14 13:33:41.827455 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:41.827434    2569 generic.go:356] "Generic (PLEG): container finished" podID="b9488dcf-d5f5-472c-a31b-30ab21f755a3" containerID="04c0c34a35454b33506002f1c32fddf70e5545b6282251678bcf945f0367bd5a" exitCode=0
Jul 14 13:33:41.827526 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:41.827507    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582" event={"ID":"b9488dcf-d5f5-472c-a31b-30ab21f755a3","Type":"ContainerDied","Data":"04c0c34a35454b33506002f1c32fddf70e5545b6282251678bcf945f0367bd5a"}
Jul 14 13:33:41.952159 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:41.952109    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v"
Jul 14 13:33:42.070103 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:42.070077    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0ff5e6d-1a86-4426-92a1-6765774cfe35-bundle\") pod \"e0ff5e6d-1a86-4426-92a1-6765774cfe35\" (UID: \"e0ff5e6d-1a86-4426-92a1-6765774cfe35\") "
Jul 14 13:33:42.070204 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:42.070151    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvnmx\" (UniqueName: \"kubernetes.io/projected/e0ff5e6d-1a86-4426-92a1-6765774cfe35-kube-api-access-gvnmx\") pod \"e0ff5e6d-1a86-4426-92a1-6765774cfe35\" (UID: \"e0ff5e6d-1a86-4426-92a1-6765774cfe35\") "
Jul 14 13:33:42.070204 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:42.070193    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0ff5e6d-1a86-4426-92a1-6765774cfe35-util\") pod \"e0ff5e6d-1a86-4426-92a1-6765774cfe35\" (UID: \"e0ff5e6d-1a86-4426-92a1-6765774cfe35\") "
Jul 14 13:33:42.070586 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:42.070556    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ff5e6d-1a86-4426-92a1-6765774cfe35-bundle" (OuterVolumeSpecName: "bundle") pod "e0ff5e6d-1a86-4426-92a1-6765774cfe35" (UID: "e0ff5e6d-1a86-4426-92a1-6765774cfe35"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 13:33:42.072208 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:42.072188    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ff5e6d-1a86-4426-92a1-6765774cfe35-kube-api-access-gvnmx" (OuterVolumeSpecName: "kube-api-access-gvnmx") pod "e0ff5e6d-1a86-4426-92a1-6765774cfe35" (UID: "e0ff5e6d-1a86-4426-92a1-6765774cfe35"). InnerVolumeSpecName "kube-api-access-gvnmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 13:33:42.075821 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:42.075784    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ff5e6d-1a86-4426-92a1-6765774cfe35-util" (OuterVolumeSpecName: "util") pod "e0ff5e6d-1a86-4426-92a1-6765774cfe35" (UID: "e0ff5e6d-1a86-4426-92a1-6765774cfe35"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 13:33:42.171171 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:42.171141    2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0ff5e6d-1a86-4426-92a1-6765774cfe35-util\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:33:42.171171 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:42.171173    2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0ff5e6d-1a86-4426-92a1-6765774cfe35-bundle\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:33:42.171297 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:42.171190    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvnmx\" (UniqueName: \"kubernetes.io/projected/e0ff5e6d-1a86-4426-92a1-6765774cfe35-kube-api-access-gvnmx\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:33:42.833972 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:42.833930    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v" event={"ID":"e0ff5e6d-1a86-4426-92a1-6765774cfe35","Type":"ContainerDied","Data":"81bc94b835a3cf7575a9f8dbbf4d5e1c66cc391c7beac43d3159e8a1bf5bb2e4"}
Jul 14 13:33:42.833972 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:42.833953    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v"
Jul 14 13:33:42.834451 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:42.834040    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81bc94b835a3cf7575a9f8dbbf4d5e1c66cc391c7beac43d3159e8a1bf5bb2e4"
Jul 14 13:33:42.835907 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:42.835874    2569 generic.go:356] "Generic (PLEG): container finished" podID="688ce8c2-6ba4-450e-bd7c-e5a025b403a5" containerID="30adc713a7a2739061686e27c6690b7846b2ae52a1c30f731ac1ea5eda787b57" exitCode=0
Jul 14 13:33:42.836053 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:42.835912    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp" event={"ID":"688ce8c2-6ba4-450e-bd7c-e5a025b403a5","Type":"ContainerDied","Data":"30adc713a7a2739061686e27c6690b7846b2ae52a1c30f731ac1ea5eda787b57"}
Jul 14 13:33:42.989966 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:42.989944    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582"
Jul 14 13:33:42.992320 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:42.992302    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf"
Jul 14 13:33:43.078291 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:43.078263    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9488dcf-d5f5-472c-a31b-30ab21f755a3-bundle\") pod \"b9488dcf-d5f5-472c-a31b-30ab21f755a3\" (UID: \"b9488dcf-d5f5-472c-a31b-30ab21f755a3\") "
Jul 14 13:33:43.078448 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:43.078309    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55-bundle\") pod \"c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55\" (UID: \"c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55\") "
Jul 14 13:33:43.078448 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:43.078401    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55-util\") pod \"c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55\" (UID: \"c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55\") "
Jul 14 13:33:43.078448 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:43.078424    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvvds\" (UniqueName: \"kubernetes.io/projected/c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55-kube-api-access-lvvds\") pod \"c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55\" (UID: \"c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55\") "
Jul 14 13:33:43.078448 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:43.078442    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9488dcf-d5f5-472c-a31b-30ab21f755a3-util\") pod \"b9488dcf-d5f5-472c-a31b-30ab21f755a3\" (UID: \"b9488dcf-d5f5-472c-a31b-30ab21f755a3\") "
Jul 14 13:33:43.078665 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:43.078457    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77zjg\" (UniqueName: \"kubernetes.io/projected/b9488dcf-d5f5-472c-a31b-30ab21f755a3-kube-api-access-77zjg\") pod \"b9488dcf-d5f5-472c-a31b-30ab21f755a3\" (UID: \"b9488dcf-d5f5-472c-a31b-30ab21f755a3\") "
Jul 14 13:33:43.078846 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:43.078820    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55-bundle" (OuterVolumeSpecName: "bundle") pod "c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55" (UID: "c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 13:33:43.079106 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:43.079076    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9488dcf-d5f5-472c-a31b-30ab21f755a3-bundle" (OuterVolumeSpecName: "bundle") pod "b9488dcf-d5f5-472c-a31b-30ab21f755a3" (UID: "b9488dcf-d5f5-472c-a31b-30ab21f755a3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 13:33:43.080689 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:43.080651    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9488dcf-d5f5-472c-a31b-30ab21f755a3-kube-api-access-77zjg" (OuterVolumeSpecName: "kube-api-access-77zjg") pod "b9488dcf-d5f5-472c-a31b-30ab21f755a3" (UID: "b9488dcf-d5f5-472c-a31b-30ab21f755a3"). InnerVolumeSpecName "kube-api-access-77zjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 13:33:43.080789 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:43.080754    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55-kube-api-access-lvvds" (OuterVolumeSpecName: "kube-api-access-lvvds") pod "c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55" (UID: "c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55"). InnerVolumeSpecName "kube-api-access-lvvds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 13:33:43.083778 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:43.083756    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55-util" (OuterVolumeSpecName: "util") pod "c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55" (UID: "c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 13:33:43.084223 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:43.084178    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9488dcf-d5f5-472c-a31b-30ab21f755a3-util" (OuterVolumeSpecName: "util") pod "b9488dcf-d5f5-472c-a31b-30ab21f755a3" (UID: "b9488dcf-d5f5-472c-a31b-30ab21f755a3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 13:33:43.179711 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:43.179688    2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9488dcf-d5f5-472c-a31b-30ab21f755a3-util\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:33:43.179711 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:43.179709    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77zjg\" (UniqueName: \"kubernetes.io/projected/b9488dcf-d5f5-472c-a31b-30ab21f755a3-kube-api-access-77zjg\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:33:43.179825 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:43.179719    2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9488dcf-d5f5-472c-a31b-30ab21f755a3-bundle\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:33:43.179825 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:43.179728    2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55-bundle\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:33:43.179825 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:43.179735    2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55-util\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:33:43.179825 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:43.179742    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvvds\" (UniqueName: \"kubernetes.io/projected/c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55-kube-api-access-lvvds\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:33:43.842874 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:43.842845    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf"
Jul 14 13:33:43.843304 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:43.842837    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf" event={"ID":"c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55","Type":"ContainerDied","Data":"53cc235285f0320136717aa12e2c126177f600d73ec04c0bd8fbf0a0892e4a72"}
Jul 14 13:33:43.843304 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:43.842966    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53cc235285f0320136717aa12e2c126177f600d73ec04c0bd8fbf0a0892e4a72"
Jul 14 13:33:43.844965 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:43.844925    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582" event={"ID":"b9488dcf-d5f5-472c-a31b-30ab21f755a3","Type":"ContainerDied","Data":"16b01d45c4cfb6a698db064208e03108a3f84bcdb229aaac999e12d487322775"}
Jul 14 13:33:43.845114 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:43.844998    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16b01d45c4cfb6a698db064208e03108a3f84bcdb229aaac999e12d487322775"
Jul 14 13:33:43.845265 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:43.845246    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582"
Jul 14 13:33:43.974166 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:43.974144    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp"
Jul 14 13:33:44.086260 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:44.086234    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/688ce8c2-6ba4-450e-bd7c-e5a025b403a5-util\") pod \"688ce8c2-6ba4-450e-bd7c-e5a025b403a5\" (UID: \"688ce8c2-6ba4-450e-bd7c-e5a025b403a5\") "
Jul 14 13:33:44.086405 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:44.086311    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/688ce8c2-6ba4-450e-bd7c-e5a025b403a5-bundle\") pod \"688ce8c2-6ba4-450e-bd7c-e5a025b403a5\" (UID: \"688ce8c2-6ba4-450e-bd7c-e5a025b403a5\") "
Jul 14 13:33:44.086405 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:44.086341    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qr2nk\" (UniqueName: \"kubernetes.io/projected/688ce8c2-6ba4-450e-bd7c-e5a025b403a5-kube-api-access-qr2nk\") pod \"688ce8c2-6ba4-450e-bd7c-e5a025b403a5\" (UID: \"688ce8c2-6ba4-450e-bd7c-e5a025b403a5\") "
Jul 14 13:33:44.086939 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:44.086901    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/688ce8c2-6ba4-450e-bd7c-e5a025b403a5-bundle" (OuterVolumeSpecName: "bundle") pod "688ce8c2-6ba4-450e-bd7c-e5a025b403a5" (UID: "688ce8c2-6ba4-450e-bd7c-e5a025b403a5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 13:33:44.089045 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:44.088977    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/688ce8c2-6ba4-450e-bd7c-e5a025b403a5-kube-api-access-qr2nk" (OuterVolumeSpecName: "kube-api-access-qr2nk") pod "688ce8c2-6ba4-450e-bd7c-e5a025b403a5" (UID: "688ce8c2-6ba4-450e-bd7c-e5a025b403a5"). InnerVolumeSpecName "kube-api-access-qr2nk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 13:33:44.091773 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:44.091752    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/688ce8c2-6ba4-450e-bd7c-e5a025b403a5-util" (OuterVolumeSpecName: "util") pod "688ce8c2-6ba4-450e-bd7c-e5a025b403a5" (UID: "688ce8c2-6ba4-450e-bd7c-e5a025b403a5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 13:33:44.186858 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:44.186834    2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/688ce8c2-6ba4-450e-bd7c-e5a025b403a5-bundle\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:33:44.186933 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:44.186861    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr2nk\" (UniqueName: \"kubernetes.io/projected/688ce8c2-6ba4-450e-bd7c-e5a025b403a5-kube-api-access-qr2nk\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:33:44.186933 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:44.186876    2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/688ce8c2-6ba4-450e-bd7c-e5a025b403a5-util\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:33:44.851354 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:44.851329    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp"
Jul 14 13:33:44.851700 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:44.851327    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp" event={"ID":"688ce8c2-6ba4-450e-bd7c-e5a025b403a5","Type":"ContainerDied","Data":"886e62eb60b9cf307f214c5b6ac9af8bd75797809f7cdf6d86ec2b6aedc3fdfc"}
Jul 14 13:33:44.851700 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:44.851458    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="886e62eb60b9cf307f214c5b6ac9af8bd75797809f7cdf6d86ec2b6aedc3fdfc"
Jul 14 13:33:54.675603 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.675572    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6dfbcbd67f-4d6h8"]
Jul 14 13:33:54.675947 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.675835    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9488dcf-d5f5-472c-a31b-30ab21f755a3" containerName="util"
Jul 14 13:33:54.675947 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.675845    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9488dcf-d5f5-472c-a31b-30ab21f755a3" containerName="util"
Jul 14 13:33:54.675947 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.675855    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="688ce8c2-6ba4-450e-bd7c-e5a025b403a5" containerName="pull"
Jul 14 13:33:54.675947 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.675861    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="688ce8c2-6ba4-450e-bd7c-e5a025b403a5" containerName="pull"
Jul 14 13:33:54.675947 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.675869    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55" containerName="pull"
Jul 14 13:33:54.675947 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.675874    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55" containerName="pull"
Jul 14 13:33:54.675947 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.675881    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55" containerName="extract"
Jul 14 13:33:54.675947 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.675886    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55" containerName="extract"
Jul 14 13:33:54.675947 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.675899    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9488dcf-d5f5-472c-a31b-30ab21f755a3" containerName="extract"
Jul 14 13:33:54.675947 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.675905    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9488dcf-d5f5-472c-a31b-30ab21f755a3" containerName="extract"
Jul 14 13:33:54.675947 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.675911    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0ff5e6d-1a86-4426-92a1-6765774cfe35" containerName="util"
Jul 14 13:33:54.675947 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.675917    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0ff5e6d-1a86-4426-92a1-6765774cfe35" containerName="util"
Jul 14 13:33:54.675947 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.675923    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="688ce8c2-6ba4-450e-bd7c-e5a025b403a5" containerName="util"
Jul 14 13:33:54.675947 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.675928    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="688ce8c2-6ba4-450e-bd7c-e5a025b403a5" containerName="util"
Jul 14 13:33:54.675947 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.675934    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9488dcf-d5f5-472c-a31b-30ab21f755a3" containerName="pull"
Jul 14 13:33:54.675947 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.675939    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9488dcf-d5f5-472c-a31b-30ab21f755a3" containerName="pull"
Jul 14 13:33:54.675947 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.675946    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0ff5e6d-1a86-4426-92a1-6765774cfe35" containerName="pull"
Jul 14 13:33:54.675947 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.675951    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0ff5e6d-1a86-4426-92a1-6765774cfe35" containerName="pull"
Jul 14 13:33:54.675947 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.675956    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="688ce8c2-6ba4-450e-bd7c-e5a025b403a5" containerName="extract"
Jul 14 13:33:54.676510 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.675961    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="688ce8c2-6ba4-450e-bd7c-e5a025b403a5" containerName="extract"
Jul 14 13:33:54.676510 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.675968    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55" containerName="util"
Jul 14 13:33:54.676510 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.675973    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55" containerName="util"
Jul 14 13:33:54.676510 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.675978    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0ff5e6d-1a86-4426-92a1-6765774cfe35" containerName="extract"
Jul 14 13:33:54.676510 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.675982    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0ff5e6d-1a86-4426-92a1-6765774cfe35" containerName="extract"
Jul 14 13:33:54.676510 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.676040    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="688ce8c2-6ba4-450e-bd7c-e5a025b403a5" containerName="extract"
Jul 14 13:33:54.676510 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.676048    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0ff5e6d-1a86-4426-92a1-6765774cfe35" containerName="extract"
Jul 14 13:33:54.676510 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.676055    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55" containerName="extract"
Jul 14 13:33:54.676510 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.676061    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9488dcf-d5f5-472c-a31b-30ab21f755a3" containerName="extract"
Jul 14 13:33:54.679141 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.679125    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dfbcbd67f-4d6h8"
Jul 14 13:33:54.694288 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.694264    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dfbcbd67f-4d6h8"]
Jul 14 13:33:54.772107 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.772082    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8b2ecbf2-91f6-4669-b754-335e27b35c7c-console-config\") pod \"console-6dfbcbd67f-4d6h8\" (UID: \"8b2ecbf2-91f6-4669-b754-335e27b35c7c\") " pod="openshift-console/console-6dfbcbd67f-4d6h8"
Jul 14 13:33:54.772218 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.772123    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hc7d\" (UniqueName: \"kubernetes.io/projected/8b2ecbf2-91f6-4669-b754-335e27b35c7c-kube-api-access-6hc7d\") pod \"console-6dfbcbd67f-4d6h8\" (UID: \"8b2ecbf2-91f6-4669-b754-335e27b35c7c\") " pod="openshift-console/console-6dfbcbd67f-4d6h8"
Jul 14 13:33:54.772218 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.772154    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8b2ecbf2-91f6-4669-b754-335e27b35c7c-oauth-serving-cert\") pod \"console-6dfbcbd67f-4d6h8\" (UID: \"8b2ecbf2-91f6-4669-b754-335e27b35c7c\") " pod="openshift-console/console-6dfbcbd67f-4d6h8"
Jul 14 13:33:54.772218 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.772194    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8b2ecbf2-91f6-4669-b754-335e27b35c7c-console-serving-cert\") pod \"console-6dfbcbd67f-4d6h8\" (UID: \"8b2ecbf2-91f6-4669-b754-335e27b35c7c\") " pod="openshift-console/console-6dfbcbd67f-4d6h8"
Jul 14 13:33:54.772327 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.772218    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b2ecbf2-91f6-4669-b754-335e27b35c7c-trusted-ca-bundle\") pod \"console-6dfbcbd67f-4d6h8\" (UID: \"8b2ecbf2-91f6-4669-b754-335e27b35c7c\") " pod="openshift-console/console-6dfbcbd67f-4d6h8"
Jul 14 13:33:54.772327 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.772261    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8b2ecbf2-91f6-4669-b754-335e27b35c7c-console-oauth-config\") pod \"console-6dfbcbd67f-4d6h8\" (UID: \"8b2ecbf2-91f6-4669-b754-335e27b35c7c\") " pod="openshift-console/console-6dfbcbd67f-4d6h8"
Jul 14 13:33:54.772327 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.772284    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8b2ecbf2-91f6-4669-b754-335e27b35c7c-service-ca\") pod \"console-6dfbcbd67f-4d6h8\" (UID: \"8b2ecbf2-91f6-4669-b754-335e27b35c7c\") " pod="openshift-console/console-6dfbcbd67f-4d6h8"
Jul 14 13:33:54.873381 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.873350    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8b2ecbf2-91f6-4669-b754-335e27b35c7c-console-config\") pod \"console-6dfbcbd67f-4d6h8\" (UID: \"8b2ecbf2-91f6-4669-b754-335e27b35c7c\") " pod="openshift-console/console-6dfbcbd67f-4d6h8"
Jul 14 13:33:54.873509 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.873393    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hc7d\" (UniqueName: \"kubernetes.io/projected/8b2ecbf2-91f6-4669-b754-335e27b35c7c-kube-api-access-6hc7d\") pod \"console-6dfbcbd67f-4d6h8\" (UID: \"8b2ecbf2-91f6-4669-b754-335e27b35c7c\") " pod="openshift-console/console-6dfbcbd67f-4d6h8"
Jul 14 13:33:54.873509 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.873413    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8b2ecbf2-91f6-4669-b754-335e27b35c7c-oauth-serving-cert\") pod \"console-6dfbcbd67f-4d6h8\" (UID: \"8b2ecbf2-91f6-4669-b754-335e27b35c7c\") " pod="openshift-console/console-6dfbcbd67f-4d6h8"
Jul 14 13:33:54.873509 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.873438    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8b2ecbf2-91f6-4669-b754-335e27b35c7c-console-serving-cert\") pod \"console-6dfbcbd67f-4d6h8\" (UID: \"8b2ecbf2-91f6-4669-b754-335e27b35c7c\") " pod="openshift-console/console-6dfbcbd67f-4d6h8"
Jul 14 13:33:54.873509 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.873458    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b2ecbf2-91f6-4669-b754-335e27b35c7c-trusted-ca-bundle\") pod \"console-6dfbcbd67f-4d6h8\" (UID: \"8b2ecbf2-91f6-4669-b754-335e27b35c7c\") " pod="openshift-console/console-6dfbcbd67f-4d6h8"
Jul 14 13:33:54.873509 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.873479    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8b2ecbf2-91f6-4669-b754-335e27b35c7c-console-oauth-config\") pod \"console-6dfbcbd67f-4d6h8\" (UID: \"8b2ecbf2-91f6-4669-b754-335e27b35c7c\") " pod="openshift-console/console-6dfbcbd67f-4d6h8"
Jul 14 13:33:54.873509 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.873496    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8b2ecbf2-91f6-4669-b754-335e27b35c7c-service-ca\") pod \"console-6dfbcbd67f-4d6h8\" (UID: \"8b2ecbf2-91f6-4669-b754-335e27b35c7c\") " pod="openshift-console/console-6dfbcbd67f-4d6h8"
Jul 14 13:33:54.874189 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.874153    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8b2ecbf2-91f6-4669-b754-335e27b35c7c-oauth-serving-cert\") pod \"console-6dfbcbd67f-4d6h8\" (UID: \"8b2ecbf2-91f6-4669-b754-335e27b35c7c\") " pod="openshift-console/console-6dfbcbd67f-4d6h8"
Jul 14 13:33:54.874189 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.874169    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8b2ecbf2-91f6-4669-b754-335e27b35c7c-console-config\") pod \"console-6dfbcbd67f-4d6h8\" (UID: \"8b2ecbf2-91f6-4669-b754-335e27b35c7c\") " pod="openshift-console/console-6dfbcbd67f-4d6h8"
Jul 14 13:33:54.874320 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.874279    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8b2ecbf2-91f6-4669-b754-335e27b35c7c-service-ca\") pod \"console-6dfbcbd67f-4d6h8\" (UID: \"8b2ecbf2-91f6-4669-b754-335e27b35c7c\") " pod="openshift-console/console-6dfbcbd67f-4d6h8"
Jul 14 13:33:54.874364 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.874333    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b2ecbf2-91f6-4669-b754-335e27b35c7c-trusted-ca-bundle\") pod \"console-6dfbcbd67f-4d6h8\" (UID: \"8b2ecbf2-91f6-4669-b754-335e27b35c7c\") " pod="openshift-console/console-6dfbcbd67f-4d6h8"
Jul 14 13:33:54.876077 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.876055    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8b2ecbf2-91f6-4669-b754-335e27b35c7c-console-oauth-config\") pod \"console-6dfbcbd67f-4d6h8\" (UID: \"8b2ecbf2-91f6-4669-b754-335e27b35c7c\") " pod="openshift-console/console-6dfbcbd67f-4d6h8"
Jul 14 13:33:54.876150 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.876113    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8b2ecbf2-91f6-4669-b754-335e27b35c7c-console-serving-cert\") pod \"console-6dfbcbd67f-4d6h8\" (UID: \"8b2ecbf2-91f6-4669-b754-335e27b35c7c\") " pod="openshift-console/console-6dfbcbd67f-4d6h8"
Jul 14 13:33:54.887269 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.887250    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hc7d\" (UniqueName: \"kubernetes.io/projected/8b2ecbf2-91f6-4669-b754-335e27b35c7c-kube-api-access-6hc7d\") pod \"console-6dfbcbd67f-4d6h8\" (UID: \"8b2ecbf2-91f6-4669-b754-335e27b35c7c\") " pod="openshift-console/console-6dfbcbd67f-4d6h8"
Jul 14 13:33:54.988092 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:54.988070    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dfbcbd67f-4d6h8"
Jul 14 13:33:55.128076 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:55.128047    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dfbcbd67f-4d6h8"]
Jul 14 13:33:55.130270 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:33:55.130239    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b2ecbf2_91f6_4669_b754_335e27b35c7c.slice/crio-94cf734d2cc0b39415ac8ed9edacf2f84bfe7a1ccf22b7ace49068907d119123 WatchSource:0}: Error finding container 94cf734d2cc0b39415ac8ed9edacf2f84bfe7a1ccf22b7ace49068907d119123: Status 404 returned error can't find the container with id 94cf734d2cc0b39415ac8ed9edacf2f84bfe7a1ccf22b7ace49068907d119123
Jul 14 13:33:55.902159 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:55.902084    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dfbcbd67f-4d6h8" event={"ID":"8b2ecbf2-91f6-4669-b754-335e27b35c7c","Type":"ContainerStarted","Data":"b3a3735590c7ad5a880bb3c0577ce263625e720d201d3cbb9940d3c1e783a7c7"}
Jul 14 13:33:55.902159 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:55.902127    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dfbcbd67f-4d6h8" event={"ID":"8b2ecbf2-91f6-4669-b754-335e27b35c7c","Type":"ContainerStarted","Data":"94cf734d2cc0b39415ac8ed9edacf2f84bfe7a1ccf22b7ace49068907d119123"}
Jul 14 13:33:55.929893 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:33:55.929847    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6dfbcbd67f-4d6h8" podStartSLOduration=1.92983153 podStartE2EDuration="1.92983153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:33:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:33:55.927102298 +0000 UTC m=+401.253422229" watchObservedRunningTime="2026-07-14 13:33:55.92983153 +0000 UTC m=+401.256151458"
Jul 14 13:34:04.988497 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:04.988471    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6dfbcbd67f-4d6h8"
Jul 14 13:34:04.988855 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:04.988522    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6dfbcbd67f-4d6h8"
Jul 14 13:34:04.993198 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:04.993177    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6dfbcbd67f-4d6h8"
Jul 14 13:34:05.950747 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:05.950716    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6dfbcbd67f-4d6h8"
Jul 14 13:34:06.018869 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:06.018834    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-546d6554f6-6xfzk"]
Jul 14 13:34:14.480800 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:14.480762    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-vpptc"]
Jul 14 13:34:14.485736 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:14.485714    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vpptc"
Jul 14 13:34:14.488805 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:14.488776    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 14 13:34:14.488934 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:14.488883    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fspqb\""
Jul 14 13:34:14.488934 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:14.488918    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 14 13:34:14.489080 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:14.488948    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 14 13:34:14.489963 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:14.489947    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 14 13:34:14.494129 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:14.494109    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-vpptc"]
Jul 14 13:34:14.624763 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:14.624736    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2827e420-b281-45e5-86a6-48dd0735031a-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-vpptc\" (UID: \"2827e420-b281-45e5-86a6-48dd0735031a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vpptc"
Jul 14 13:34:14.624885 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:14.624772    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2827e420-b281-45e5-86a6-48dd0735031a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-vpptc\" (UID: \"2827e420-b281-45e5-86a6-48dd0735031a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vpptc"
Jul 14 13:34:14.624885 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:14.624855    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-896bf\" (UniqueName: \"kubernetes.io/projected/2827e420-b281-45e5-86a6-48dd0735031a-kube-api-access-896bf\") pod \"kuadrant-console-plugin-66b5774985-vpptc\" (UID: \"2827e420-b281-45e5-86a6-48dd0735031a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vpptc"
Jul 14 13:34:14.725902 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:14.725872    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-896bf\" (UniqueName: \"kubernetes.io/projected/2827e420-b281-45e5-86a6-48dd0735031a-kube-api-access-896bf\") pod \"kuadrant-console-plugin-66b5774985-vpptc\" (UID: \"2827e420-b281-45e5-86a6-48dd0735031a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vpptc"
Jul 14 13:34:14.726108 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:14.725917    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2827e420-b281-45e5-86a6-48dd0735031a-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-vpptc\" (UID: \"2827e420-b281-45e5-86a6-48dd0735031a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vpptc"
Jul 14 13:34:14.726108 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:14.725948    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2827e420-b281-45e5-86a6-48dd0735031a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-vpptc\" (UID: \"2827e420-b281-45e5-86a6-48dd0735031a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vpptc"
Jul 14 13:34:14.726529 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:14.726508    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2827e420-b281-45e5-86a6-48dd0735031a-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-vpptc\" (UID: \"2827e420-b281-45e5-86a6-48dd0735031a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vpptc"
Jul 14 13:34:14.728398 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:14.728372    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2827e420-b281-45e5-86a6-48dd0735031a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-vpptc\" (UID: \"2827e420-b281-45e5-86a6-48dd0735031a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vpptc"
Jul 14 13:34:14.740799 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:14.740739    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-896bf\" (UniqueName: \"kubernetes.io/projected/2827e420-b281-45e5-86a6-48dd0735031a-kube-api-access-896bf\") pod \"kuadrant-console-plugin-66b5774985-vpptc\" (UID: \"2827e420-b281-45e5-86a6-48dd0735031a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vpptc"
Jul 14 13:34:14.796398 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:14.796376    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vpptc"
Jul 14 13:34:14.929767 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:14.929735    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-vpptc"]
Jul 14 13:34:14.932229 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:34:14.932206    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2827e420_b281_45e5_86a6_48dd0735031a.slice/crio-b59e6741c6fe8bdb37a614000b74e53c077a87085455decd90c7e57777b1af6a WatchSource:0}: Error finding container b59e6741c6fe8bdb37a614000b74e53c077a87085455decd90c7e57777b1af6a: Status 404 returned error can't find the container with id b59e6741c6fe8bdb37a614000b74e53c077a87085455decd90c7e57777b1af6a
Jul 14 13:34:14.988882 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:14.988857    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vpptc" event={"ID":"2827e420-b281-45e5-86a6-48dd0735031a","Type":"ContainerStarted","Data":"b59e6741c6fe8bdb37a614000b74e53c077a87085455decd90c7e57777b1af6a"}
Jul 14 13:34:24.765590 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:24.765557    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqj6m"]
Jul 14 13:34:24.768962 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:24.768943    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqj6m"
Jul 14 13:34:24.781378 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:24.781360    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xnwll\""
Jul 14 13:34:24.791714 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:24.791691    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqj6m"]
Jul 14 13:34:24.821579 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:24.821552    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msvh4\" (UniqueName: \"kubernetes.io/projected/2c0d339b-633f-453b-bf63-ceca41042de1-kube-api-access-msvh4\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jqj6m\" (UID: \"2c0d339b-633f-453b-bf63-ceca41042de1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqj6m"
Jul 14 13:34:24.821688 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:24.821591    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c0d339b-633f-453b-bf63-ceca41042de1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jqj6m\" (UID: \"2c0d339b-633f-453b-bf63-ceca41042de1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqj6m"
Jul 14 13:34:24.922657 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:24.922622    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msvh4\" (UniqueName: \"kubernetes.io/projected/2c0d339b-633f-453b-bf63-ceca41042de1-kube-api-access-msvh4\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jqj6m\" (UID: \"2c0d339b-633f-453b-bf63-ceca41042de1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqj6m"
Jul 14 13:34:24.922802 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:24.922671    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c0d339b-633f-453b-bf63-ceca41042de1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jqj6m\" (UID: \"2c0d339b-633f-453b-bf63-ceca41042de1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqj6m"
Jul 14 13:34:24.923089 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:24.923060    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c0d339b-633f-453b-bf63-ceca41042de1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jqj6m\" (UID: \"2c0d339b-633f-453b-bf63-ceca41042de1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqj6m"
Jul 14 13:34:24.953815 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:24.953790    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msvh4\" (UniqueName: \"kubernetes.io/projected/2c0d339b-633f-453b-bf63-ceca41042de1-kube-api-access-msvh4\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jqj6m\" (UID: \"2c0d339b-633f-453b-bf63-ceca41042de1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqj6m"
Jul 14 13:34:24.996606 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:24.996577    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqj6m"]
Jul 14 13:34:24.996895 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:24.996877    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqj6m"
Jul 14 13:34:25.601657 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:25.601614    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqj6m"]
Jul 14 13:34:25.609042 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:25.608965    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqj6m"]
Jul 14 13:34:31.040189 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:31.040116    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-546d6554f6-6xfzk" podUID="751bad59-841d-4ec7-a16d-90b66fdb38c6" containerName="console" containerID="cri-o://4cebd9b47428e8f60f81702f7586629523d9512a2e2049f8953dd79e4d660f72" gracePeriod=15
Jul 14 13:34:39.111353 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:39.111327    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546d6554f6-6xfzk"
Jul 14 13:34:39.123943 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:39.123881    2569 generic.go:356] "Generic (PLEG): container finished" podID="751bad59-841d-4ec7-a16d-90b66fdb38c6" containerID="4cebd9b47428e8f60f81702f7586629523d9512a2e2049f8953dd79e4d660f72" exitCode=0
Jul 14 13:34:39.123943 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:39.123921    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546d6554f6-6xfzk"
Jul 14 13:34:39.124132 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:39.123960    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546d6554f6-6xfzk" event={"ID":"751bad59-841d-4ec7-a16d-90b66fdb38c6","Type":"ContainerDied","Data":"4cebd9b47428e8f60f81702f7586629523d9512a2e2049f8953dd79e4d660f72"}
Jul 14 13:34:39.124132 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:39.124020    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546d6554f6-6xfzk" event={"ID":"751bad59-841d-4ec7-a16d-90b66fdb38c6","Type":"ContainerDied","Data":"5bfb45c55beafdd959668a70346b6b3c42d08c882197f7a748a4bcd5d565a227"}
Jul 14 13:34:39.124132 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:39.124040    2569 scope.go:117] "RemoveContainer" containerID="4cebd9b47428e8f60f81702f7586629523d9512a2e2049f8953dd79e4d660f72"
Jul 14 13:34:39.133350 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:39.133333    2569 scope.go:117] "RemoveContainer" containerID="4cebd9b47428e8f60f81702f7586629523d9512a2e2049f8953dd79e4d660f72"
Jul 14 13:34:39.133614 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:34:39.133596    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cebd9b47428e8f60f81702f7586629523d9512a2e2049f8953dd79e4d660f72\": container with ID starting with 4cebd9b47428e8f60f81702f7586629523d9512a2e2049f8953dd79e4d660f72 not found: ID does not exist" containerID="4cebd9b47428e8f60f81702f7586629523d9512a2e2049f8953dd79e4d660f72"
Jul 14 13:34:39.133684 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:39.133623    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cebd9b47428e8f60f81702f7586629523d9512a2e2049f8953dd79e4d660f72"} err="failed to get container status \"4cebd9b47428e8f60f81702f7586629523d9512a2e2049f8953dd79e4d660f72\": rpc error: code = NotFound desc = could not find container \"4cebd9b47428e8f60f81702f7586629523d9512a2e2049f8953dd79e4d660f72\": container with ID starting with 4cebd9b47428e8f60f81702f7586629523d9512a2e2049f8953dd79e4d660f72 not found: ID does not exist"
Jul 14 13:34:39.247693 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:39.247665    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzkks\" (UniqueName: \"kubernetes.io/projected/751bad59-841d-4ec7-a16d-90b66fdb38c6-kube-api-access-bzkks\") pod \"751bad59-841d-4ec7-a16d-90b66fdb38c6\" (UID: \"751bad59-841d-4ec7-a16d-90b66fdb38c6\") "
Jul 14 13:34:39.247797 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:39.247710    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/751bad59-841d-4ec7-a16d-90b66fdb38c6-console-oauth-config\") pod \"751bad59-841d-4ec7-a16d-90b66fdb38c6\" (UID: \"751bad59-841d-4ec7-a16d-90b66fdb38c6\") "
Jul 14 13:34:39.247797 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:39.247770    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/751bad59-841d-4ec7-a16d-90b66fdb38c6-oauth-serving-cert\") pod \"751bad59-841d-4ec7-a16d-90b66fdb38c6\" (UID: \"751bad59-841d-4ec7-a16d-90b66fdb38c6\") "
Jul 14 13:34:39.247921 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:39.247812    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/751bad59-841d-4ec7-a16d-90b66fdb38c6-trusted-ca-bundle\") pod \"751bad59-841d-4ec7-a16d-90b66fdb38c6\" (UID: \"751bad59-841d-4ec7-a16d-90b66fdb38c6\") "
Jul 14 13:34:39.247921 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:39.247859    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/751bad59-841d-4ec7-a16d-90b66fdb38c6-console-config\") pod \"751bad59-841d-4ec7-a16d-90b66fdb38c6\" (UID: \"751bad59-841d-4ec7-a16d-90b66fdb38c6\") "
Jul 14 13:34:39.248138 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:39.248121    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/751bad59-841d-4ec7-a16d-90b66fdb38c6-console-serving-cert\") pod \"751bad59-841d-4ec7-a16d-90b66fdb38c6\" (UID: \"751bad59-841d-4ec7-a16d-90b66fdb38c6\") "
Jul 14 13:34:39.248227 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:39.248164    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/751bad59-841d-4ec7-a16d-90b66fdb38c6-service-ca\") pod \"751bad59-841d-4ec7-a16d-90b66fdb38c6\" (UID: \"751bad59-841d-4ec7-a16d-90b66fdb38c6\") "
Jul 14 13:34:39.248284 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:39.248219    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/751bad59-841d-4ec7-a16d-90b66fdb38c6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "751bad59-841d-4ec7-a16d-90b66fdb38c6" (UID: "751bad59-841d-4ec7-a16d-90b66fdb38c6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 13:34:39.248539 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:39.248504    2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/751bad59-841d-4ec7-a16d-90b66fdb38c6-oauth-serving-cert\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:34:39.248539 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:39.248504    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/751bad59-841d-4ec7-a16d-90b66fdb38c6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "751bad59-841d-4ec7-a16d-90b66fdb38c6" (UID: "751bad59-841d-4ec7-a16d-90b66fdb38c6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 13:34:39.248539 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:39.248519    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/751bad59-841d-4ec7-a16d-90b66fdb38c6-service-ca" (OuterVolumeSpecName: "service-ca") pod "751bad59-841d-4ec7-a16d-90b66fdb38c6" (UID: "751bad59-841d-4ec7-a16d-90b66fdb38c6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 13:34:39.248742 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:39.248654    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/751bad59-841d-4ec7-a16d-90b66fdb38c6-console-config" (OuterVolumeSpecName: "console-config") pod "751bad59-841d-4ec7-a16d-90b66fdb38c6" (UID: "751bad59-841d-4ec7-a16d-90b66fdb38c6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 13:34:39.250224 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:39.250198    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/751bad59-841d-4ec7-a16d-90b66fdb38c6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "751bad59-841d-4ec7-a16d-90b66fdb38c6" (UID: "751bad59-841d-4ec7-a16d-90b66fdb38c6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 13:34:39.250224 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:39.250206    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/751bad59-841d-4ec7-a16d-90b66fdb38c6-kube-api-access-bzkks" (OuterVolumeSpecName: "kube-api-access-bzkks") pod "751bad59-841d-4ec7-a16d-90b66fdb38c6" (UID: "751bad59-841d-4ec7-a16d-90b66fdb38c6"). InnerVolumeSpecName "kube-api-access-bzkks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 13:34:39.250381 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:39.250362    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/751bad59-841d-4ec7-a16d-90b66fdb38c6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "751bad59-841d-4ec7-a16d-90b66fdb38c6" (UID: "751bad59-841d-4ec7-a16d-90b66fdb38c6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 13:34:39.348949 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:39.348926    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/751bad59-841d-4ec7-a16d-90b66fdb38c6-trusted-ca-bundle\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:34:39.348949 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:39.348946    2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/751bad59-841d-4ec7-a16d-90b66fdb38c6-console-config\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:34:39.349112 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:39.348957    2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/751bad59-841d-4ec7-a16d-90b66fdb38c6-console-serving-cert\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:34:39.349112 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:39.348966    2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/751bad59-841d-4ec7-a16d-90b66fdb38c6-service-ca\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:34:39.349112 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:39.348975    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzkks\" (UniqueName: \"kubernetes.io/projected/751bad59-841d-4ec7-a16d-90b66fdb38c6-kube-api-access-bzkks\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:34:39.349112 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:39.348983    2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/751bad59-841d-4ec7-a16d-90b66fdb38c6-console-oauth-config\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:34:39.453333 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:39.453308    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-546d6554f6-6xfzk"]
Jul 14 13:34:39.457749 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:39.457720    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-546d6554f6-6xfzk"]
Jul 14 13:34:39.950140 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:39.950101    2569 patch_prober.go:28] interesting pod/console-546d6554f6-6xfzk container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.18:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 14 13:34:39.950268 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:39.950169    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-546d6554f6-6xfzk" podUID="751bad59-841d-4ec7-a16d-90b66fdb38c6" containerName="console" probeResult="failure" output="Get \"https://10.133.0.18:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 14 13:34:40.131732 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:40.131656    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vpptc" event={"ID":"2827e420-b281-45e5-86a6-48dd0735031a","Type":"ContainerStarted","Data":"ec2f398763d27c980c0957d875df7543eea00fc3857924c3c3df123e53556589"}
Jul 14 13:34:40.162161 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:40.162113    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vpptc" podStartSLOduration=1.958579785 podStartE2EDuration="26.162098097s" totalImagesPullingTime="24.203518312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:34:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:34:40.160002625 +0000 UTC m=+445.486322566" watchObservedRunningTime="2026-07-14 13:34:40.162098097 +0000 UTC m=+445.488418025"
Jul 14 13:34:41.192483 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:41.192453    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="751bad59-841d-4ec7-a16d-90b66fdb38c6" path="/var/lib/kubelet/pods/751bad59-841d-4ec7-a16d-90b66fdb38c6/volumes"
Jul 14 13:34:41.403668 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:34:41.403633    2569 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-jqj6m_kuadrant-system_2c0d339b-633f-453b-bf63-ceca41042de1_0(4d5e36917b0f6707edf6a2432027ec642f0dde8b9e8dbbc02a2adc1f42736895): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-jqj6m to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}"
Jul 14 13:34:41.403742 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:34:41.403692    2569 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-jqj6m_kuadrant-system_2c0d339b-633f-453b-bf63-ceca41042de1_0(4d5e36917b0f6707edf6a2432027ec642f0dde8b9e8dbbc02a2adc1f42736895): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-jqj6m to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqj6m"
Jul 14 13:34:42.140959 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:42.140926    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqj6m"
Jul 14 13:34:42.144132 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:34:42.144089    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-jqj6m\" is forbidden: User \"system:node:ip-10-0-140-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-78.ec2.internal' and this object" podUID="2c0d339b-633f-453b-bf63-ceca41042de1" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqj6m"
Jul 14 13:34:42.247411 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:42.247383    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqj6m"
Jul 14 13:34:42.250427 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:34:42.250405    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-jqj6m\" is forbidden: User \"system:node:ip-10-0-140-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-78.ec2.internal' and this object" podUID="2c0d339b-633f-453b-bf63-ceca41042de1" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqj6m"
Jul 14 13:34:42.374866 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:42.374837    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c0d339b-633f-453b-bf63-ceca41042de1-extensions-socket-volume\") pod \"2c0d339b-633f-453b-bf63-ceca41042de1\" (UID: \"2c0d339b-633f-453b-bf63-ceca41042de1\") "
Jul 14 13:34:42.375073 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:42.374957    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msvh4\" (UniqueName: \"kubernetes.io/projected/2c0d339b-633f-453b-bf63-ceca41042de1-kube-api-access-msvh4\") pod \"2c0d339b-633f-453b-bf63-ceca41042de1\" (UID: \"2c0d339b-633f-453b-bf63-ceca41042de1\") "
Jul 14 13:34:42.375143 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:42.375099    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c0d339b-633f-453b-bf63-ceca41042de1-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2c0d339b-633f-453b-bf63-ceca41042de1" (UID: "2c0d339b-633f-453b-bf63-ceca41042de1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 13:34:42.376961 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:42.376940    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c0d339b-633f-453b-bf63-ceca41042de1-kube-api-access-msvh4" (OuterVolumeSpecName: "kube-api-access-msvh4") pod "2c0d339b-633f-453b-bf63-ceca41042de1" (UID: "2c0d339b-633f-453b-bf63-ceca41042de1"). InnerVolumeSpecName "kube-api-access-msvh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 13:34:42.476315 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:42.476257    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msvh4\" (UniqueName: \"kubernetes.io/projected/2c0d339b-633f-453b-bf63-ceca41042de1-kube-api-access-msvh4\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:34:42.476315 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:42.476278    2569 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c0d339b-633f-453b-bf63-ceca41042de1-extensions-socket-volume\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:34:43.146503 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:43.146472    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqj6m"
Jul 14 13:34:43.151635 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:34:43.151482    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-jqj6m\" is forbidden: User \"system:node:ip-10-0-140-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-78.ec2.internal' and this object" podUID="2c0d339b-633f-453b-bf63-ceca41042de1" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqj6m"
Jul 14 13:34:43.154199 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:34:43.154174    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-jqj6m\" is forbidden: User \"system:node:ip-10-0-140-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-78.ec2.internal' and this object" podUID="2c0d339b-633f-453b-bf63-ceca41042de1" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jqj6m"
Jul 14 13:34:43.192170 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:34:43.192145    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c0d339b-633f-453b-bf63-ceca41042de1" path="/var/lib/kubelet/pods/2c0d339b-633f-453b-bf63-ceca41042de1/volumes"
Jul 14 13:35:13.467950 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:13.467916    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2fmb8"]
Jul 14 13:35:13.468411 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:13.468224    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="751bad59-841d-4ec7-a16d-90b66fdb38c6" containerName="console"
Jul 14 13:35:13.468411 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:13.468236    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="751bad59-841d-4ec7-a16d-90b66fdb38c6" containerName="console"
Jul 14 13:35:13.468411 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:13.468297    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="751bad59-841d-4ec7-a16d-90b66fdb38c6" containerName="console"
Jul 14 13:35:13.504616 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:13.504588    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2fmb8"]
Jul 14 13:35:13.504759 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:13.504678    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-2fmb8"
Jul 14 13:35:13.507688 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:13.507667    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 14 13:35:13.559128 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:13.559101    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2fmb8"]
Jul 14 13:35:13.603338 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:13.603315    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/06c6c11f-71fd-496b-a713-7b082c85f904-config-file\") pod \"limitador-limitador-64c94fd64f-2fmb8\" (UID: \"06c6c11f-71fd-496b-a713-7b082c85f904\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2fmb8"
Jul 14 13:35:13.603479 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:13.603378    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqkcw\" (UniqueName: \"kubernetes.io/projected/06c6c11f-71fd-496b-a713-7b082c85f904-kube-api-access-rqkcw\") pod \"limitador-limitador-64c94fd64f-2fmb8\" (UID: \"06c6c11f-71fd-496b-a713-7b082c85f904\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2fmb8"
Jul 14 13:35:13.704283 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:13.704253    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/06c6c11f-71fd-496b-a713-7b082c85f904-config-file\") pod \"limitador-limitador-64c94fd64f-2fmb8\" (UID: \"06c6c11f-71fd-496b-a713-7b082c85f904\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2fmb8"
Jul 14 13:35:13.704400 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:13.704318    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqkcw\" (UniqueName: \"kubernetes.io/projected/06c6c11f-71fd-496b-a713-7b082c85f904-kube-api-access-rqkcw\") pod \"limitador-limitador-64c94fd64f-2fmb8\" (UID: \"06c6c11f-71fd-496b-a713-7b082c85f904\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2fmb8"
Jul 14 13:35:13.704899 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:13.704880    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/06c6c11f-71fd-496b-a713-7b082c85f904-config-file\") pod \"limitador-limitador-64c94fd64f-2fmb8\" (UID: \"06c6c11f-71fd-496b-a713-7b082c85f904\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2fmb8"
Jul 14 13:35:13.714190 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:13.714164    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqkcw\" (UniqueName: \"kubernetes.io/projected/06c6c11f-71fd-496b-a713-7b082c85f904-kube-api-access-rqkcw\") pod \"limitador-limitador-64c94fd64f-2fmb8\" (UID: \"06c6c11f-71fd-496b-a713-7b082c85f904\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2fmb8"
Jul 14 13:35:13.814309 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:13.814286    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-2fmb8"
Jul 14 13:35:13.955653 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:13.955630    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2fmb8"]
Jul 14 13:35:13.957788 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:35:13.957760    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06c6c11f_71fd_496b_a713_7b082c85f904.slice/crio-8f0b08d96db80e8a52b193706f3df6dbb7205aadf66cf360cf221bb404d11e71 WatchSource:0}: Error finding container 8f0b08d96db80e8a52b193706f3df6dbb7205aadf66cf360cf221bb404d11e71: Status 404 returned error can't find the container with id 8f0b08d96db80e8a52b193706f3df6dbb7205aadf66cf360cf221bb404d11e71
Jul 14 13:35:14.287211 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:14.287181    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-2fmb8" event={"ID":"06c6c11f-71fd-496b-a713-7b082c85f904","Type":"ContainerStarted","Data":"8f0b08d96db80e8a52b193706f3df6dbb7205aadf66cf360cf221bb404d11e71"}
Jul 14 13:35:14.397199 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:14.397168    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-kj8hz"]
Jul 14 13:35:14.399999 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:14.399983    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-kj8hz"
Jul 14 13:35:14.403181 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:14.403161    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vhxm4\""
Jul 14 13:35:14.408766 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:14.408746    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-kj8hz"]
Jul 14 13:35:14.510198 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:14.510174    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrckq\" (UniqueName: \"kubernetes.io/projected/4c8f4f83-dd7f-4c53-96df-8b927c60a310-kube-api-access-jrckq\") pod \"authorino-58cd955675-kj8hz\" (UID: \"4c8f4f83-dd7f-4c53-96df-8b927c60a310\") " pod="kuadrant-system/authorino-58cd955675-kj8hz"
Jul 14 13:35:14.555059 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:14.554993    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-kqdrd"]
Jul 14 13:35:14.557110 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:14.557091    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-kqdrd"
Jul 14 13:35:14.571656 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:14.571639    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-kqdrd"]
Jul 14 13:35:14.611374 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:14.611343    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrckq\" (UniqueName: \"kubernetes.io/projected/4c8f4f83-dd7f-4c53-96df-8b927c60a310-kube-api-access-jrckq\") pod \"authorino-58cd955675-kj8hz\" (UID: \"4c8f4f83-dd7f-4c53-96df-8b927c60a310\") " pod="kuadrant-system/authorino-58cd955675-kj8hz"
Jul 14 13:35:14.621035 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:14.620991    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrckq\" (UniqueName: \"kubernetes.io/projected/4c8f4f83-dd7f-4c53-96df-8b927c60a310-kube-api-access-jrckq\") pod \"authorino-58cd955675-kj8hz\" (UID: \"4c8f4f83-dd7f-4c53-96df-8b927c60a310\") " pod="kuadrant-system/authorino-58cd955675-kj8hz"
Jul 14 13:35:14.709327 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:14.709303    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-kj8hz"
Jul 14 13:35:14.711766 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:14.711743    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmwl4\" (UniqueName: \"kubernetes.io/projected/8e746d96-729e-40da-a1b2-6ef375c49ee1-kube-api-access-kmwl4\") pod \"authorino-75d556bd77-kqdrd\" (UID: \"8e746d96-729e-40da-a1b2-6ef375c49ee1\") " pod="kuadrant-system/authorino-75d556bd77-kqdrd"
Jul 14 13:35:14.813344 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:14.813246    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmwl4\" (UniqueName: \"kubernetes.io/projected/8e746d96-729e-40da-a1b2-6ef375c49ee1-kube-api-access-kmwl4\") pod \"authorino-75d556bd77-kqdrd\" (UID: \"8e746d96-729e-40da-a1b2-6ef375c49ee1\") " pod="kuadrant-system/authorino-75d556bd77-kqdrd"
Jul 14 13:35:14.826030 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:14.825950    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmwl4\" (UniqueName: \"kubernetes.io/projected/8e746d96-729e-40da-a1b2-6ef375c49ee1-kube-api-access-kmwl4\") pod \"authorino-75d556bd77-kqdrd\" (UID: \"8e746d96-729e-40da-a1b2-6ef375c49ee1\") " pod="kuadrant-system/authorino-75d556bd77-kqdrd"
Jul 14 13:35:14.862900 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:14.862835    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-kj8hz"]
Jul 14 13:35:14.865060 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:14.864804    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-kqdrd"
Jul 14 13:35:14.871205 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:35:14.871178    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c8f4f83_dd7f_4c53_96df_8b927c60a310.slice/crio-0a395c76f5f3dbe6de134884576f2a65a4d01c445bde57ce367bac66b0568951 WatchSource:0}: Error finding container 0a395c76f5f3dbe6de134884576f2a65a4d01c445bde57ce367bac66b0568951: Status 404 returned error can't find the container with id 0a395c76f5f3dbe6de134884576f2a65a4d01c445bde57ce367bac66b0568951
Jul 14 13:35:15.047574 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:15.047545    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-kqdrd"]
Jul 14 13:35:15.294633 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:15.294600    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-kqdrd" event={"ID":"8e746d96-729e-40da-a1b2-6ef375c49ee1","Type":"ContainerStarted","Data":"7d9fd02f2615c28dd77bf1b135ff702dc1736a83abb8cf5789f1ac935c13e80e"}
Jul 14 13:35:15.295872 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:15.295845    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-kj8hz" event={"ID":"4c8f4f83-dd7f-4c53-96df-8b927c60a310","Type":"ContainerStarted","Data":"0a395c76f5f3dbe6de134884576f2a65a4d01c445bde57ce367bac66b0568951"}
Jul 14 13:35:17.118966 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:17.118929    2569 scope.go:117] "RemoveContainer" containerID="3f66065637cf6bb0aa36b4902d9eb4af9d69b7a29382ee745e4937a76afbffd0"
Jul 14 13:35:19.319481 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:19.319449    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-kqdrd" event={"ID":"8e746d96-729e-40da-a1b2-6ef375c49ee1","Type":"ContainerStarted","Data":"aa4606b785dd99b885355256a2c09026999576213e44153c52ca9cd5d0319332"}
Jul 14 13:35:19.320918 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:19.320893    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-kj8hz" event={"ID":"4c8f4f83-dd7f-4c53-96df-8b927c60a310","Type":"ContainerStarted","Data":"8c396b12a07d2777ecbfdeb9dcddd650361d1392ede2f8b0a43df523def108dd"}
Jul 14 13:35:19.322341 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:19.322315    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-2fmb8" event={"ID":"06c6c11f-71fd-496b-a713-7b082c85f904","Type":"ContainerStarted","Data":"9fd1e442b0fc8e1b78947481c523acf672a670d57cffac0aa09cd4dd89d92a80"}
Jul 14 13:35:19.322449 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:19.322375    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-2fmb8"
Jul 14 13:35:19.337801 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:19.337758    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-kqdrd" podStartSLOduration=1.643673502 podStartE2EDuration="5.337745424s" totalImagesPullingTime="3.694071922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:35:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:35:19.336375135 +0000 UTC m=+484.662695064" watchObservedRunningTime="2026-07-14 13:35:19.337745424 +0000 UTC m=+484.664065330"
Jul 14 13:35:19.355261 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:19.355218    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-kj8hz" podStartSLOduration=1.482425469 podStartE2EDuration="5.35520461s" totalImagesPullingTime="3.872779141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:35:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:35:19.352971798 +0000 UTC m=+484.679291741" watchObservedRunningTime="2026-07-14 13:35:19.35520461 +0000 UTC m=+484.681524531"
Jul 14 13:35:19.366989 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:19.366966    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-kj8hz"]
Jul 14 13:35:19.373719 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:19.373679    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-2fmb8" podStartSLOduration=1.5916748680000001 podStartE2EDuration="6.373666301s" totalImagesPullingTime="4.781991433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:35:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:35:19.3704335 +0000 UTC m=+484.696753429" watchObservedRunningTime="2026-07-14 13:35:19.373666301 +0000 UTC m=+484.699986252"
Jul 14 13:35:21.332451 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:21.332414    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-kj8hz" podUID="4c8f4f83-dd7f-4c53-96df-8b927c60a310" containerName="authorino" containerID="cri-o://8c396b12a07d2777ecbfdeb9dcddd650361d1392ede2f8b0a43df523def108dd" gracePeriod=30
Jul 14 13:35:21.568442 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:21.568419    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-kj8hz"
Jul 14 13:35:21.672655 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:21.672591    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrckq\" (UniqueName: \"kubernetes.io/projected/4c8f4f83-dd7f-4c53-96df-8b927c60a310-kube-api-access-jrckq\") pod \"4c8f4f83-dd7f-4c53-96df-8b927c60a310\" (UID: \"4c8f4f83-dd7f-4c53-96df-8b927c60a310\") "
Jul 14 13:35:21.674591 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:21.674564    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c8f4f83-dd7f-4c53-96df-8b927c60a310-kube-api-access-jrckq" (OuterVolumeSpecName: "kube-api-access-jrckq") pod "4c8f4f83-dd7f-4c53-96df-8b927c60a310" (UID: "4c8f4f83-dd7f-4c53-96df-8b927c60a310"). InnerVolumeSpecName "kube-api-access-jrckq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 13:35:21.773388 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:21.773360    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrckq\" (UniqueName: \"kubernetes.io/projected/4c8f4f83-dd7f-4c53-96df-8b927c60a310-kube-api-access-jrckq\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:35:22.339486 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:22.339447    2569 generic.go:356] "Generic (PLEG): container finished" podID="4c8f4f83-dd7f-4c53-96df-8b927c60a310" containerID="8c396b12a07d2777ecbfdeb9dcddd650361d1392ede2f8b0a43df523def108dd" exitCode=0
Jul 14 13:35:22.339881 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:22.339491    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-kj8hz"
Jul 14 13:35:22.339881 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:22.339531    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-kj8hz" event={"ID":"4c8f4f83-dd7f-4c53-96df-8b927c60a310","Type":"ContainerDied","Data":"8c396b12a07d2777ecbfdeb9dcddd650361d1392ede2f8b0a43df523def108dd"}
Jul 14 13:35:22.339881 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:22.339570    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-kj8hz" event={"ID":"4c8f4f83-dd7f-4c53-96df-8b927c60a310","Type":"ContainerDied","Data":"0a395c76f5f3dbe6de134884576f2a65a4d01c445bde57ce367bac66b0568951"}
Jul 14 13:35:22.339881 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:22.339587    2569 scope.go:117] "RemoveContainer" containerID="8c396b12a07d2777ecbfdeb9dcddd650361d1392ede2f8b0a43df523def108dd"
Jul 14 13:35:22.348412 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:22.348390    2569 scope.go:117] "RemoveContainer" containerID="8c396b12a07d2777ecbfdeb9dcddd650361d1392ede2f8b0a43df523def108dd"
Jul 14 13:35:22.348641 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:35:22.348624    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c396b12a07d2777ecbfdeb9dcddd650361d1392ede2f8b0a43df523def108dd\": container with ID starting with 8c396b12a07d2777ecbfdeb9dcddd650361d1392ede2f8b0a43df523def108dd not found: ID does not exist" containerID="8c396b12a07d2777ecbfdeb9dcddd650361d1392ede2f8b0a43df523def108dd"
Jul 14 13:35:22.348685 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:22.348649    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c396b12a07d2777ecbfdeb9dcddd650361d1392ede2f8b0a43df523def108dd"} err="failed to get container status \"8c396b12a07d2777ecbfdeb9dcddd650361d1392ede2f8b0a43df523def108dd\": rpc error: code = NotFound desc = could not find container \"8c396b12a07d2777ecbfdeb9dcddd650361d1392ede2f8b0a43df523def108dd\": container with ID starting with 8c396b12a07d2777ecbfdeb9dcddd650361d1392ede2f8b0a43df523def108dd not found: ID does not exist"
Jul 14 13:35:22.364910 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:22.364889    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-kj8hz"]
Jul 14 13:35:22.370406 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:22.370386    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-kj8hz"]
Jul 14 13:35:23.192879 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:23.192846    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c8f4f83-dd7f-4c53-96df-8b927c60a310" path="/var/lib/kubelet/pods/4c8f4f83-dd7f-4c53-96df-8b927c60a310/volumes"
Jul 14 13:35:28.575661 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:28.575628    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2fmb8"]
Jul 14 13:35:28.576195 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:28.575818    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-2fmb8" podUID="06c6c11f-71fd-496b-a713-7b082c85f904" containerName="limitador" containerID="cri-o://9fd1e442b0fc8e1b78947481c523acf672a670d57cffac0aa09cd4dd89d92a80" gracePeriod=30
Jul 14 13:35:28.576607 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:28.576450    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-2fmb8"
Jul 14 13:35:29.125365 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:29.125343    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-2fmb8"
Jul 14 13:35:29.322709 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:29.322681    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqkcw\" (UniqueName: \"kubernetes.io/projected/06c6c11f-71fd-496b-a713-7b082c85f904-kube-api-access-rqkcw\") pod \"06c6c11f-71fd-496b-a713-7b082c85f904\" (UID: \"06c6c11f-71fd-496b-a713-7b082c85f904\") "
Jul 14 13:35:29.322894 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:29.322727    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/06c6c11f-71fd-496b-a713-7b082c85f904-config-file\") pod \"06c6c11f-71fd-496b-a713-7b082c85f904\" (UID: \"06c6c11f-71fd-496b-a713-7b082c85f904\") "
Jul 14 13:35:29.323094 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:29.323073    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06c6c11f-71fd-496b-a713-7b082c85f904-config-file" (OuterVolumeSpecName: "config-file") pod "06c6c11f-71fd-496b-a713-7b082c85f904" (UID: "06c6c11f-71fd-496b-a713-7b082c85f904"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 13:35:29.324737 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:29.324716    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c6c11f-71fd-496b-a713-7b082c85f904-kube-api-access-rqkcw" (OuterVolumeSpecName: "kube-api-access-rqkcw") pod "06c6c11f-71fd-496b-a713-7b082c85f904" (UID: "06c6c11f-71fd-496b-a713-7b082c85f904"). InnerVolumeSpecName "kube-api-access-rqkcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 13:35:29.373427 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:29.373396    2569 generic.go:356] "Generic (PLEG): container finished" podID="06c6c11f-71fd-496b-a713-7b082c85f904" containerID="9fd1e442b0fc8e1b78947481c523acf672a670d57cffac0aa09cd4dd89d92a80" exitCode=0
Jul 14 13:35:29.373545 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:29.373444    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-2fmb8"
Jul 14 13:35:29.373545 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:29.373494    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-2fmb8" event={"ID":"06c6c11f-71fd-496b-a713-7b082c85f904","Type":"ContainerDied","Data":"9fd1e442b0fc8e1b78947481c523acf672a670d57cffac0aa09cd4dd89d92a80"}
Jul 14 13:35:29.373637 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:29.373556    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-2fmb8" event={"ID":"06c6c11f-71fd-496b-a713-7b082c85f904","Type":"ContainerDied","Data":"8f0b08d96db80e8a52b193706f3df6dbb7205aadf66cf360cf221bb404d11e71"}
Jul 14 13:35:29.373637 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:29.373589    2569 scope.go:117] "RemoveContainer" containerID="9fd1e442b0fc8e1b78947481c523acf672a670d57cffac0aa09cd4dd89d92a80"
Jul 14 13:35:29.382304 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:29.382284    2569 scope.go:117] "RemoveContainer" containerID="9fd1e442b0fc8e1b78947481c523acf672a670d57cffac0aa09cd4dd89d92a80"
Jul 14 13:35:29.382557 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:35:29.382540    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fd1e442b0fc8e1b78947481c523acf672a670d57cffac0aa09cd4dd89d92a80\": container with ID starting with 9fd1e442b0fc8e1b78947481c523acf672a670d57cffac0aa09cd4dd89d92a80 not found: ID does not exist" containerID="9fd1e442b0fc8e1b78947481c523acf672a670d57cffac0aa09cd4dd89d92a80"
Jul 14 13:35:29.382610 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:29.382565    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fd1e442b0fc8e1b78947481c523acf672a670d57cffac0aa09cd4dd89d92a80"} err="failed to get container status \"9fd1e442b0fc8e1b78947481c523acf672a670d57cffac0aa09cd4dd89d92a80\": rpc error: code = NotFound desc = could not find container \"9fd1e442b0fc8e1b78947481c523acf672a670d57cffac0aa09cd4dd89d92a80\": container with ID starting with 9fd1e442b0fc8e1b78947481c523acf672a670d57cffac0aa09cd4dd89d92a80 not found: ID does not exist"
Jul 14 13:35:29.400859 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:29.400835    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2fmb8"]
Jul 14 13:35:29.423429 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:29.423409    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqkcw\" (UniqueName: \"kubernetes.io/projected/06c6c11f-71fd-496b-a713-7b082c85f904-kube-api-access-rqkcw\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:35:29.423514 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:29.423431    2569 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/06c6c11f-71fd-496b-a713-7b082c85f904-config-file\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:35:29.424066 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:29.424049    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2fmb8"]
Jul 14 13:35:31.193304 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:31.193266    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06c6c11f-71fd-496b-a713-7b082c85f904" path="/var/lib/kubelet/pods/06c6c11f-71fd-496b-a713-7b082c85f904/volumes"
Jul 14 13:35:46.386517 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:46.386485    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-zlc5s"]
Jul 14 13:35:46.386887 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:46.386738    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06c6c11f-71fd-496b-a713-7b082c85f904" containerName="limitador"
Jul 14 13:35:46.386887 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:46.386748    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="06c6c11f-71fd-496b-a713-7b082c85f904" containerName="limitador"
Jul 14 13:35:46.386887 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:46.386755    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c8f4f83-dd7f-4c53-96df-8b927c60a310" containerName="authorino"
Jul 14 13:35:46.386887 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:46.386760    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c8f4f83-dd7f-4c53-96df-8b927c60a310" containerName="authorino"
Jul 14 13:35:46.386887 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:46.386818    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c8f4f83-dd7f-4c53-96df-8b927c60a310" containerName="authorino"
Jul 14 13:35:46.386887 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:46.386827    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="06c6c11f-71fd-496b-a713-7b082c85f904" containerName="limitador"
Jul 14 13:35:46.389583 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:46.389569    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-zlc5s"
Jul 14 13:35:46.402919 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:46.402895    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-zlc5s"]
Jul 14 13:35:46.439638 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:46.439616    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psdwm\" (UniqueName: \"kubernetes.io/projected/4cf0e4f4-b736-47cf-b0d8-74bfc7b72691-kube-api-access-psdwm\") pod \"authorino-5d4b67c446-zlc5s\" (UID: \"4cf0e4f4-b736-47cf-b0d8-74bfc7b72691\") " pod="kuadrant-system/authorino-5d4b67c446-zlc5s"
Jul 14 13:35:46.539904 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:46.539880    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psdwm\" (UniqueName: \"kubernetes.io/projected/4cf0e4f4-b736-47cf-b0d8-74bfc7b72691-kube-api-access-psdwm\") pod \"authorino-5d4b67c446-zlc5s\" (UID: \"4cf0e4f4-b736-47cf-b0d8-74bfc7b72691\") " pod="kuadrant-system/authorino-5d4b67c446-zlc5s"
Jul 14 13:35:46.551132 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:46.551107    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psdwm\" (UniqueName: \"kubernetes.io/projected/4cf0e4f4-b736-47cf-b0d8-74bfc7b72691-kube-api-access-psdwm\") pod \"authorino-5d4b67c446-zlc5s\" (UID: \"4cf0e4f4-b736-47cf-b0d8-74bfc7b72691\") " pod="kuadrant-system/authorino-5d4b67c446-zlc5s"
Jul 14 13:35:46.636148 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:46.636126    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-zlc5s"]
Jul 14 13:35:46.636321 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:46.636310    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-zlc5s"
Jul 14 13:35:46.667909 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:46.667884    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-fc85fdc9c-fmrqr"]
Jul 14 13:35:46.671342 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:46.671324    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-fc85fdc9c-fmrqr"
Jul 14 13:35:46.688887 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:46.688861    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-fc85fdc9c-fmrqr"]
Jul 14 13:35:46.741629 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:46.741596    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng6m8\" (UniqueName: \"kubernetes.io/projected/7bc1ccca-3466-4897-ba3c-ba8341d488cf-kube-api-access-ng6m8\") pod \"authorino-fc85fdc9c-fmrqr\" (UID: \"7bc1ccca-3466-4897-ba3c-ba8341d488cf\") " pod="kuadrant-system/authorino-fc85fdc9c-fmrqr"
Jul 14 13:35:46.750146 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:46.750118    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-fc85fdc9c-fmrqr"]
Jul 14 13:35:46.750362 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:35:46.750344    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-ng6m8], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-fc85fdc9c-fmrqr" podUID="7bc1ccca-3466-4897-ba3c-ba8341d488cf"
Jul 14 13:35:46.775380 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:46.775320    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-zlc5s"]
Jul 14 13:35:46.777836 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:35:46.777811    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4cf0e4f4_b736_47cf_b0d8_74bfc7b72691.slice/crio-918c2ed27ac55a33abd4e173c4c7c42cb9e64dd8836e1d3761b774d2717efa7c WatchSource:0}: Error finding container 918c2ed27ac55a33abd4e173c4c7c42cb9e64dd8836e1d3761b774d2717efa7c: Status 404 returned error can't find the container with id 918c2ed27ac55a33abd4e173c4c7c42cb9e64dd8836e1d3761b774d2717efa7c
Jul 14 13:35:46.791605 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:46.791579    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-774844cfd9-srllq"]
Jul 14 13:35:46.795079 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:46.795062    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-774844cfd9-srllq"
Jul 14 13:35:46.797659 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:46.797644    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 14 13:35:46.805613 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:46.805592    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-774844cfd9-srllq"]
Jul 14 13:35:46.842229 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:46.842208    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng6m8\" (UniqueName: \"kubernetes.io/projected/7bc1ccca-3466-4897-ba3c-ba8341d488cf-kube-api-access-ng6m8\") pod \"authorino-fc85fdc9c-fmrqr\" (UID: \"7bc1ccca-3466-4897-ba3c-ba8341d488cf\") " pod="kuadrant-system/authorino-fc85fdc9c-fmrqr"
Jul 14 13:35:46.842311 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:46.842246    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/82ee10e7-0c93-4175-90e6-b8f765c346fa-tls-cert\") pod \"authorino-774844cfd9-srllq\" (UID: \"82ee10e7-0c93-4175-90e6-b8f765c346fa\") " pod="kuadrant-system/authorino-774844cfd9-srllq"
Jul 14 13:35:46.842311 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:46.842263    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hkh5\" (UniqueName: \"kubernetes.io/projected/82ee10e7-0c93-4175-90e6-b8f765c346fa-kube-api-access-9hkh5\") pod \"authorino-774844cfd9-srllq\" (UID: \"82ee10e7-0c93-4175-90e6-b8f765c346fa\") " pod="kuadrant-system/authorino-774844cfd9-srllq"
Jul 14 13:35:46.851878 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:46.851857    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng6m8\" (UniqueName: \"kubernetes.io/projected/7bc1ccca-3466-4897-ba3c-ba8341d488cf-kube-api-access-ng6m8\") pod \"authorino-fc85fdc9c-fmrqr\" (UID: \"7bc1ccca-3466-4897-ba3c-ba8341d488cf\") " pod="kuadrant-system/authorino-fc85fdc9c-fmrqr"
Jul 14 13:35:46.943235 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:46.943174    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/82ee10e7-0c93-4175-90e6-b8f765c346fa-tls-cert\") pod \"authorino-774844cfd9-srllq\" (UID: \"82ee10e7-0c93-4175-90e6-b8f765c346fa\") " pod="kuadrant-system/authorino-774844cfd9-srllq"
Jul 14 13:35:46.943235 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:46.943207    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hkh5\" (UniqueName: \"kubernetes.io/projected/82ee10e7-0c93-4175-90e6-b8f765c346fa-kube-api-access-9hkh5\") pod \"authorino-774844cfd9-srllq\" (UID: \"82ee10e7-0c93-4175-90e6-b8f765c346fa\") " pod="kuadrant-system/authorino-774844cfd9-srllq"
Jul 14 13:35:46.945335 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:46.945317    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/82ee10e7-0c93-4175-90e6-b8f765c346fa-tls-cert\") pod \"authorino-774844cfd9-srllq\" (UID: \"82ee10e7-0c93-4175-90e6-b8f765c346fa\") " pod="kuadrant-system/authorino-774844cfd9-srllq"
Jul 14 13:35:46.952266 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:46.952249    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hkh5\" (UniqueName: \"kubernetes.io/projected/82ee10e7-0c93-4175-90e6-b8f765c346fa-kube-api-access-9hkh5\") pod \"authorino-774844cfd9-srllq\" (UID: \"82ee10e7-0c93-4175-90e6-b8f765c346fa\") " pod="kuadrant-system/authorino-774844cfd9-srllq"
Jul 14 13:35:47.104368 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:47.104345    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-774844cfd9-srllq"
Jul 14 13:35:47.247637 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:47.246895    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-774844cfd9-srllq"]
Jul 14 13:35:47.248571 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:35:47.248544    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82ee10e7_0c93_4175_90e6_b8f765c346fa.slice/crio-9756a922ae491dabf960a629e409d02236652e0671802e253e0c78c04f50d1b2 WatchSource:0}: Error finding container 9756a922ae491dabf960a629e409d02236652e0671802e253e0c78c04f50d1b2: Status 404 returned error can't find the container with id 9756a922ae491dabf960a629e409d02236652e0671802e253e0c78c04f50d1b2
Jul 14 13:35:47.458138 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:47.458104    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-zlc5s" event={"ID":"4cf0e4f4-b736-47cf-b0d8-74bfc7b72691","Type":"ContainerStarted","Data":"1cf31998931ebc90dcefd3d14927bfa15430b6286edf56d7a0316ae2eb286cea"}
Jul 14 13:35:47.458138 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:47.458130    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-zlc5s" podUID="4cf0e4f4-b736-47cf-b0d8-74bfc7b72691" containerName="authorino" containerID="cri-o://1cf31998931ebc90dcefd3d14927bfa15430b6286edf56d7a0316ae2eb286cea" gracePeriod=30
Jul 14 13:35:47.458615 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:47.458152    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-zlc5s" event={"ID":"4cf0e4f4-b736-47cf-b0d8-74bfc7b72691","Type":"ContainerStarted","Data":"918c2ed27ac55a33abd4e173c4c7c42cb9e64dd8836e1d3761b774d2717efa7c"}
Jul 14 13:35:47.459379 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:47.459357    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-774844cfd9-srllq" event={"ID":"82ee10e7-0c93-4175-90e6-b8f765c346fa","Type":"ContainerStarted","Data":"9756a922ae491dabf960a629e409d02236652e0671802e253e0c78c04f50d1b2"}
Jul 14 13:35:47.459493 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:47.459397    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-fc85fdc9c-fmrqr"
Jul 14 13:35:47.465926 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:47.465908    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-fc85fdc9c-fmrqr"
Jul 14 13:35:47.477867 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:47.477827    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-zlc5s" podStartSLOduration=1.166082844 podStartE2EDuration="1.477815555s" totalImagesPullingTime="311.732711ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:35:47.476948651 +0000 UTC m=+512.803268579" watchObservedRunningTime="2026-07-14 13:35:47.477815555 +0000 UTC m=+512.804135483"
Jul 14 13:35:47.547639 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:47.547585    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ng6m8\" (UniqueName: \"kubernetes.io/projected/7bc1ccca-3466-4897-ba3c-ba8341d488cf-kube-api-access-ng6m8\") pod \"7bc1ccca-3466-4897-ba3c-ba8341d488cf\" (UID: \"7bc1ccca-3466-4897-ba3c-ba8341d488cf\") "
Jul 14 13:35:47.549570 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:47.549548    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc1ccca-3466-4897-ba3c-ba8341d488cf-kube-api-access-ng6m8" (OuterVolumeSpecName: "kube-api-access-ng6m8") pod "7bc1ccca-3466-4897-ba3c-ba8341d488cf" (UID: "7bc1ccca-3466-4897-ba3c-ba8341d488cf"). InnerVolumeSpecName "kube-api-access-ng6m8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 13:35:47.648424 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:47.648402    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng6m8\" (UniqueName: \"kubernetes.io/projected/7bc1ccca-3466-4897-ba3c-ba8341d488cf-kube-api-access-ng6m8\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:35:47.710205 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:47.710184    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-zlc5s"
Jul 14 13:35:47.749159 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:47.749122    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psdwm\" (UniqueName: \"kubernetes.io/projected/4cf0e4f4-b736-47cf-b0d8-74bfc7b72691-kube-api-access-psdwm\") pod \"4cf0e4f4-b736-47cf-b0d8-74bfc7b72691\" (UID: \"4cf0e4f4-b736-47cf-b0d8-74bfc7b72691\") "
Jul 14 13:35:47.751147 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:47.751121    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cf0e4f4-b736-47cf-b0d8-74bfc7b72691-kube-api-access-psdwm" (OuterVolumeSpecName: "kube-api-access-psdwm") pod "4cf0e4f4-b736-47cf-b0d8-74bfc7b72691" (UID: "4cf0e4f4-b736-47cf-b0d8-74bfc7b72691"). InnerVolumeSpecName "kube-api-access-psdwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 13:35:47.849879 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:47.849813    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psdwm\" (UniqueName: \"kubernetes.io/projected/4cf0e4f4-b736-47cf-b0d8-74bfc7b72691-kube-api-access-psdwm\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:35:48.465088 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:48.465052    2569 generic.go:356] "Generic (PLEG): container finished" podID="4cf0e4f4-b736-47cf-b0d8-74bfc7b72691" containerID="1cf31998931ebc90dcefd3d14927bfa15430b6286edf56d7a0316ae2eb286cea" exitCode=0
Jul 14 13:35:48.465544 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:48.465094    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-zlc5s"
Jul 14 13:35:48.465544 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:48.465128    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-zlc5s" event={"ID":"4cf0e4f4-b736-47cf-b0d8-74bfc7b72691","Type":"ContainerDied","Data":"1cf31998931ebc90dcefd3d14927bfa15430b6286edf56d7a0316ae2eb286cea"}
Jul 14 13:35:48.465544 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:48.465174    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-zlc5s" event={"ID":"4cf0e4f4-b736-47cf-b0d8-74bfc7b72691","Type":"ContainerDied","Data":"918c2ed27ac55a33abd4e173c4c7c42cb9e64dd8836e1d3761b774d2717efa7c"}
Jul 14 13:35:48.465544 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:48.465196    2569 scope.go:117] "RemoveContainer" containerID="1cf31998931ebc90dcefd3d14927bfa15430b6286edf56d7a0316ae2eb286cea"
Jul 14 13:35:48.466763 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:48.466741    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-fc85fdc9c-fmrqr"
Jul 14 13:35:48.466861 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:48.466738    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-774844cfd9-srllq" event={"ID":"82ee10e7-0c93-4175-90e6-b8f765c346fa","Type":"ContainerStarted","Data":"1a612499ef74dd5675cdce0ded48db2c3576b11238d6652008c5fa7b9b251d1f"}
Jul 14 13:35:48.473517 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:48.473502    2569 scope.go:117] "RemoveContainer" containerID="1cf31998931ebc90dcefd3d14927bfa15430b6286edf56d7a0316ae2eb286cea"
Jul 14 13:35:48.473818 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:35:48.473793    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cf31998931ebc90dcefd3d14927bfa15430b6286edf56d7a0316ae2eb286cea\": container with ID starting with 1cf31998931ebc90dcefd3d14927bfa15430b6286edf56d7a0316ae2eb286cea not found: ID does not exist" containerID="1cf31998931ebc90dcefd3d14927bfa15430b6286edf56d7a0316ae2eb286cea"
Jul 14 13:35:48.473893 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:48.473831    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cf31998931ebc90dcefd3d14927bfa15430b6286edf56d7a0316ae2eb286cea"} err="failed to get container status \"1cf31998931ebc90dcefd3d14927bfa15430b6286edf56d7a0316ae2eb286cea\": rpc error: code = NotFound desc = could not find container \"1cf31998931ebc90dcefd3d14927bfa15430b6286edf56d7a0316ae2eb286cea\": container with ID starting with 1cf31998931ebc90dcefd3d14927bfa15430b6286edf56d7a0316ae2eb286cea not found: ID does not exist"
Jul 14 13:35:48.510699 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:48.510643    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-774844cfd9-srllq" podStartSLOduration=2.139882551 podStartE2EDuration="2.510626202s" totalImagesPullingTime="370.743651ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:35:48.508984817 +0000 UTC m=+513.835304745" watchObservedRunningTime="2026-07-14 13:35:48.510626202 +0000 UTC m=+513.836946135"
Jul 14 13:35:48.556972 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:48.556947    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-zlc5s"]
Jul 14 13:35:48.565240 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:48.565216    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-zlc5s"]
Jul 14 13:35:48.572377 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:48.572351    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-kqdrd"]
Jul 14 13:35:48.572530 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:48.572512    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-kqdrd" podUID="8e746d96-729e-40da-a1b2-6ef375c49ee1" containerName="authorino" containerID="cri-o://aa4606b785dd99b885355256a2c09026999576213e44153c52ca9cd5d0319332" gracePeriod=30
Jul 14 13:35:48.613631 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:48.613599    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-fc85fdc9c-fmrqr"]
Jul 14 13:35:48.617458 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:48.617437    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-fc85fdc9c-fmrqr"]
Jul 14 13:35:48.813698 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:48.813678    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-kqdrd"
Jul 14 13:35:48.856159 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:48.856132    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmwl4\" (UniqueName: \"kubernetes.io/projected/8e746d96-729e-40da-a1b2-6ef375c49ee1-kube-api-access-kmwl4\") pod \"8e746d96-729e-40da-a1b2-6ef375c49ee1\" (UID: \"8e746d96-729e-40da-a1b2-6ef375c49ee1\") "
Jul 14 13:35:48.858081 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:48.858052    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e746d96-729e-40da-a1b2-6ef375c49ee1-kube-api-access-kmwl4" (OuterVolumeSpecName: "kube-api-access-kmwl4") pod "8e746d96-729e-40da-a1b2-6ef375c49ee1" (UID: "8e746d96-729e-40da-a1b2-6ef375c49ee1"). InnerVolumeSpecName "kube-api-access-kmwl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 13:35:48.957216 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:48.957180    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmwl4\" (UniqueName: \"kubernetes.io/projected/8e746d96-729e-40da-a1b2-6ef375c49ee1-kube-api-access-kmwl4\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:35:49.192952 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:49.192883    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cf0e4f4-b736-47cf-b0d8-74bfc7b72691" path="/var/lib/kubelet/pods/4cf0e4f4-b736-47cf-b0d8-74bfc7b72691/volumes"
Jul 14 13:35:49.193229 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:49.193216    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bc1ccca-3466-4897-ba3c-ba8341d488cf" path="/var/lib/kubelet/pods/7bc1ccca-3466-4897-ba3c-ba8341d488cf/volumes"
Jul 14 13:35:49.474671 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:49.474560    2569 generic.go:356] "Generic (PLEG): container finished" podID="8e746d96-729e-40da-a1b2-6ef375c49ee1" containerID="aa4606b785dd99b885355256a2c09026999576213e44153c52ca9cd5d0319332" exitCode=0
Jul 14 13:35:49.475106 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:49.474635    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-kqdrd" event={"ID":"8e746d96-729e-40da-a1b2-6ef375c49ee1","Type":"ContainerDied","Data":"aa4606b785dd99b885355256a2c09026999576213e44153c52ca9cd5d0319332"}
Jul 14 13:35:49.475106 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:49.474795    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-kqdrd" event={"ID":"8e746d96-729e-40da-a1b2-6ef375c49ee1","Type":"ContainerDied","Data":"7d9fd02f2615c28dd77bf1b135ff702dc1736a83abb8cf5789f1ac935c13e80e"}
Jul 14 13:35:49.475106 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:49.474816    2569 scope.go:117] "RemoveContainer" containerID="aa4606b785dd99b885355256a2c09026999576213e44153c52ca9cd5d0319332"
Jul 14 13:35:49.475106 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:49.474659    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-kqdrd"
Jul 14 13:35:49.484090 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:49.484069    2569 scope.go:117] "RemoveContainer" containerID="aa4606b785dd99b885355256a2c09026999576213e44153c52ca9cd5d0319332"
Jul 14 13:35:49.484352 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:35:49.484326    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa4606b785dd99b885355256a2c09026999576213e44153c52ca9cd5d0319332\": container with ID starting with aa4606b785dd99b885355256a2c09026999576213e44153c52ca9cd5d0319332 not found: ID does not exist" containerID="aa4606b785dd99b885355256a2c09026999576213e44153c52ca9cd5d0319332"
Jul 14 13:35:49.484418 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:49.484363    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa4606b785dd99b885355256a2c09026999576213e44153c52ca9cd5d0319332"} err="failed to get container status \"aa4606b785dd99b885355256a2c09026999576213e44153c52ca9cd5d0319332\": rpc error: code = NotFound desc = could not find container \"aa4606b785dd99b885355256a2c09026999576213e44153c52ca9cd5d0319332\": container with ID starting with aa4606b785dd99b885355256a2c09026999576213e44153c52ca9cd5d0319332 not found: ID does not exist"
Jul 14 13:35:49.510047 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:49.509993    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-kqdrd"]
Jul 14 13:35:49.512918 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:49.512898    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-kqdrd"]
Jul 14 13:35:51.192182 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:51.192137    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e746d96-729e-40da-a1b2-6ef375c49ee1" path="/var/lib/kubelet/pods/8e746d96-729e-40da-a1b2-6ef375c49ee1/volumes"
Jul 14 13:35:53.747414 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:53.747376    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6cdbf9df59-8g957"]
Jul 14 13:35:53.747859 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:53.747670    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e746d96-729e-40da-a1b2-6ef375c49ee1" containerName="authorino"
Jul 14 13:35:53.747859 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:53.747681    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e746d96-729e-40da-a1b2-6ef375c49ee1" containerName="authorino"
Jul 14 13:35:53.747859 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:53.747697    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cf0e4f4-b736-47cf-b0d8-74bfc7b72691" containerName="authorino"
Jul 14 13:35:53.747859 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:53.747702    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cf0e4f4-b736-47cf-b0d8-74bfc7b72691" containerName="authorino"
Jul 14 13:35:53.747859 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:53.747746    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cf0e4f4-b736-47cf-b0d8-74bfc7b72691" containerName="authorino"
Jul 14 13:35:53.747859 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:53.747755    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e746d96-729e-40da-a1b2-6ef375c49ee1" containerName="authorino"
Jul 14 13:35:53.754650 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:53.754628    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cdbf9df59-8g957"
Jul 14 13:35:53.757743 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:53.757723    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-s5tmc\""
Jul 14 13:35:53.758078 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:53.758061    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 14 13:35:53.759040 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:53.759004    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 14 13:35:53.775957 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:53.775932    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cdbf9df59-8g957"]
Jul 14 13:35:53.791304 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:53.791284    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2cq8\" (UniqueName: \"kubernetes.io/projected/bfba9323-68ea-4343-8b44-5762948bed7a-kube-api-access-l2cq8\") pod \"maas-controller-6cdbf9df59-8g957\" (UID: \"bfba9323-68ea-4343-8b44-5762948bed7a\") " pod="opendatahub/maas-controller-6cdbf9df59-8g957"
Jul 14 13:35:53.791378 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:53.791324    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bfba9323-68ea-4343-8b44-5762948bed7a-webhook-cert\") pod \"maas-controller-6cdbf9df59-8g957\" (UID: \"bfba9323-68ea-4343-8b44-5762948bed7a\") " pod="opendatahub/maas-controller-6cdbf9df59-8g957"
Jul 14 13:35:53.892359 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:53.892333    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2cq8\" (UniqueName: \"kubernetes.io/projected/bfba9323-68ea-4343-8b44-5762948bed7a-kube-api-access-l2cq8\") pod \"maas-controller-6cdbf9df59-8g957\" (UID: \"bfba9323-68ea-4343-8b44-5762948bed7a\") " pod="opendatahub/maas-controller-6cdbf9df59-8g957"
Jul 14 13:35:53.892452 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:53.892375    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bfba9323-68ea-4343-8b44-5762948bed7a-webhook-cert\") pod \"maas-controller-6cdbf9df59-8g957\" (UID: \"bfba9323-68ea-4343-8b44-5762948bed7a\") " pod="opendatahub/maas-controller-6cdbf9df59-8g957"
Jul 14 13:35:53.892537 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:35:53.892525    2569 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 14 13:35:53.892592 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:35:53.892583    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfba9323-68ea-4343-8b44-5762948bed7a-webhook-cert podName:bfba9323-68ea-4343-8b44-5762948bed7a nodeName:}" failed. No retries permitted until 2026-07-14 13:35:54.392567705 +0000 UTC m=+519.718887611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/bfba9323-68ea-4343-8b44-5762948bed7a-webhook-cert") pod "maas-controller-6cdbf9df59-8g957" (UID: "bfba9323-68ea-4343-8b44-5762948bed7a") : secret "maas-controller-webhook-cert" not found
Jul 14 13:35:53.909633 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:53.909606    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2cq8\" (UniqueName: \"kubernetes.io/projected/bfba9323-68ea-4343-8b44-5762948bed7a-kube-api-access-l2cq8\") pod \"maas-controller-6cdbf9df59-8g957\" (UID: \"bfba9323-68ea-4343-8b44-5762948bed7a\") " pod="opendatahub/maas-controller-6cdbf9df59-8g957"
Jul 14 13:35:54.395027 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:54.394992    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bfba9323-68ea-4343-8b44-5762948bed7a-webhook-cert\") pod \"maas-controller-6cdbf9df59-8g957\" (UID: \"bfba9323-68ea-4343-8b44-5762948bed7a\") " pod="opendatahub/maas-controller-6cdbf9df59-8g957"
Jul 14 13:35:54.397115 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:54.397087    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bfba9323-68ea-4343-8b44-5762948bed7a-webhook-cert\") pod \"maas-controller-6cdbf9df59-8g957\" (UID: \"bfba9323-68ea-4343-8b44-5762948bed7a\") " pod="opendatahub/maas-controller-6cdbf9df59-8g957"
Jul 14 13:35:54.664546 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:54.664474    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cdbf9df59-8g957"
Jul 14 13:35:54.791182 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:54.791159    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cdbf9df59-8g957"]
Jul 14 13:35:54.793051 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:35:54.793005    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfba9323_68ea_4343_8b44_5762948bed7a.slice/crio-f7b292dd50be33fedac68261e659de50e3e49dbdf109fc990ca7131e9885f25d WatchSource:0}: Error finding container f7b292dd50be33fedac68261e659de50e3e49dbdf109fc990ca7131e9885f25d: Status 404 returned error can't find the container with id f7b292dd50be33fedac68261e659de50e3e49dbdf109fc990ca7131e9885f25d
Jul 14 13:35:55.504896 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:55.504860    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cdbf9df59-8g957" event={"ID":"bfba9323-68ea-4343-8b44-5762948bed7a","Type":"ContainerStarted","Data":"f7b292dd50be33fedac68261e659de50e3e49dbdf109fc990ca7131e9885f25d"}
Jul 14 13:35:57.517440 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:57.517409    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cdbf9df59-8g957" event={"ID":"bfba9323-68ea-4343-8b44-5762948bed7a","Type":"ContainerStarted","Data":"078279f0027eeeb699039bb143f096af4cdfe0c33bf7f3c43fef052f26237626"}
Jul 14 13:35:57.517820 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:57.517471    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6cdbf9df59-8g957"
Jul 14 13:35:57.539041 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:35:57.538976    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6cdbf9df59-8g957" podStartSLOduration=2.608993898 podStartE2EDuration="4.538959607s" totalImagesPullingTime="1.929965709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:35:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:35:57.536913397 +0000 UTC m=+522.863233325" watchObservedRunningTime="2026-07-14 13:35:57.538959607 +0000 UTC m=+522.865279585"
Jul 14 13:36:08.524064 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:08.523973    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6cdbf9df59-8g957"
Jul 14 13:36:08.893307 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:08.893231    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6cdbf9df59-8g957"]
Jul 14 13:36:08.893447 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:08.893400    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6cdbf9df59-8g957" podUID="bfba9323-68ea-4343-8b44-5762948bed7a" containerName="manager" containerID="cri-o://078279f0027eeeb699039bb143f096af4cdfe0c33bf7f3c43fef052f26237626" gracePeriod=30
Jul 14 13:36:09.142375 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:09.142347    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cdbf9df59-8g957"
Jul 14 13:36:09.215177 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:09.215121    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bfba9323-68ea-4343-8b44-5762948bed7a-webhook-cert\") pod \"bfba9323-68ea-4343-8b44-5762948bed7a\" (UID: \"bfba9323-68ea-4343-8b44-5762948bed7a\") "
Jul 14 13:36:09.215177 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:09.215169    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2cq8\" (UniqueName: \"kubernetes.io/projected/bfba9323-68ea-4343-8b44-5762948bed7a-kube-api-access-l2cq8\") pod \"bfba9323-68ea-4343-8b44-5762948bed7a\" (UID: \"bfba9323-68ea-4343-8b44-5762948bed7a\") "
Jul 14 13:36:09.217297 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:09.217266    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfba9323-68ea-4343-8b44-5762948bed7a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "bfba9323-68ea-4343-8b44-5762948bed7a" (UID: "bfba9323-68ea-4343-8b44-5762948bed7a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 13:36:09.217414 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:09.217293    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfba9323-68ea-4343-8b44-5762948bed7a-kube-api-access-l2cq8" (OuterVolumeSpecName: "kube-api-access-l2cq8") pod "bfba9323-68ea-4343-8b44-5762948bed7a" (UID: "bfba9323-68ea-4343-8b44-5762948bed7a"). InnerVolumeSpecName "kube-api-access-l2cq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 13:36:09.316089 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:09.316066    2569 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bfba9323-68ea-4343-8b44-5762948bed7a-webhook-cert\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:36:09.316089 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:09.316087    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2cq8\" (UniqueName: \"kubernetes.io/projected/bfba9323-68ea-4343-8b44-5762948bed7a-kube-api-access-l2cq8\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:36:09.577534 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:09.577503    2569 generic.go:356] "Generic (PLEG): container finished" podID="bfba9323-68ea-4343-8b44-5762948bed7a" containerID="078279f0027eeeb699039bb143f096af4cdfe0c33bf7f3c43fef052f26237626" exitCode=0
Jul 14 13:36:09.577942 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:09.577552    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cdbf9df59-8g957"
Jul 14 13:36:09.577942 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:09.577593    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cdbf9df59-8g957" event={"ID":"bfba9323-68ea-4343-8b44-5762948bed7a","Type":"ContainerDied","Data":"078279f0027eeeb699039bb143f096af4cdfe0c33bf7f3c43fef052f26237626"}
Jul 14 13:36:09.577942 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:09.577664    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cdbf9df59-8g957" event={"ID":"bfba9323-68ea-4343-8b44-5762948bed7a","Type":"ContainerDied","Data":"f7b292dd50be33fedac68261e659de50e3e49dbdf109fc990ca7131e9885f25d"}
Jul 14 13:36:09.577942 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:09.577694    2569 scope.go:117] "RemoveContainer" containerID="078279f0027eeeb699039bb143f096af4cdfe0c33bf7f3c43fef052f26237626"
Jul 14 13:36:09.586855 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:09.586837    2569 scope.go:117] "RemoveContainer" containerID="078279f0027eeeb699039bb143f096af4cdfe0c33bf7f3c43fef052f26237626"
Jul 14 13:36:09.587144 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:36:09.587123    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"078279f0027eeeb699039bb143f096af4cdfe0c33bf7f3c43fef052f26237626\": container with ID starting with 078279f0027eeeb699039bb143f096af4cdfe0c33bf7f3c43fef052f26237626 not found: ID does not exist" containerID="078279f0027eeeb699039bb143f096af4cdfe0c33bf7f3c43fef052f26237626"
Jul 14 13:36:09.587200 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:09.587153    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"078279f0027eeeb699039bb143f096af4cdfe0c33bf7f3c43fef052f26237626"} err="failed to get container status \"078279f0027eeeb699039bb143f096af4cdfe0c33bf7f3c43fef052f26237626\": rpc error: code = NotFound desc = could not find container \"078279f0027eeeb699039bb143f096af4cdfe0c33bf7f3c43fef052f26237626\": container with ID starting with 078279f0027eeeb699039bb143f096af4cdfe0c33bf7f3c43fef052f26237626 not found: ID does not exist"
Jul 14 13:36:09.601856 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:09.601830    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6cdbf9df59-8g957"]
Jul 14 13:36:09.607055 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:09.607032    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6cdbf9df59-8g957"]
Jul 14 13:36:11.192418 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:11.192384    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfba9323-68ea-4343-8b44-5762948bed7a" path="/var/lib/kubelet/pods/bfba9323-68ea-4343-8b44-5762948bed7a/volumes"
Jul 14 13:36:29.991566 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:29.991535    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7b58d875b8-7x6xl"]
Jul 14 13:36:29.991988 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:29.991802    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfba9323-68ea-4343-8b44-5762948bed7a" containerName="manager"
Jul 14 13:36:29.991988 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:29.991812    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfba9323-68ea-4343-8b44-5762948bed7a" containerName="manager"
Jul 14 13:36:29.991988 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:29.991874    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfba9323-68ea-4343-8b44-5762948bed7a" containerName="manager"
Jul 14 13:36:29.997413 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:29.997393    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b58d875b8-7x6xl"
Jul 14 13:36:30.003234 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:30.003211    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 14 13:36:30.004338 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:30.004315    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 13:36:30.004442 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:30.004380    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 13:36:30.013777 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:30.013757    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b58d875b8-7x6xl"]
Jul 14 13:36:30.173797 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:30.173768    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks6vf\" (UniqueName: \"kubernetes.io/projected/29b396cc-7e5d-488a-8b17-c2058d40108b-kube-api-access-ks6vf\") pod \"maas-api-7b58d875b8-7x6xl\" (UID: \"29b396cc-7e5d-488a-8b17-c2058d40108b\") " pod="odh-ai-gateway-infra/maas-api-7b58d875b8-7x6xl"
Jul 14 13:36:30.173958 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:30.173822    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29b396cc-7e5d-488a-8b17-c2058d40108b-maas-api-tls\") pod \"maas-api-7b58d875b8-7x6xl\" (UID: \"29b396cc-7e5d-488a-8b17-c2058d40108b\") " pod="odh-ai-gateway-infra/maas-api-7b58d875b8-7x6xl"
Jul 14 13:36:30.173958 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:30.173848    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29b396cc-7e5d-488a-8b17-c2058d40108b-openshift-service-ca\") pod \"maas-api-7b58d875b8-7x6xl\" (UID: \"29b396cc-7e5d-488a-8b17-c2058d40108b\") " pod="odh-ai-gateway-infra/maas-api-7b58d875b8-7x6xl"
Jul 14 13:36:30.275322 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:30.275244    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks6vf\" (UniqueName: \"kubernetes.io/projected/29b396cc-7e5d-488a-8b17-c2058d40108b-kube-api-access-ks6vf\") pod \"maas-api-7b58d875b8-7x6xl\" (UID: \"29b396cc-7e5d-488a-8b17-c2058d40108b\") " pod="odh-ai-gateway-infra/maas-api-7b58d875b8-7x6xl"
Jul 14 13:36:30.275322 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:30.275296    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29b396cc-7e5d-488a-8b17-c2058d40108b-maas-api-tls\") pod \"maas-api-7b58d875b8-7x6xl\" (UID: \"29b396cc-7e5d-488a-8b17-c2058d40108b\") " pod="odh-ai-gateway-infra/maas-api-7b58d875b8-7x6xl"
Jul 14 13:36:30.275322 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:30.275313    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29b396cc-7e5d-488a-8b17-c2058d40108b-openshift-service-ca\") pod \"maas-api-7b58d875b8-7x6xl\" (UID: \"29b396cc-7e5d-488a-8b17-c2058d40108b\") " pod="odh-ai-gateway-infra/maas-api-7b58d875b8-7x6xl"
Jul 14 13:36:30.275860 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:30.275843    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29b396cc-7e5d-488a-8b17-c2058d40108b-openshift-service-ca\") pod \"maas-api-7b58d875b8-7x6xl\" (UID: \"29b396cc-7e5d-488a-8b17-c2058d40108b\") " pod="odh-ai-gateway-infra/maas-api-7b58d875b8-7x6xl"
Jul 14 13:36:30.277705 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:30.277673    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29b396cc-7e5d-488a-8b17-c2058d40108b-maas-api-tls\") pod \"maas-api-7b58d875b8-7x6xl\" (UID: \"29b396cc-7e5d-488a-8b17-c2058d40108b\") " pod="odh-ai-gateway-infra/maas-api-7b58d875b8-7x6xl"
Jul 14 13:36:30.294790 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:30.294767    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks6vf\" (UniqueName: \"kubernetes.io/projected/29b396cc-7e5d-488a-8b17-c2058d40108b-kube-api-access-ks6vf\") pod \"maas-api-7b58d875b8-7x6xl\" (UID: \"29b396cc-7e5d-488a-8b17-c2058d40108b\") " pod="odh-ai-gateway-infra/maas-api-7b58d875b8-7x6xl"
Jul 14 13:36:30.307488 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:30.307462    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b58d875b8-7x6xl"
Jul 14 13:36:30.445041 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:30.445001    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b58d875b8-7x6xl"]
Jul 14 13:36:30.446407 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:36:30.446387    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29b396cc_7e5d_488a_8b17_c2058d40108b.slice/crio-e2b389c8396e408a70bd982d46f857f5d27defc1b2aacfca29473c1c5abf0076 WatchSource:0}: Error finding container e2b389c8396e408a70bd982d46f857f5d27defc1b2aacfca29473c1c5abf0076: Status 404 returned error can't find the container with id e2b389c8396e408a70bd982d46f857f5d27defc1b2aacfca29473c1c5abf0076
Jul 14 13:36:30.680891 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:30.680812    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b58d875b8-7x6xl" event={"ID":"29b396cc-7e5d-488a-8b17-c2058d40108b","Type":"ContainerStarted","Data":"e2b389c8396e408a70bd982d46f857f5d27defc1b2aacfca29473c1c5abf0076"}
Jul 14 13:36:30.921788 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:30.921754    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-d28fh"]
Jul 14 13:36:30.926741 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:30.926723    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-d28fh"
Jul 14 13:36:30.933792 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:30.933730    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-vq5p7\""
Jul 14 13:36:30.933887 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:30.933852    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 14 13:36:30.953211 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:30.953190    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-d28fh"]
Jul 14 13:36:31.082805 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:31.082773    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pshch\" (UniqueName: \"kubernetes.io/projected/6cb5d20f-6774-416d-a71b-29eced8216f8-kube-api-access-pshch\") pod \"payload-processing-f9b8d9b9b-d28fh\" (UID: \"6cb5d20f-6774-416d-a71b-29eced8216f8\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-d28fh"
Jul 14 13:36:31.083310 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:31.082816    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb5d20f-6774-416d-a71b-29eced8216f8-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-d28fh\" (UID: \"6cb5d20f-6774-416d-a71b-29eced8216f8\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-d28fh"
Jul 14 13:36:31.183328 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:31.183290    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pshch\" (UniqueName: \"kubernetes.io/projected/6cb5d20f-6774-416d-a71b-29eced8216f8-kube-api-access-pshch\") pod \"payload-processing-f9b8d9b9b-d28fh\" (UID: \"6cb5d20f-6774-416d-a71b-29eced8216f8\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-d28fh"
Jul 14 13:36:31.183507 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:31.183338    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb5d20f-6774-416d-a71b-29eced8216f8-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-d28fh\" (UID: \"6cb5d20f-6774-416d-a71b-29eced8216f8\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-d28fh"
Jul 14 13:36:31.184075 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:31.184055    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb5d20f-6774-416d-a71b-29eced8216f8-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-d28fh\" (UID: \"6cb5d20f-6774-416d-a71b-29eced8216f8\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-d28fh"
Jul 14 13:36:31.195179 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:31.195159    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pshch\" (UniqueName: \"kubernetes.io/projected/6cb5d20f-6774-416d-a71b-29eced8216f8-kube-api-access-pshch\") pod \"payload-processing-f9b8d9b9b-d28fh\" (UID: \"6cb5d20f-6774-416d-a71b-29eced8216f8\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-d28fh"
Jul 14 13:36:31.237661 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:31.237635    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-d28fh"
Jul 14 13:36:31.389245 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:31.389135    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-d28fh"]
Jul 14 13:36:31.391709 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:36:31.391678    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb5d20f_6774_416d_a71b_29eced8216f8.slice/crio-9655d86aca7681bb2cc17b2123dddbc64a43afcb3ad324a6a435d682d65fce0e WatchSource:0}: Error finding container 9655d86aca7681bb2cc17b2123dddbc64a43afcb3ad324a6a435d682d65fce0e: Status 404 returned error can't find the container with id 9655d86aca7681bb2cc17b2123dddbc64a43afcb3ad324a6a435d682d65fce0e
Jul 14 13:36:31.687659 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:31.687602    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-d28fh" event={"ID":"6cb5d20f-6774-416d-a71b-29eced8216f8","Type":"ContainerStarted","Data":"9655d86aca7681bb2cc17b2123dddbc64a43afcb3ad324a6a435d682d65fce0e"}
Jul 14 13:36:32.694971 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:32.694934    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b58d875b8-7x6xl" event={"ID":"29b396cc-7e5d-488a-8b17-c2058d40108b","Type":"ContainerStarted","Data":"ea5d71ef62bbc7d441c49a71ee7400f712e6790fab9450c2f5ccd787e24b225b"}
Jul 14 13:36:32.695377 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:32.695062    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7b58d875b8-7x6xl"
Jul 14 13:36:32.721223 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:32.721170    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7b58d875b8-7x6xl" podStartSLOduration=2.2078946999999998 podStartE2EDuration="3.721155331s" totalImagesPullingTime="1.513260631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:36:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:36:32.719506093 +0000 UTC m=+558.045826022" watchObservedRunningTime="2026-07-14 13:36:32.721155331 +0000 UTC m=+558.047475237"
Jul 14 13:36:33.702173 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:33.702137    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-d28fh" event={"ID":"6cb5d20f-6774-416d-a71b-29eced8216f8","Type":"ContainerStarted","Data":"da92e7e2aa50921b4db56c1838f48f09015fceec4e10b4cfb044d3fce719fc78"}
Jul 14 13:36:33.724046 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:33.723978    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-f9b8d9b9b-d28fh" podStartSLOduration=2.339518718 podStartE2EDuration="3.723965171s" totalImagesPullingTime="1.384446453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:36:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:36:33.722957729 +0000 UTC m=+559.049277658" watchObservedRunningTime="2026-07-14 13:36:33.723965171 +0000 UTC m=+559.050285098"
Jul 14 13:36:34.707146 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:34.707119    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-d28fh"
Jul 14 13:36:38.708422 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:38.708394    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7b58d875b8-7x6xl"
Jul 14 13:36:45.714033 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:45.713981    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-d28fh"
Jul 14 13:36:55.504200 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:55.504174    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wzllw"]
Jul 14 13:36:55.507440 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:55.507423    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzllw"
Jul 14 13:36:55.513682 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:55.513650    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mmf2v\""
Jul 14 13:36:55.514737 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:55.514717    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 14 13:36:55.514737 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:55.514727    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 14 13:36:55.514903 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:55.514752    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 14 13:36:55.533361 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:55.533341    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wzllw"]
Jul 14 13:36:55.567966 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:55.567940    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9e3d658-7b47-4f4a-85d0-b79891998c5d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzllw\" (UID: \"b9e3d658-7b47-4f4a-85d0-b79891998c5d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzllw"
Jul 14 13:36:55.568096 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:55.567983    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l27qp\" (UniqueName: \"kubernetes.io/projected/b9e3d658-7b47-4f4a-85d0-b79891998c5d-kube-api-access-l27qp\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzllw\" (UID: \"b9e3d658-7b47-4f4a-85d0-b79891998c5d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzllw"
Jul 14 13:36:55.568096 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:55.568022    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9e3d658-7b47-4f4a-85d0-b79891998c5d-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzllw\" (UID: \"b9e3d658-7b47-4f4a-85d0-b79891998c5d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzllw"
Jul 14 13:36:55.568096 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:55.568051    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9e3d658-7b47-4f4a-85d0-b79891998c5d-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzllw\" (UID: \"b9e3d658-7b47-4f4a-85d0-b79891998c5d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzllw"
Jul 14 13:36:55.568096 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:55.568081    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9e3d658-7b47-4f4a-85d0-b79891998c5d-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzllw\" (UID: \"b9e3d658-7b47-4f4a-85d0-b79891998c5d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzllw"
Jul 14 13:36:55.568251 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:55.568104    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9e3d658-7b47-4f4a-85d0-b79891998c5d-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzllw\" (UID: \"b9e3d658-7b47-4f4a-85d0-b79891998c5d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzllw"
Jul 14 13:36:55.669393 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:55.669370    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l27qp\" (UniqueName: \"kubernetes.io/projected/b9e3d658-7b47-4f4a-85d0-b79891998c5d-kube-api-access-l27qp\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzllw\" (UID: \"b9e3d658-7b47-4f4a-85d0-b79891998c5d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzllw"
Jul 14 13:36:55.669526 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:55.669410    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9e3d658-7b47-4f4a-85d0-b79891998c5d-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzllw\" (UID: \"b9e3d658-7b47-4f4a-85d0-b79891998c5d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzllw"
Jul 14 13:36:55.669526 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:55.669434    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9e3d658-7b47-4f4a-85d0-b79891998c5d-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzllw\" (UID: \"b9e3d658-7b47-4f4a-85d0-b79891998c5d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzllw"
Jul 14 13:36:55.669526 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:55.669459    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9e3d658-7b47-4f4a-85d0-b79891998c5d-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzllw\" (UID: \"b9e3d658-7b47-4f4a-85d0-b79891998c5d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzllw"
Jul 14 13:36:55.669526 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:55.669483    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9e3d658-7b47-4f4a-85d0-b79891998c5d-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzllw\" (UID: \"b9e3d658-7b47-4f4a-85d0-b79891998c5d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzllw"
Jul 14 13:36:55.669734 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:55.669535    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9e3d658-7b47-4f4a-85d0-b79891998c5d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzllw\" (UID: \"b9e3d658-7b47-4f4a-85d0-b79891998c5d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzllw"
Jul 14 13:36:55.669804 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:55.669782    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9e3d658-7b47-4f4a-85d0-b79891998c5d-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzllw\" (UID: \"b9e3d658-7b47-4f4a-85d0-b79891998c5d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzllw"
Jul 14 13:36:55.669860 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:55.669821    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9e3d658-7b47-4f4a-85d0-b79891998c5d-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzllw\" (UID: \"b9e3d658-7b47-4f4a-85d0-b79891998c5d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzllw"
Jul 14 13:36:55.669897 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:55.669859    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9e3d658-7b47-4f4a-85d0-b79891998c5d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzllw\" (UID: \"b9e3d658-7b47-4f4a-85d0-b79891998c5d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzllw"
Jul 14 13:36:55.671760 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:55.671738    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9e3d658-7b47-4f4a-85d0-b79891998c5d-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzllw\" (UID: \"b9e3d658-7b47-4f4a-85d0-b79891998c5d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzllw"
Jul 14 13:36:55.672047 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:55.672027    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9e3d658-7b47-4f4a-85d0-b79891998c5d-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzllw\" (UID: \"b9e3d658-7b47-4f4a-85d0-b79891998c5d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzllw"
Jul 14 13:36:55.680257 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:55.680235    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l27qp\" (UniqueName: \"kubernetes.io/projected/b9e3d658-7b47-4f4a-85d0-b79891998c5d-kube-api-access-l27qp\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wzllw\" (UID: \"b9e3d658-7b47-4f4a-85d0-b79891998c5d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzllw"
Jul 14 13:36:55.817235 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:55.817178    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzllw"
Jul 14 13:36:55.946598 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:55.946571    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wzllw"]
Jul 14 13:36:55.948174 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:36:55.948143    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e3d658_7b47_4f4a_85d0_b79891998c5d.slice/crio-7ade7888cc365660e99553a8f26deb4514a8db9f93d8cb246f61ca9de8f3312e WatchSource:0}: Error finding container 7ade7888cc365660e99553a8f26deb4514a8db9f93d8cb246f61ca9de8f3312e: Status 404 returned error can't find the container with id 7ade7888cc365660e99553a8f26deb4514a8db9f93d8cb246f61ca9de8f3312e
Jul 14 13:36:56.822766 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:56.822721    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzllw" event={"ID":"b9e3d658-7b47-4f4a-85d0-b79891998c5d","Type":"ContainerStarted","Data":"7ade7888cc365660e99553a8f26deb4514a8db9f93d8cb246f61ca9de8f3312e"}
Jul 14 13:36:57.828933 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:57.828899    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzllw" event={"ID":"b9e3d658-7b47-4f4a-85d0-b79891998c5d","Type":"ContainerStarted","Data":"8a7d05315b36d15af8835d9810245d9a92ea92a7c362650c447cf63c6cd891a3"}
Jul 14 13:36:57.862773 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:36:57.862720    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzllw" podStartSLOduration=1.772819708 podStartE2EDuration="2.862697532s" totalImagesPullingTime="1.089877824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:36:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:36:57.86229175 +0000 UTC m=+583.188611679" watchObservedRunningTime="2026-07-14 13:36:57.862697532 +0000 UTC m=+583.189017463"
Jul 14 13:37:05.817594 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:05.817550    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzllw"
Jul 14 13:37:05.817594 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:05.817612    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzllw"
Jul 14 13:37:05.830181 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:05.830160    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzllw"
Jul 14 13:37:05.883141 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:05.883119    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wzllw"
Jul 14 13:37:15.057626 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:15.057598    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vq8tk_e33a489f-7d68-4e6c-9ffe-e7252049e15d/ovn-acl-logging/0.log"
Jul 14 13:37:15.059112 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:15.059093    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vq8tk_e33a489f-7d68-4e6c-9ffe-e7252049e15d/ovn-acl-logging/0.log"
Jul 14 13:37:30.000111 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:30.000083    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-74djq"]
Jul 14 13:37:30.003307 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:30.003292    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-74djq"
Jul 14 13:37:30.008230 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:30.008211    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 14 13:37:30.015420 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:30.015398    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-74djq"]
Jul 14 13:37:30.130186 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:30.130153    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fc34038-e1bc-49d8-acab-37badc7896ed-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-74djq\" (UID: \"6fc34038-e1bc-49d8-acab-37badc7896ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-74djq"
Jul 14 13:37:30.130186 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:30.130187    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fc34038-e1bc-49d8-acab-37badc7896ed-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-74djq\" (UID: \"6fc34038-e1bc-49d8-acab-37badc7896ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-74djq"
Jul 14 13:37:30.130392 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:30.130218    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fc34038-e1bc-49d8-acab-37badc7896ed-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-74djq\" (UID: \"6fc34038-e1bc-49d8-acab-37badc7896ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-74djq"
Jul 14 13:37:30.130392 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:30.130289    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fc34038-e1bc-49d8-acab-37badc7896ed-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-74djq\" (UID: \"6fc34038-e1bc-49d8-acab-37badc7896ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-74djq"
Jul 14 13:37:30.130392 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:30.130334    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8hpg\" (UniqueName: \"kubernetes.io/projected/6fc34038-e1bc-49d8-acab-37badc7896ed-kube-api-access-z8hpg\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-74djq\" (UID: \"6fc34038-e1bc-49d8-acab-37badc7896ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-74djq"
Jul 14 13:37:30.130512 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:30.130423    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fc34038-e1bc-49d8-acab-37badc7896ed-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-74djq\" (UID: \"6fc34038-e1bc-49d8-acab-37badc7896ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-74djq"
Jul 14 13:37:30.231381 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:30.231352    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fc34038-e1bc-49d8-acab-37badc7896ed-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-74djq\" (UID: \"6fc34038-e1bc-49d8-acab-37badc7896ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-74djq"
Jul 14 13:37:30.231513 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:30.231396    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fc34038-e1bc-49d8-acab-37badc7896ed-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-74djq\" (UID: \"6fc34038-e1bc-49d8-acab-37badc7896ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-74djq"
Jul 14 13:37:30.231513 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:30.231411    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fc34038-e1bc-49d8-acab-37badc7896ed-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-74djq\" (UID: \"6fc34038-e1bc-49d8-acab-37badc7896ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-74djq"
Jul 14 13:37:30.231513 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:30.231437    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fc34038-e1bc-49d8-acab-37badc7896ed-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-74djq\" (UID: \"6fc34038-e1bc-49d8-acab-37badc7896ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-74djq"
Jul 14 13:37:30.231513 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:30.231465    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fc34038-e1bc-49d8-acab-37badc7896ed-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-74djq\" (UID: \"6fc34038-e1bc-49d8-acab-37badc7896ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-74djq"
Jul 14 13:37:30.231513 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:30.231498    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8hpg\" (UniqueName: \"kubernetes.io/projected/6fc34038-e1bc-49d8-acab-37badc7896ed-kube-api-access-z8hpg\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-74djq\" (UID: \"6fc34038-e1bc-49d8-acab-37badc7896ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-74djq"
Jul 14 13:37:30.231801 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:30.231776    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fc34038-e1bc-49d8-acab-37badc7896ed-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-74djq\" (UID: \"6fc34038-e1bc-49d8-acab-37badc7896ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-74djq"
Jul 14 13:37:30.231871 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:30.231833    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fc34038-e1bc-49d8-acab-37badc7896ed-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-74djq\" (UID: \"6fc34038-e1bc-49d8-acab-37badc7896ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-74djq"
Jul 14 13:37:30.231965 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:30.231946    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fc34038-e1bc-49d8-acab-37badc7896ed-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-74djq\" (UID: \"6fc34038-e1bc-49d8-acab-37badc7896ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-74djq"
Jul 14 13:37:30.233643 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:30.233626    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fc34038-e1bc-49d8-acab-37badc7896ed-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-74djq\" (UID: \"6fc34038-e1bc-49d8-acab-37badc7896ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-74djq"
Jul 14 13:37:30.233973 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:30.233958    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fc34038-e1bc-49d8-acab-37badc7896ed-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-74djq\" (UID: \"6fc34038-e1bc-49d8-acab-37badc7896ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-74djq"
Jul 14 13:37:30.244326 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:30.244299    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8hpg\" (UniqueName: \"kubernetes.io/projected/6fc34038-e1bc-49d8-acab-37badc7896ed-kube-api-access-z8hpg\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-74djq\" (UID: \"6fc34038-e1bc-49d8-acab-37badc7896ed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-74djq"
Jul 14 13:37:30.312990 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:30.312929    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-74djq"
Jul 14 13:37:30.447757 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:30.447729    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-74djq"]
Jul 14 13:37:30.448399 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:37:30.448368    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc34038_e1bc_49d8_acab_37badc7896ed.slice/crio-06baf6b4e8fc7d9a6c768928370a4ac94a551015bada53dcc571f9a04e9b581d WatchSource:0}: Error finding container 06baf6b4e8fc7d9a6c768928370a4ac94a551015bada53dcc571f9a04e9b581d: Status 404 returned error can't find the container with id 06baf6b4e8fc7d9a6c768928370a4ac94a551015bada53dcc571f9a04e9b581d
Jul 14 13:37:30.449680 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:30.449663    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 13:37:30.993635 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:30.993604    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-74djq" event={"ID":"6fc34038-e1bc-49d8-acab-37badc7896ed","Type":"ContainerStarted","Data":"ff151249d6faedfa4c845b28fb71eac51d5645c7e118111a1cba743c926dde63"}
Jul 14 13:37:30.993784 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:30.993654    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-74djq" event={"ID":"6fc34038-e1bc-49d8-acab-37badc7896ed","Type":"ContainerStarted","Data":"06baf6b4e8fc7d9a6c768928370a4ac94a551015bada53dcc571f9a04e9b581d"}
Jul 14 13:37:31.017531 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:31.017484    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-74djq" podStartSLOduration=1.8246993630000001 podStartE2EDuration="2.017470684s" totalImagesPullingTime="192.771321ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:37:31.016904865 +0000 UTC m=+616.343224810" watchObservedRunningTime="2026-07-14 13:37:31.017470684 +0000 UTC m=+616.343790612"
Jul 14 13:37:40.313656 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:40.313577    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-74djq"
Jul 14 13:37:40.314112 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:40.313828    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-74djq"
Jul 14 13:37:40.326037 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:40.325998    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-74djq"
Jul 14 13:37:41.062415 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:41.062384    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-74djq"
Jul 14 13:37:50.905825 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:50.905793    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv"]
Jul 14 13:37:50.911348 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:50.911322    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv"
Jul 14 13:37:50.914779 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:50.914759    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 14 13:37:50.922282 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:50.922260    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv"]
Jul 14 13:37:50.995767 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:50.995740    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da3ed516-912b-4b7f-84a3-a05c30a5732f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv\" (UID: \"da3ed516-912b-4b7f-84a3-a05c30a5732f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv"
Jul 14 13:37:50.995866 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:50.995773    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da3ed516-912b-4b7f-84a3-a05c30a5732f-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv\" (UID: \"da3ed516-912b-4b7f-84a3-a05c30a5732f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv"
Jul 14 13:37:50.995866 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:50.995791    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g47wj\" (UniqueName: \"kubernetes.io/projected/da3ed516-912b-4b7f-84a3-a05c30a5732f-kube-api-access-g47wj\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv\" (UID: \"da3ed516-912b-4b7f-84a3-a05c30a5732f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv"
Jul 14 13:37:50.995866 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:50.995816    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da3ed516-912b-4b7f-84a3-a05c30a5732f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv\" (UID: \"da3ed516-912b-4b7f-84a3-a05c30a5732f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv"
Jul 14 13:37:50.995866 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:50.995847    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da3ed516-912b-4b7f-84a3-a05c30a5732f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv\" (UID: \"da3ed516-912b-4b7f-84a3-a05c30a5732f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv"
Jul 14 13:37:50.995866 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:50.995864    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da3ed516-912b-4b7f-84a3-a05c30a5732f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv\" (UID: \"da3ed516-912b-4b7f-84a3-a05c30a5732f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv"
Jul 14 13:37:51.096338 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:51.096315    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da3ed516-912b-4b7f-84a3-a05c30a5732f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv\" (UID: \"da3ed516-912b-4b7f-84a3-a05c30a5732f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv"
Jul 14 13:37:51.096451 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:51.096351    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da3ed516-912b-4b7f-84a3-a05c30a5732f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv\" (UID: \"da3ed516-912b-4b7f-84a3-a05c30a5732f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv"
Jul 14 13:37:51.096505 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:51.096446    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da3ed516-912b-4b7f-84a3-a05c30a5732f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv\" (UID: \"da3ed516-912b-4b7f-84a3-a05c30a5732f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv"
Jul 14 13:37:51.096505 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:51.096481    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da3ed516-912b-4b7f-84a3-a05c30a5732f-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv\" (UID: \"da3ed516-912b-4b7f-84a3-a05c30a5732f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv"
Jul 14 13:37:51.096596 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:51.096506    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g47wj\" (UniqueName: \"kubernetes.io/projected/da3ed516-912b-4b7f-84a3-a05c30a5732f-kube-api-access-g47wj\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv\" (UID: \"da3ed516-912b-4b7f-84a3-a05c30a5732f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv"
Jul 14 13:37:51.096596 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:51.096537    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da3ed516-912b-4b7f-84a3-a05c30a5732f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv\" (UID: \"da3ed516-912b-4b7f-84a3-a05c30a5732f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv"
Jul 14 13:37:51.096740 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:51.096712    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da3ed516-912b-4b7f-84a3-a05c30a5732f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv\" (UID: \"da3ed516-912b-4b7f-84a3-a05c30a5732f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv"
Jul 14 13:37:51.096851 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:51.096788    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da3ed516-912b-4b7f-84a3-a05c30a5732f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv\" (UID: \"da3ed516-912b-4b7f-84a3-a05c30a5732f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv"
Jul 14 13:37:51.096851 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:51.096831    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da3ed516-912b-4b7f-84a3-a05c30a5732f-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv\" (UID: \"da3ed516-912b-4b7f-84a3-a05c30a5732f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv"
Jul 14 13:37:51.098775 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:51.098749    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da3ed516-912b-4b7f-84a3-a05c30a5732f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv\" (UID: \"da3ed516-912b-4b7f-84a3-a05c30a5732f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv"
Jul 14 13:37:51.099141 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:51.099121    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da3ed516-912b-4b7f-84a3-a05c30a5732f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv\" (UID: \"da3ed516-912b-4b7f-84a3-a05c30a5732f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv"
Jul 14 13:37:51.109049 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:51.109031    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g47wj\" (UniqueName: \"kubernetes.io/projected/da3ed516-912b-4b7f-84a3-a05c30a5732f-kube-api-access-g47wj\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv\" (UID: \"da3ed516-912b-4b7f-84a3-a05c30a5732f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv"
Jul 14 13:37:51.221176 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:51.221113    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv"
Jul 14 13:37:51.354507 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:51.354484    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv"]
Jul 14 13:37:51.355749 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:37:51.355725    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda3ed516_912b_4b7f_84a3_a05c30a5732f.slice/crio-907b6b0300250415dfa8a36a6fa8306445c45bf1016e085e7d3e14ea837797b8 WatchSource:0}: Error finding container 907b6b0300250415dfa8a36a6fa8306445c45bf1016e085e7d3e14ea837797b8: Status 404 returned error can't find the container with id 907b6b0300250415dfa8a36a6fa8306445c45bf1016e085e7d3e14ea837797b8
Jul 14 13:37:52.106223 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:52.106183    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv" event={"ID":"da3ed516-912b-4b7f-84a3-a05c30a5732f","Type":"ContainerStarted","Data":"9c492d8657514e3583973c8d529cc0471f9453bfea12cea6e8b0193dbc212704"}
Jul 14 13:37:52.106642 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:37:52.106235    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv" event={"ID":"da3ed516-912b-4b7f-84a3-a05c30a5732f","Type":"ContainerStarted","Data":"907b6b0300250415dfa8a36a6fa8306445c45bf1016e085e7d3e14ea837797b8"}
Jul 14 13:38:01.221535 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:38:01.221504    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv"
Jul 14 13:38:01.221982 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:38:01.221747    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv"
Jul 14 13:38:01.234590 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:38:01.234561    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv"
Jul 14 13:38:01.267498 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:38:01.267452    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv" podStartSLOduration=11.075231184 podStartE2EDuration="11.267439742s" totalImagesPullingTime="192.208558ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:37:52.130134234 +0000 UTC m=+637.456454164" watchObservedRunningTime="2026-07-14 13:38:01.267439742 +0000 UTC m=+646.593759670"
Jul 14 13:38:02.165195 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:38:02.165167    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv"
Jul 14 13:39:01.385764 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:39:01.385730    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-fb96d7cb4-7m689"]
Jul 14 13:39:01.388959 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:39:01.388943    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-fb96d7cb4-7m689"
Jul 14 13:39:01.401172 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:39:01.401151    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-fb96d7cb4-7m689"]
Jul 14 13:39:01.427125 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:39:01.427102    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/529223ad-b813-4f48-862a-25a29e1df450-tls-cert\") pod \"authorino-fb96d7cb4-7m689\" (UID: \"529223ad-b813-4f48-862a-25a29e1df450\") " pod="kuadrant-system/authorino-fb96d7cb4-7m689"
Jul 14 13:39:01.427256 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:39:01.427133    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skcql\" (UniqueName: \"kubernetes.io/projected/529223ad-b813-4f48-862a-25a29e1df450-kube-api-access-skcql\") pod \"authorino-fb96d7cb4-7m689\" (UID: \"529223ad-b813-4f48-862a-25a29e1df450\") " pod="kuadrant-system/authorino-fb96d7cb4-7m689"
Jul 14 13:39:01.528227 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:39:01.528204    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/529223ad-b813-4f48-862a-25a29e1df450-tls-cert\") pod \"authorino-fb96d7cb4-7m689\" (UID: \"529223ad-b813-4f48-862a-25a29e1df450\") " pod="kuadrant-system/authorino-fb96d7cb4-7m689"
Jul 14 13:39:01.528328 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:39:01.528231    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skcql\" (UniqueName: \"kubernetes.io/projected/529223ad-b813-4f48-862a-25a29e1df450-kube-api-access-skcql\") pod \"authorino-fb96d7cb4-7m689\" (UID: \"529223ad-b813-4f48-862a-25a29e1df450\") " pod="kuadrant-system/authorino-fb96d7cb4-7m689"
Jul 14 13:39:01.530492 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:39:01.530473    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/529223ad-b813-4f48-862a-25a29e1df450-tls-cert\") pod \"authorino-fb96d7cb4-7m689\" (UID: \"529223ad-b813-4f48-862a-25a29e1df450\") " pod="kuadrant-system/authorino-fb96d7cb4-7m689"
Jul 14 13:39:01.540521 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:39:01.540498    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skcql\" (UniqueName: \"kubernetes.io/projected/529223ad-b813-4f48-862a-25a29e1df450-kube-api-access-skcql\") pod \"authorino-fb96d7cb4-7m689\" (UID: \"529223ad-b813-4f48-862a-25a29e1df450\") " pod="kuadrant-system/authorino-fb96d7cb4-7m689"
Jul 14 13:39:01.697150 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:39:01.697078    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-fb96d7cb4-7m689"
Jul 14 13:39:01.840801 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:39:01.840777    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-fb96d7cb4-7m689"]
Jul 14 13:39:01.842691 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:39:01.842665    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod529223ad_b813_4f48_862a_25a29e1df450.slice/crio-21e4e991c1e9597d08763fd693a42939161e8d5ab0ed16a977a0bf59bbf50de3 WatchSource:0}: Error finding container 21e4e991c1e9597d08763fd693a42939161e8d5ab0ed16a977a0bf59bbf50de3: Status 404 returned error can't find the container with id 21e4e991c1e9597d08763fd693a42939161e8d5ab0ed16a977a0bf59bbf50de3
Jul 14 13:39:02.436885 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:39:02.436853    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-fb96d7cb4-7m689" event={"ID":"529223ad-b813-4f48-862a-25a29e1df450","Type":"ContainerStarted","Data":"21e4e991c1e9597d08763fd693a42939161e8d5ab0ed16a977a0bf59bbf50de3"}
Jul 14 13:39:03.444293 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:39:03.444256    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-fb96d7cb4-7m689" event={"ID":"529223ad-b813-4f48-862a-25a29e1df450","Type":"ContainerStarted","Data":"77a0a57c234f6c66438b328867c5c4ab9acf8c9614c6fb1de0893a66fafda720"}
Jul 14 13:39:03.479171 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:39:03.479108    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-fb96d7cb4-7m689" podStartSLOduration=1.7600550560000001 podStartE2EDuration="2.479085769s" totalImagesPullingTime="719.030713ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:39:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:39:03.476357522 +0000 UTC m=+708.802677462" watchObservedRunningTime="2026-07-14 13:39:03.479085769 +0000 UTC m=+708.805405698"
Jul 14 13:39:03.529827 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:39:03.529788    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-774844cfd9-srllq"]
Jul 14 13:39:03.530050 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:39:03.530004    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-774844cfd9-srllq" podUID="82ee10e7-0c93-4175-90e6-b8f765c346fa" containerName="authorino" containerID="cri-o://1a612499ef74dd5675cdce0ded48db2c3576b11238d6652008c5fa7b9b251d1f" gracePeriod=30
Jul 14 13:39:03.774778 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:39:03.774755    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-774844cfd9-srllq"
Jul 14 13:39:03.845935 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:39:03.845888    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hkh5\" (UniqueName: \"kubernetes.io/projected/82ee10e7-0c93-4175-90e6-b8f765c346fa-kube-api-access-9hkh5\") pod \"82ee10e7-0c93-4175-90e6-b8f765c346fa\" (UID: \"82ee10e7-0c93-4175-90e6-b8f765c346fa\") "
Jul 14 13:39:03.845935 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:39:03.845923    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/82ee10e7-0c93-4175-90e6-b8f765c346fa-tls-cert\") pod \"82ee10e7-0c93-4175-90e6-b8f765c346fa\" (UID: \"82ee10e7-0c93-4175-90e6-b8f765c346fa\") "
Jul 14 13:39:03.847969 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:39:03.847939    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ee10e7-0c93-4175-90e6-b8f765c346fa-kube-api-access-9hkh5" (OuterVolumeSpecName: "kube-api-access-9hkh5") pod "82ee10e7-0c93-4175-90e6-b8f765c346fa" (UID: "82ee10e7-0c93-4175-90e6-b8f765c346fa"). InnerVolumeSpecName "kube-api-access-9hkh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 13:39:03.856131 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:39:03.856103    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82ee10e7-0c93-4175-90e6-b8f765c346fa-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "82ee10e7-0c93-4175-90e6-b8f765c346fa" (UID: "82ee10e7-0c93-4175-90e6-b8f765c346fa"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 13:39:03.947358 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:39:03.947277    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hkh5\" (UniqueName: \"kubernetes.io/projected/82ee10e7-0c93-4175-90e6-b8f765c346fa-kube-api-access-9hkh5\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:39:03.947358 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:39:03.947313    2569 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/82ee10e7-0c93-4175-90e6-b8f765c346fa-tls-cert\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:39:04.451333 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:39:04.451302    2569 generic.go:356] "Generic (PLEG): container finished" podID="82ee10e7-0c93-4175-90e6-b8f765c346fa" containerID="1a612499ef74dd5675cdce0ded48db2c3576b11238d6652008c5fa7b9b251d1f" exitCode=0
Jul 14 13:39:04.451753 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:39:04.451364    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-774844cfd9-srllq"
Jul 14 13:39:04.451753 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:39:04.451392    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-774844cfd9-srllq" event={"ID":"82ee10e7-0c93-4175-90e6-b8f765c346fa","Type":"ContainerDied","Data":"1a612499ef74dd5675cdce0ded48db2c3576b11238d6652008c5fa7b9b251d1f"}
Jul 14 13:39:04.451753 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:39:04.451445    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-774844cfd9-srllq" event={"ID":"82ee10e7-0c93-4175-90e6-b8f765c346fa","Type":"ContainerDied","Data":"9756a922ae491dabf960a629e409d02236652e0671802e253e0c78c04f50d1b2"}
Jul 14 13:39:04.451753 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:39:04.451463    2569 scope.go:117] "RemoveContainer" containerID="1a612499ef74dd5675cdce0ded48db2c3576b11238d6652008c5fa7b9b251d1f"
Jul 14 13:39:04.460162 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:39:04.460147    2569 scope.go:117] "RemoveContainer" containerID="1a612499ef74dd5675cdce0ded48db2c3576b11238d6652008c5fa7b9b251d1f"
Jul 14 13:39:04.460428 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:39:04.460408    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a612499ef74dd5675cdce0ded48db2c3576b11238d6652008c5fa7b9b251d1f\": container with ID starting with 1a612499ef74dd5675cdce0ded48db2c3576b11238d6652008c5fa7b9b251d1f not found: ID does not exist" containerID="1a612499ef74dd5675cdce0ded48db2c3576b11238d6652008c5fa7b9b251d1f"
Jul 14 13:39:04.460500 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:39:04.460436    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a612499ef74dd5675cdce0ded48db2c3576b11238d6652008c5fa7b9b251d1f"} err="failed to get container status \"1a612499ef74dd5675cdce0ded48db2c3576b11238d6652008c5fa7b9b251d1f\": rpc error: code = NotFound desc = could not find container \"1a612499ef74dd5675cdce0ded48db2c3576b11238d6652008c5fa7b9b251d1f\": container with ID starting with 1a612499ef74dd5675cdce0ded48db2c3576b11238d6652008c5fa7b9b251d1f not found: ID does not exist"
Jul 14 13:39:04.481555 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:39:04.481527    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-774844cfd9-srllq"]
Jul 14 13:39:04.486818 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:39:04.486796    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-774844cfd9-srllq"]
Jul 14 13:39:05.192470 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:39:05.192441    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82ee10e7-0c93-4175-90e6-b8f765c346fa" path="/var/lib/kubelet/pods/82ee10e7-0c93-4175-90e6-b8f765c346fa/volumes"
Jul 14 13:42:15.106161 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:42:15.106130    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vq8tk_e33a489f-7d68-4e6c-9ffe-e7252049e15d/ovn-acl-logging/0.log"
Jul 14 13:42:15.108154 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:42:15.108136    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vq8tk_e33a489f-7d68-4e6c-9ffe-e7252049e15d/ovn-acl-logging/0.log"
Jul 14 13:47:15.137909 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:47:15.137876    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vq8tk_e33a489f-7d68-4e6c-9ffe-e7252049e15d/ovn-acl-logging/0.log"
Jul 14 13:47:15.141990 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:47:15.141968    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vq8tk_e33a489f-7d68-4e6c-9ffe-e7252049e15d/ovn-acl-logging/0.log"
Jul 14 13:52:15.170622 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:52:15.170590    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vq8tk_e33a489f-7d68-4e6c-9ffe-e7252049e15d/ovn-acl-logging/0.log"
Jul 14 13:52:15.177394 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:52:15.177366    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vq8tk_e33a489f-7d68-4e6c-9ffe-e7252049e15d/ovn-acl-logging/0.log"
Jul 14 13:52:41.300344 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:52:41.300310    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw"]
Jul 14 13:52:41.300697 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:52:41.300641    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82ee10e7-0c93-4175-90e6-b8f765c346fa" containerName="authorino"
Jul 14 13:52:41.300697 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:52:41.300656    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="82ee10e7-0c93-4175-90e6-b8f765c346fa" containerName="authorino"
Jul 14 13:52:41.300779 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:52:41.300711    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="82ee10e7-0c93-4175-90e6-b8f765c346fa" containerName="authorino"
Jul 14 13:52:41.303572 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:52:41.303556    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw"
Jul 14 13:52:41.306181 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:52:41.306158    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b0840d76-serving-cert\""
Jul 14 13:52:41.306308 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:52:41.306222    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wkxqn\""
Jul 14 13:52:41.314938 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:52:41.314916    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw"]
Jul 14 13:52:41.322849 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:52:41.322826    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c16e89a5-3044-4e40-a4ae-af3f150239db-openshift-service-ca\") pod \"maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw\" (UID: \"c16e89a5-3044-4e40-a4ae-af3f150239db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw"
Jul 14 13:52:41.322938 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:52:41.322859    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c16e89a5-3044-4e40-a4ae-af3f150239db-maas-api-tls\") pod \"maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw\" (UID: \"c16e89a5-3044-4e40-a4ae-af3f150239db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw"
Jul 14 13:52:41.322938 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:52:41.322888    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnbpv\" (UniqueName: \"kubernetes.io/projected/c16e89a5-3044-4e40-a4ae-af3f150239db-kube-api-access-bnbpv\") pod \"maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw\" (UID: \"c16e89a5-3044-4e40-a4ae-af3f150239db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw"
Jul 14 13:52:41.423291 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:52:41.423270    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c16e89a5-3044-4e40-a4ae-af3f150239db-openshift-service-ca\") pod \"maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw\" (UID: \"c16e89a5-3044-4e40-a4ae-af3f150239db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw"
Jul 14 13:52:41.423437 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:52:41.423311    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c16e89a5-3044-4e40-a4ae-af3f150239db-maas-api-tls\") pod \"maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw\" (UID: \"c16e89a5-3044-4e40-a4ae-af3f150239db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw"
Jul 14 13:52:41.423437 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:52:41.423357    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnbpv\" (UniqueName: \"kubernetes.io/projected/c16e89a5-3044-4e40-a4ae-af3f150239db-kube-api-access-bnbpv\") pod \"maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw\" (UID: \"c16e89a5-3044-4e40-a4ae-af3f150239db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw"
Jul 14 13:52:41.423534 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:52:41.423472    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b0840d76-serving-cert: secret "maas-api-e2e-ait-b0840d76-serving-cert" not found
Jul 14 13:52:41.423573 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:52:41.423556    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c16e89a5-3044-4e40-a4ae-af3f150239db-maas-api-tls podName:c16e89a5-3044-4e40-a4ae-af3f150239db nodeName:}" failed. No retries permitted until 2026-07-14 13:52:41.923533891 +0000 UTC m=+1527.249853848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c16e89a5-3044-4e40-a4ae-af3f150239db-maas-api-tls") pod "maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw" (UID: "c16e89a5-3044-4e40-a4ae-af3f150239db") : secret "maas-api-e2e-ait-b0840d76-serving-cert" not found
Jul 14 13:52:41.423880 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:52:41.423861    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c16e89a5-3044-4e40-a4ae-af3f150239db-openshift-service-ca\") pod \"maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw\" (UID: \"c16e89a5-3044-4e40-a4ae-af3f150239db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw"
Jul 14 13:52:41.437116 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:52:41.437097    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnbpv\" (UniqueName: \"kubernetes.io/projected/c16e89a5-3044-4e40-a4ae-af3f150239db-kube-api-access-bnbpv\") pod \"maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw\" (UID: \"c16e89a5-3044-4e40-a4ae-af3f150239db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw"
Jul 14 13:52:41.927734 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:52:41.927703    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c16e89a5-3044-4e40-a4ae-af3f150239db-maas-api-tls\") pod \"maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw\" (UID: \"c16e89a5-3044-4e40-a4ae-af3f150239db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw"
Jul 14 13:52:41.930228 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:52:41.930207    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c16e89a5-3044-4e40-a4ae-af3f150239db-maas-api-tls\") pod \"maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw\" (UID: \"c16e89a5-3044-4e40-a4ae-af3f150239db\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw"
Jul 14 13:52:42.215228 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:52:42.215143    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw"
Jul 14 13:52:42.351350 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:52:42.351325    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw"]
Jul 14 13:52:42.352636 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:52:42.352610    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc16e89a5_3044_4e40_a4ae_af3f150239db.slice/crio-c52ec873cebcb1068db5badac31beab91fead4d6006a743b85bab6fa2811b8c4 WatchSource:0}: Error finding container c52ec873cebcb1068db5badac31beab91fead4d6006a743b85bab6fa2811b8c4: Status 404 returned error can't find the container with id c52ec873cebcb1068db5badac31beab91fead4d6006a743b85bab6fa2811b8c4
Jul 14 13:52:42.353938 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:52:42.353919    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 13:52:42.431721 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:52:42.431697    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw" event={"ID":"c16e89a5-3044-4e40-a4ae-af3f150239db","Type":"ContainerStarted","Data":"c52ec873cebcb1068db5badac31beab91fead4d6006a743b85bab6fa2811b8c4"}
Jul 14 13:52:44.442432 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:52:44.442397    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw" event={"ID":"c16e89a5-3044-4e40-a4ae-af3f150239db","Type":"ContainerStarted","Data":"f61e673055adfac150db88d611cdd7d6a1b24b2804696b67869f25b2b1cd0cc2"}
Jul 14 13:52:44.442861 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:52:44.442515    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw"
Jul 14 13:52:44.462633 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:52:44.462591    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw" podStartSLOduration=1.935525172 podStartE2EDuration="3.462578293s" totalImagesPullingTime="1.527053121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:52:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:52:44.461191021 +0000 UTC m=+1529.787510953" watchObservedRunningTime="2026-07-14 13:52:44.462578293 +0000 UTC m=+1529.788898268"
Jul 14 13:52:50.452260 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:52:50.452233    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw"
Jul 14 13:53:46.633047 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:53:46.632998    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw"]
Jul 14 13:53:46.633642 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:53:46.633272    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw" podUID="c16e89a5-3044-4e40-a4ae-af3f150239db" containerName="maas-api" containerID="cri-o://f61e673055adfac150db88d611cdd7d6a1b24b2804696b67869f25b2b1cd0cc2" gracePeriod=30
Jul 14 13:53:46.874968 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:53:46.874937    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw"
Jul 14 13:53:46.933823 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:53:46.933758    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c16e89a5-3044-4e40-a4ae-af3f150239db-maas-api-tls\") pod \"c16e89a5-3044-4e40-a4ae-af3f150239db\" (UID: \"c16e89a5-3044-4e40-a4ae-af3f150239db\") "
Jul 14 13:53:46.933823 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:53:46.933800    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnbpv\" (UniqueName: \"kubernetes.io/projected/c16e89a5-3044-4e40-a4ae-af3f150239db-kube-api-access-bnbpv\") pod \"c16e89a5-3044-4e40-a4ae-af3f150239db\" (UID: \"c16e89a5-3044-4e40-a4ae-af3f150239db\") "
Jul 14 13:53:46.934044 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:53:46.933838    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c16e89a5-3044-4e40-a4ae-af3f150239db-openshift-service-ca\") pod \"c16e89a5-3044-4e40-a4ae-af3f150239db\" (UID: \"c16e89a5-3044-4e40-a4ae-af3f150239db\") "
Jul 14 13:53:46.934221 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:53:46.934192    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c16e89a5-3044-4e40-a4ae-af3f150239db-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c16e89a5-3044-4e40-a4ae-af3f150239db" (UID: "c16e89a5-3044-4e40-a4ae-af3f150239db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 13:53:46.935947 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:53:46.935926    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c16e89a5-3044-4e40-a4ae-af3f150239db-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c16e89a5-3044-4e40-a4ae-af3f150239db" (UID: "c16e89a5-3044-4e40-a4ae-af3f150239db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 13:53:46.936074 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:53:46.936058    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c16e89a5-3044-4e40-a4ae-af3f150239db-kube-api-access-bnbpv" (OuterVolumeSpecName: "kube-api-access-bnbpv") pod "c16e89a5-3044-4e40-a4ae-af3f150239db" (UID: "c16e89a5-3044-4e40-a4ae-af3f150239db"). InnerVolumeSpecName "kube-api-access-bnbpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 13:53:47.034951 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:53:47.034928    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c16e89a5-3044-4e40-a4ae-af3f150239db-maas-api-tls\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:53:47.034951 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:53:47.034951    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnbpv\" (UniqueName: \"kubernetes.io/projected/c16e89a5-3044-4e40-a4ae-af3f150239db-kube-api-access-bnbpv\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:53:47.035131 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:53:47.034961    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c16e89a5-3044-4e40-a4ae-af3f150239db-openshift-service-ca\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:53:47.744394 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:53:47.744360    2569 generic.go:356] "Generic (PLEG): container finished" podID="c16e89a5-3044-4e40-a4ae-af3f150239db" containerID="f61e673055adfac150db88d611cdd7d6a1b24b2804696b67869f25b2b1cd0cc2" exitCode=0
Jul 14 13:53:47.744807 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:53:47.744400    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw"
Jul 14 13:53:47.744807 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:53:47.744428    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw" event={"ID":"c16e89a5-3044-4e40-a4ae-af3f150239db","Type":"ContainerDied","Data":"f61e673055adfac150db88d611cdd7d6a1b24b2804696b67869f25b2b1cd0cc2"}
Jul 14 13:53:47.744807 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:53:47.744466    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw" event={"ID":"c16e89a5-3044-4e40-a4ae-af3f150239db","Type":"ContainerDied","Data":"c52ec873cebcb1068db5badac31beab91fead4d6006a743b85bab6fa2811b8c4"}
Jul 14 13:53:47.744807 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:53:47.744487    2569 scope.go:117] "RemoveContainer" containerID="f61e673055adfac150db88d611cdd7d6a1b24b2804696b67869f25b2b1cd0cc2"
Jul 14 13:53:47.752951 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:53:47.752930    2569 scope.go:117] "RemoveContainer" containerID="f61e673055adfac150db88d611cdd7d6a1b24b2804696b67869f25b2b1cd0cc2"
Jul 14 13:53:47.753239 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:53:47.753222    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f61e673055adfac150db88d611cdd7d6a1b24b2804696b67869f25b2b1cd0cc2\": container with ID starting with f61e673055adfac150db88d611cdd7d6a1b24b2804696b67869f25b2b1cd0cc2 not found: ID does not exist" containerID="f61e673055adfac150db88d611cdd7d6a1b24b2804696b67869f25b2b1cd0cc2"
Jul 14 13:53:47.753305 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:53:47.753249    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f61e673055adfac150db88d611cdd7d6a1b24b2804696b67869f25b2b1cd0cc2"} err="failed to get container status \"f61e673055adfac150db88d611cdd7d6a1b24b2804696b67869f25b2b1cd0cc2\": rpc error: code = NotFound desc = could not find container \"f61e673055adfac150db88d611cdd7d6a1b24b2804696b67869f25b2b1cd0cc2\": container with ID starting with f61e673055adfac150db88d611cdd7d6a1b24b2804696b67869f25b2b1cd0cc2 not found: ID does not exist"
Jul 14 13:53:47.765050 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:53:47.765025    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw"]
Jul 14 13:53:47.771687 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:53:47.771667    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0840d76-78f4f6b56d-47mzw"]
Jul 14 13:53:49.192327 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:53:49.192297    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c16e89a5-3044-4e40-a4ae-af3f150239db" path="/var/lib/kubelet/pods/c16e89a5-3044-4e40-a4ae-af3f150239db/volumes"
Jul 14 13:54:03.590191 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:54:03.590156    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4"]
Jul 14 13:54:03.590596 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:54:03.590578    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c16e89a5-3044-4e40-a4ae-af3f150239db" containerName="maas-api"
Jul 14 13:54:03.590723 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:54:03.590600    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c16e89a5-3044-4e40-a4ae-af3f150239db" containerName="maas-api"
Jul 14 13:54:03.590723 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:54:03.590687    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c16e89a5-3044-4e40-a4ae-af3f150239db" containerName="maas-api"
Jul 14 13:54:03.594846 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:54:03.594827    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4"
Jul 14 13:54:03.597651 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:54:03.597626    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wkxqn\""
Jul 14 13:54:03.597760 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:54:03.597631    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-0df6d38a-serving-cert\""
Jul 14 13:54:03.603654 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:54:03.603618    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4"]
Jul 14 13:54:03.670060 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:54:03.670038    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf8b7\" (UniqueName: \"kubernetes.io/projected/01ce1fd5-a5f3-4a20-a4f3-00269218985f-kube-api-access-tf8b7\") pod \"maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4\" (UID: \"01ce1fd5-a5f3-4a20-a4f3-00269218985f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4"
Jul 14 13:54:03.670175 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:54:03.670092    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ce1fd5-a5f3-4a20-a4f3-00269218985f-openshift-service-ca\") pod \"maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4\" (UID: \"01ce1fd5-a5f3-4a20-a4f3-00269218985f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4"
Jul 14 13:54:03.670175 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:54:03.670136    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ce1fd5-a5f3-4a20-a4f3-00269218985f-maas-api-tls\") pod \"maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4\" (UID: \"01ce1fd5-a5f3-4a20-a4f3-00269218985f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4"
Jul 14 13:54:03.770740 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:54:03.770711    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf8b7\" (UniqueName: \"kubernetes.io/projected/01ce1fd5-a5f3-4a20-a4f3-00269218985f-kube-api-access-tf8b7\") pod \"maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4\" (UID: \"01ce1fd5-a5f3-4a20-a4f3-00269218985f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4"
Jul 14 13:54:03.770907 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:54:03.770765    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ce1fd5-a5f3-4a20-a4f3-00269218985f-openshift-service-ca\") pod \"maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4\" (UID: \"01ce1fd5-a5f3-4a20-a4f3-00269218985f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4"
Jul 14 13:54:03.770907 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:54:03.770792    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ce1fd5-a5f3-4a20-a4f3-00269218985f-maas-api-tls\") pod \"maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4\" (UID: \"01ce1fd5-a5f3-4a20-a4f3-00269218985f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4"
Jul 14 13:54:03.770907 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:54:03.770901    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-0df6d38a-serving-cert: secret "maas-api-e2e-migrate-0df6d38a-serving-cert" not found
Jul 14 13:54:03.771102 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:54:03.770979    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01ce1fd5-a5f3-4a20-a4f3-00269218985f-maas-api-tls podName:01ce1fd5-a5f3-4a20-a4f3-00269218985f nodeName:}" failed. No retries permitted until 2026-07-14 13:54:04.270958775 +0000 UTC m=+1609.597278683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/01ce1fd5-a5f3-4a20-a4f3-00269218985f-maas-api-tls") pod "maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4" (UID: "01ce1fd5-a5f3-4a20-a4f3-00269218985f") : secret "maas-api-e2e-migrate-0df6d38a-serving-cert" not found
Jul 14 13:54:03.771517 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:54:03.771496    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ce1fd5-a5f3-4a20-a4f3-00269218985f-openshift-service-ca\") pod \"maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4\" (UID: \"01ce1fd5-a5f3-4a20-a4f3-00269218985f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4"
Jul 14 13:54:03.780758 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:54:03.780733    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf8b7\" (UniqueName: \"kubernetes.io/projected/01ce1fd5-a5f3-4a20-a4f3-00269218985f-kube-api-access-tf8b7\") pod \"maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4\" (UID: \"01ce1fd5-a5f3-4a20-a4f3-00269218985f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4"
Jul 14 13:54:04.275787 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:54:04.275754    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ce1fd5-a5f3-4a20-a4f3-00269218985f-maas-api-tls\") pod \"maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4\" (UID: \"01ce1fd5-a5f3-4a20-a4f3-00269218985f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4"
Jul 14 13:54:04.278150 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:54:04.278127    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ce1fd5-a5f3-4a20-a4f3-00269218985f-maas-api-tls\") pod \"maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4\" (UID: \"01ce1fd5-a5f3-4a20-a4f3-00269218985f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4"
Jul 14 13:54:04.506229 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:54:04.506203    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4"
Jul 14 13:54:04.636649 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:54:04.636617    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4"]
Jul 14 13:54:04.637538 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:54:04.637513    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ce1fd5_a5f3_4a20_a4f3_00269218985f.slice/crio-6739067f2ed9b0c797e93fb72e26278cfa0d0763a31a0392da7b27791ee79d48 WatchSource:0}: Error finding container 6739067f2ed9b0c797e93fb72e26278cfa0d0763a31a0392da7b27791ee79d48: Status 404 returned error can't find the container with id 6739067f2ed9b0c797e93fb72e26278cfa0d0763a31a0392da7b27791ee79d48
Jul 14 13:54:04.835431 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:54:04.835358    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4" event={"ID":"01ce1fd5-a5f3-4a20-a4f3-00269218985f","Type":"ContainerStarted","Data":"6739067f2ed9b0c797e93fb72e26278cfa0d0763a31a0392da7b27791ee79d48"}
Jul 14 13:54:05.842088 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:54:05.842055    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4" event={"ID":"01ce1fd5-a5f3-4a20-a4f3-00269218985f","Type":"ContainerStarted","Data":"b2564f8471a4c357523b700cb6a338578c66d6c49bd593c21d5bfabd18df6cb1"}
Jul 14 13:54:05.842487 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:54:05.842110    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4"
Jul 14 13:54:05.861868 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:54:05.861820    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4" podStartSLOduration=2.445204275 podStartE2EDuration="2.861806805s" totalImagesPullingTime="416.60253ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:54:05.859933662 +0000 UTC m=+1611.186253626" watchObservedRunningTime="2026-07-14 13:54:05.861806805 +0000 UTC m=+1611.188126733"
Jul 14 13:54:11.853200 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:54:11.853170    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4"
Jul 14 13:55:08.677054 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:08.676999    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4"]
Jul 14 13:55:08.677517 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:08.677274    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4" podUID="01ce1fd5-a5f3-4a20-a4f3-00269218985f" containerName="maas-api" containerID="cri-o://b2564f8471a4c357523b700cb6a338578c66d6c49bd593c21d5bfabd18df6cb1" gracePeriod=30
Jul 14 13:55:08.916750 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:08.916727    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4"
Jul 14 13:55:08.985210 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:08.985181    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ce1fd5-a5f3-4a20-a4f3-00269218985f-openshift-service-ca\") pod \"01ce1fd5-a5f3-4a20-a4f3-00269218985f\" (UID: \"01ce1fd5-a5f3-4a20-a4f3-00269218985f\") "
Jul 14 13:55:08.985361 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:08.985237    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ce1fd5-a5f3-4a20-a4f3-00269218985f-maas-api-tls\") pod \"01ce1fd5-a5f3-4a20-a4f3-00269218985f\" (UID: \"01ce1fd5-a5f3-4a20-a4f3-00269218985f\") "
Jul 14 13:55:08.985361 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:08.985295    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tf8b7\" (UniqueName: \"kubernetes.io/projected/01ce1fd5-a5f3-4a20-a4f3-00269218985f-kube-api-access-tf8b7\") pod \"01ce1fd5-a5f3-4a20-a4f3-00269218985f\" (UID: \"01ce1fd5-a5f3-4a20-a4f3-00269218985f\") "
Jul 14 13:55:08.985523 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:08.985501    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ce1fd5-a5f3-4a20-a4f3-00269218985f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "01ce1fd5-a5f3-4a20-a4f3-00269218985f" (UID: "01ce1fd5-a5f3-4a20-a4f3-00269218985f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 13:55:08.987287 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:08.987261    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ce1fd5-a5f3-4a20-a4f3-00269218985f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "01ce1fd5-a5f3-4a20-a4f3-00269218985f" (UID: "01ce1fd5-a5f3-4a20-a4f3-00269218985f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 13:55:08.987287 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:08.987275    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ce1fd5-a5f3-4a20-a4f3-00269218985f-kube-api-access-tf8b7" (OuterVolumeSpecName: "kube-api-access-tf8b7") pod "01ce1fd5-a5f3-4a20-a4f3-00269218985f" (UID: "01ce1fd5-a5f3-4a20-a4f3-00269218985f"). InnerVolumeSpecName "kube-api-access-tf8b7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 13:55:09.085834 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:09.085810    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ce1fd5-a5f3-4a20-a4f3-00269218985f-openshift-service-ca\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:55:09.085834 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:09.085830    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ce1fd5-a5f3-4a20-a4f3-00269218985f-maas-api-tls\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:55:09.085968 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:09.085840    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf8b7\" (UniqueName: \"kubernetes.io/projected/01ce1fd5-a5f3-4a20-a4f3-00269218985f-kube-api-access-tf8b7\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:55:09.140209 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:09.140186    2569 generic.go:356] "Generic (PLEG): container finished" podID="01ce1fd5-a5f3-4a20-a4f3-00269218985f" containerID="b2564f8471a4c357523b700cb6a338578c66d6c49bd593c21d5bfabd18df6cb1" exitCode=0
Jul 14 13:55:09.140298 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:09.140233    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4"
Jul 14 13:55:09.140298 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:09.140274    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4" event={"ID":"01ce1fd5-a5f3-4a20-a4f3-00269218985f","Type":"ContainerDied","Data":"b2564f8471a4c357523b700cb6a338578c66d6c49bd593c21d5bfabd18df6cb1"}
Jul 14 13:55:09.140384 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:09.140332    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4" event={"ID":"01ce1fd5-a5f3-4a20-a4f3-00269218985f","Type":"ContainerDied","Data":"6739067f2ed9b0c797e93fb72e26278cfa0d0763a31a0392da7b27791ee79d48"}
Jul 14 13:55:09.140384 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:09.140362    2569 scope.go:117] "RemoveContainer" containerID="b2564f8471a4c357523b700cb6a338578c66d6c49bd593c21d5bfabd18df6cb1"
Jul 14 13:55:09.148766 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:09.148751    2569 scope.go:117] "RemoveContainer" containerID="b2564f8471a4c357523b700cb6a338578c66d6c49bd593c21d5bfabd18df6cb1"
Jul 14 13:55:09.149003 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:55:09.148989    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2564f8471a4c357523b700cb6a338578c66d6c49bd593c21d5bfabd18df6cb1\": container with ID starting with b2564f8471a4c357523b700cb6a338578c66d6c49bd593c21d5bfabd18df6cb1 not found: ID does not exist" containerID="b2564f8471a4c357523b700cb6a338578c66d6c49bd593c21d5bfabd18df6cb1"
Jul 14 13:55:09.149066 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:09.149037    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2564f8471a4c357523b700cb6a338578c66d6c49bd593c21d5bfabd18df6cb1"} err="failed to get container status \"b2564f8471a4c357523b700cb6a338578c66d6c49bd593c21d5bfabd18df6cb1\": rpc error: code = NotFound desc = could not find container \"b2564f8471a4c357523b700cb6a338578c66d6c49bd593c21d5bfabd18df6cb1\": container with ID starting with b2564f8471a4c357523b700cb6a338578c66d6c49bd593c21d5bfabd18df6cb1 not found: ID does not exist"
Jul 14 13:55:09.164247 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:09.164223    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4"]
Jul 14 13:55:09.169321 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:09.169298    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0df6d38a-6b6d66d548-vjnw4"]
Jul 14 13:55:09.191813 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:09.191794    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01ce1fd5-a5f3-4a20-a4f3-00269218985f" path="/var/lib/kubelet/pods/01ce1fd5-a5f3-4a20-a4f3-00269218985f/volumes"
Jul 14 13:55:15.976770 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:15.976738    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz"]
Jul 14 13:55:15.977148 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:15.977120    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01ce1fd5-a5f3-4a20-a4f3-00269218985f" containerName="maas-api"
Jul 14 13:55:15.977148 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:15.977134    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="01ce1fd5-a5f3-4a20-a4f3-00269218985f" containerName="maas-api"
Jul 14 13:55:15.977228 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:15.977193    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="01ce1fd5-a5f3-4a20-a4f3-00269218985f" containerName="maas-api"
Jul 14 13:55:15.981184 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:15.981167    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz"
Jul 14 13:55:15.983983 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:15.983963    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b3b6f94a-serving-cert\""
Jul 14 13:55:15.984109 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:15.984059    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wkxqn\""
Jul 14 13:55:16.020482 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:16.020460    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz"]
Jul 14 13:55:16.145179 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:16.145151    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e7e8370-7fdf-4c44-a805-0f1ba568f2ca-openshift-service-ca\") pod \"maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz\" (UID: \"4e7e8370-7fdf-4c44-a805-0f1ba568f2ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz"
Jul 14 13:55:16.145337 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:16.145185    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e7e8370-7fdf-4c44-a805-0f1ba568f2ca-maas-api-tls\") pod \"maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz\" (UID: \"4e7e8370-7fdf-4c44-a805-0f1ba568f2ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz"
Jul 14 13:55:16.145337 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:16.145229    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdlzs\" (UniqueName: \"kubernetes.io/projected/4e7e8370-7fdf-4c44-a805-0f1ba568f2ca-kube-api-access-tdlzs\") pod \"maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz\" (UID: \"4e7e8370-7fdf-4c44-a805-0f1ba568f2ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz"
Jul 14 13:55:16.245842 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:16.245816    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e7e8370-7fdf-4c44-a805-0f1ba568f2ca-openshift-service-ca\") pod \"maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz\" (UID: \"4e7e8370-7fdf-4c44-a805-0f1ba568f2ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz"
Jul 14 13:55:16.245973 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:16.245848    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e7e8370-7fdf-4c44-a805-0f1ba568f2ca-maas-api-tls\") pod \"maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz\" (UID: \"4e7e8370-7fdf-4c44-a805-0f1ba568f2ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz"
Jul 14 13:55:16.245973 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:16.245886    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdlzs\" (UniqueName: \"kubernetes.io/projected/4e7e8370-7fdf-4c44-a805-0f1ba568f2ca-kube-api-access-tdlzs\") pod \"maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz\" (UID: \"4e7e8370-7fdf-4c44-a805-0f1ba568f2ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz"
Jul 14 13:55:16.246081 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:55:16.245979    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b3b6f94a-serving-cert: secret "maas-api-e2e-ait-b3b6f94a-serving-cert" not found
Jul 14 13:55:16.246081 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:55:16.246073    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e7e8370-7fdf-4c44-a805-0f1ba568f2ca-maas-api-tls podName:4e7e8370-7fdf-4c44-a805-0f1ba568f2ca nodeName:}" failed. No retries permitted until 2026-07-14 13:55:16.746049434 +0000 UTC m=+1682.072369343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4e7e8370-7fdf-4c44-a805-0f1ba568f2ca-maas-api-tls") pod "maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz" (UID: "4e7e8370-7fdf-4c44-a805-0f1ba568f2ca") : secret "maas-api-e2e-ait-b3b6f94a-serving-cert" not found
Jul 14 13:55:16.246534 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:16.246512    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e7e8370-7fdf-4c44-a805-0f1ba568f2ca-openshift-service-ca\") pod \"maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz\" (UID: \"4e7e8370-7fdf-4c44-a805-0f1ba568f2ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz"
Jul 14 13:55:16.257490 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:16.257463    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdlzs\" (UniqueName: \"kubernetes.io/projected/4e7e8370-7fdf-4c44-a805-0f1ba568f2ca-kube-api-access-tdlzs\") pod \"maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz\" (UID: \"4e7e8370-7fdf-4c44-a805-0f1ba568f2ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz"
Jul 14 13:55:16.750372 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:16.750347    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e7e8370-7fdf-4c44-a805-0f1ba568f2ca-maas-api-tls\") pod \"maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz\" (UID: \"4e7e8370-7fdf-4c44-a805-0f1ba568f2ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz"
Jul 14 13:55:16.752803 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:16.752767    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e7e8370-7fdf-4c44-a805-0f1ba568f2ca-maas-api-tls\") pod \"maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz\" (UID: \"4e7e8370-7fdf-4c44-a805-0f1ba568f2ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz"
Jul 14 13:55:16.890872 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:16.890842    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz"
Jul 14 13:55:17.030199 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:17.030176    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz"]
Jul 14 13:55:17.031900 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:55:17.031874    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e7e8370_7fdf_4c44_a805_0f1ba568f2ca.slice/crio-97774c3c2490a4445f10814012568baa9c36dea72623e68b59dc60d04b3e2eaa WatchSource:0}: Error finding container 97774c3c2490a4445f10814012568baa9c36dea72623e68b59dc60d04b3e2eaa: Status 404 returned error can't find the container with id 97774c3c2490a4445f10814012568baa9c36dea72623e68b59dc60d04b3e2eaa
Jul 14 13:55:17.179929 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:17.179899    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz" event={"ID":"4e7e8370-7fdf-4c44-a805-0f1ba568f2ca","Type":"ContainerStarted","Data":"97774c3c2490a4445f10814012568baa9c36dea72623e68b59dc60d04b3e2eaa"}
Jul 14 13:55:18.186775 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:18.186744    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz" event={"ID":"4e7e8370-7fdf-4c44-a805-0f1ba568f2ca","Type":"ContainerStarted","Data":"7917b3e8971d64e52ebe9fbf81c446a030809dfd8ac89072bb5d57bb4c1b95d9"}
Jul 14 13:55:18.187143 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:18.186837    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz"
Jul 14 13:55:18.218406 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:18.218328    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz" podStartSLOduration=2.60566101 podStartE2EDuration="3.218297456s" totalImagesPullingTime="612.636446ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:55:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:55:18.214911804 +0000 UTC m=+1683.541231736" watchObservedRunningTime="2026-07-14 13:55:18.218297456 +0000 UTC m=+1683.544617385"
Jul 14 13:55:24.202914 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:55:24.202879    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz"
Jul 14 13:56:31.458951 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:31.458921    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz"]
Jul 14 13:56:31.459480 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:31.459122    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz" podUID="4e7e8370-7fdf-4c44-a805-0f1ba568f2ca" containerName="maas-api" containerID="cri-o://7917b3e8971d64e52ebe9fbf81c446a030809dfd8ac89072bb5d57bb4c1b95d9" gracePeriod=30
Jul 14 13:56:31.701885 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:31.701861    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz"
Jul 14 13:56:31.839355 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:31.839331    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e7e8370-7fdf-4c44-a805-0f1ba568f2ca-maas-api-tls\") pod \"4e7e8370-7fdf-4c44-a805-0f1ba568f2ca\" (UID: \"4e7e8370-7fdf-4c44-a805-0f1ba568f2ca\") "
Jul 14 13:56:31.839505 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:31.839405    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e7e8370-7fdf-4c44-a805-0f1ba568f2ca-openshift-service-ca\") pod \"4e7e8370-7fdf-4c44-a805-0f1ba568f2ca\" (UID: \"4e7e8370-7fdf-4c44-a805-0f1ba568f2ca\") "
Jul 14 13:56:31.839505 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:31.839439    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdlzs\" (UniqueName: \"kubernetes.io/projected/4e7e8370-7fdf-4c44-a805-0f1ba568f2ca-kube-api-access-tdlzs\") pod \"4e7e8370-7fdf-4c44-a805-0f1ba568f2ca\" (UID: \"4e7e8370-7fdf-4c44-a805-0f1ba568f2ca\") "
Jul 14 13:56:31.839790 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:31.839759    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e7e8370-7fdf-4c44-a805-0f1ba568f2ca-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4e7e8370-7fdf-4c44-a805-0f1ba568f2ca" (UID: "4e7e8370-7fdf-4c44-a805-0f1ba568f2ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 13:56:31.841411 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:31.841383    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e7e8370-7fdf-4c44-a805-0f1ba568f2ca-kube-api-access-tdlzs" (OuterVolumeSpecName: "kube-api-access-tdlzs") pod "4e7e8370-7fdf-4c44-a805-0f1ba568f2ca" (UID: "4e7e8370-7fdf-4c44-a805-0f1ba568f2ca"). InnerVolumeSpecName "kube-api-access-tdlzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 13:56:31.841496 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:31.841409    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e7e8370-7fdf-4c44-a805-0f1ba568f2ca-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4e7e8370-7fdf-4c44-a805-0f1ba568f2ca" (UID: "4e7e8370-7fdf-4c44-a805-0f1ba568f2ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 13:56:31.940367 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:31.940343    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e7e8370-7fdf-4c44-a805-0f1ba568f2ca-openshift-service-ca\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:56:31.940367 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:31.940364    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdlzs\" (UniqueName: \"kubernetes.io/projected/4e7e8370-7fdf-4c44-a805-0f1ba568f2ca-kube-api-access-tdlzs\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:56:31.940493 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:31.940373    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e7e8370-7fdf-4c44-a805-0f1ba568f2ca-maas-api-tls\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:56:32.544522 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:32.544486    2569 generic.go:356] "Generic (PLEG): container finished" podID="4e7e8370-7fdf-4c44-a805-0f1ba568f2ca" containerID="7917b3e8971d64e52ebe9fbf81c446a030809dfd8ac89072bb5d57bb4c1b95d9" exitCode=0
Jul 14 13:56:32.544991 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:32.544531    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz"
Jul 14 13:56:32.544991 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:32.544568    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz" event={"ID":"4e7e8370-7fdf-4c44-a805-0f1ba568f2ca","Type":"ContainerDied","Data":"7917b3e8971d64e52ebe9fbf81c446a030809dfd8ac89072bb5d57bb4c1b95d9"}
Jul 14 13:56:32.544991 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:32.544613    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz" event={"ID":"4e7e8370-7fdf-4c44-a805-0f1ba568f2ca","Type":"ContainerDied","Data":"97774c3c2490a4445f10814012568baa9c36dea72623e68b59dc60d04b3e2eaa"}
Jul 14 13:56:32.544991 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:32.544637    2569 scope.go:117] "RemoveContainer" containerID="7917b3e8971d64e52ebe9fbf81c446a030809dfd8ac89072bb5d57bb4c1b95d9"
Jul 14 13:56:32.559629 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:32.559612    2569 scope.go:117] "RemoveContainer" containerID="7917b3e8971d64e52ebe9fbf81c446a030809dfd8ac89072bb5d57bb4c1b95d9"
Jul 14 13:56:32.559896 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:56:32.559879    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7917b3e8971d64e52ebe9fbf81c446a030809dfd8ac89072bb5d57bb4c1b95d9\": container with ID starting with 7917b3e8971d64e52ebe9fbf81c446a030809dfd8ac89072bb5d57bb4c1b95d9 not found: ID does not exist" containerID="7917b3e8971d64e52ebe9fbf81c446a030809dfd8ac89072bb5d57bb4c1b95d9"
Jul 14 13:56:32.559970 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:32.559900    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7917b3e8971d64e52ebe9fbf81c446a030809dfd8ac89072bb5d57bb4c1b95d9"} err="failed to get container status \"7917b3e8971d64e52ebe9fbf81c446a030809dfd8ac89072bb5d57bb4c1b95d9\": rpc error: code = NotFound desc = could not find container \"7917b3e8971d64e52ebe9fbf81c446a030809dfd8ac89072bb5d57bb4c1b95d9\": container with ID starting with 7917b3e8971d64e52ebe9fbf81c446a030809dfd8ac89072bb5d57bb4c1b95d9 not found: ID does not exist"
Jul 14 13:56:32.580928 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:32.580905    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz"]
Jul 14 13:56:32.590251 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:32.590229    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b3b6f94a-cdc957898-kgxhz"]
Jul 14 13:56:33.192303 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:33.192272    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e7e8370-7fdf-4c44-a805-0f1ba568f2ca" path="/var/lib/kubelet/pods/4e7e8370-7fdf-4c44-a805-0f1ba568f2ca/volumes"
Jul 14 13:56:48.713241 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:48.713204    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk"]
Jul 14 13:56:48.713678 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:48.713663    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e7e8370-7fdf-4c44-a805-0f1ba568f2ca" containerName="maas-api"
Jul 14 13:56:48.713721 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:48.713681    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e7e8370-7fdf-4c44-a805-0f1ba568f2ca" containerName="maas-api"
Jul 14 13:56:48.713784 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:48.713773    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e7e8370-7fdf-4c44-a805-0f1ba568f2ca" containerName="maas-api"
Jul 14 13:56:48.716801 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:48.716785    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk"
Jul 14 13:56:48.719293 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:48.719271    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-fe2b25e2-serving-cert\""
Jul 14 13:56:48.719419 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:48.719327    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wkxqn\""
Jul 14 13:56:48.727533 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:48.727514    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk"]
Jul 14 13:56:48.774223 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:48.774198    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88032d4b-5ada-4861-9a68-047a62ef806e-openshift-service-ca\") pod \"maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk\" (UID: \"88032d4b-5ada-4861-9a68-047a62ef806e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk"
Jul 14 13:56:48.774349 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:48.774241    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdpxq\" (UniqueName: \"kubernetes.io/projected/88032d4b-5ada-4861-9a68-047a62ef806e-kube-api-access-gdpxq\") pod \"maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk\" (UID: \"88032d4b-5ada-4861-9a68-047a62ef806e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk"
Jul 14 13:56:48.774349 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:48.774302    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88032d4b-5ada-4861-9a68-047a62ef806e-maas-api-tls\") pod \"maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk\" (UID: \"88032d4b-5ada-4861-9a68-047a62ef806e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk"
Jul 14 13:56:48.874762 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:48.874740    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88032d4b-5ada-4861-9a68-047a62ef806e-openshift-service-ca\") pod \"maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk\" (UID: \"88032d4b-5ada-4861-9a68-047a62ef806e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk"
Jul 14 13:56:48.874890 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:48.874778    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdpxq\" (UniqueName: \"kubernetes.io/projected/88032d4b-5ada-4861-9a68-047a62ef806e-kube-api-access-gdpxq\") pod \"maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk\" (UID: \"88032d4b-5ada-4861-9a68-047a62ef806e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk"
Jul 14 13:56:48.874890 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:48.874805    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88032d4b-5ada-4861-9a68-047a62ef806e-maas-api-tls\") pod \"maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk\" (UID: \"88032d4b-5ada-4861-9a68-047a62ef806e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk"
Jul 14 13:56:48.874970 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:56:48.874941    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-fe2b25e2-serving-cert: secret "maas-api-e2e-derive-fe2b25e2-serving-cert" not found
Jul 14 13:56:48.875044 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:56:48.875006    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88032d4b-5ada-4861-9a68-047a62ef806e-maas-api-tls podName:88032d4b-5ada-4861-9a68-047a62ef806e nodeName:}" failed. No retries permitted until 2026-07-14 13:56:49.374988334 +0000 UTC m=+1774.701308257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/88032d4b-5ada-4861-9a68-047a62ef806e-maas-api-tls") pod "maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk" (UID: "88032d4b-5ada-4861-9a68-047a62ef806e") : secret "maas-api-e2e-derive-fe2b25e2-serving-cert" not found
Jul 14 13:56:48.875373 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:48.875355    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88032d4b-5ada-4861-9a68-047a62ef806e-openshift-service-ca\") pod \"maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk\" (UID: \"88032d4b-5ada-4861-9a68-047a62ef806e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk"
Jul 14 13:56:48.885198 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:48.885181    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdpxq\" (UniqueName: \"kubernetes.io/projected/88032d4b-5ada-4861-9a68-047a62ef806e-kube-api-access-gdpxq\") pod \"maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk\" (UID: \"88032d4b-5ada-4861-9a68-047a62ef806e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk"
Jul 14 13:56:49.378415 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:49.378382    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88032d4b-5ada-4861-9a68-047a62ef806e-maas-api-tls\") pod \"maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk\" (UID: \"88032d4b-5ada-4861-9a68-047a62ef806e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk"
Jul 14 13:56:49.380692 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:49.380674    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88032d4b-5ada-4861-9a68-047a62ef806e-maas-api-tls\") pod \"maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk\" (UID: \"88032d4b-5ada-4861-9a68-047a62ef806e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk"
Jul 14 13:56:49.628131 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:49.628093    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk"
Jul 14 13:56:49.757062 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:49.756999    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk"]
Jul 14 13:56:49.758579 ip-10-0-140-78 kubenswrapper[2569]: W0714 13:56:49.758556    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88032d4b_5ada_4861_9a68_047a62ef806e.slice/crio-3aeb9556353cf583c36ac248b33a57eaa8a510e7b81f64b40f14678a94692157 WatchSource:0}: Error finding container 3aeb9556353cf583c36ac248b33a57eaa8a510e7b81f64b40f14678a94692157: Status 404 returned error can't find the container with id 3aeb9556353cf583c36ac248b33a57eaa8a510e7b81f64b40f14678a94692157
Jul 14 13:56:50.637574 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:50.637543    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk" event={"ID":"88032d4b-5ada-4861-9a68-047a62ef806e","Type":"ContainerStarted","Data":"177e9c24a42da43cb27bcba32466333157158e1415f0f1c9e80842ec867d86e7"}
Jul 14 13:56:50.637760 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:50.637591    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk" event={"ID":"88032d4b-5ada-4861-9a68-047a62ef806e","Type":"ContainerStarted","Data":"3aeb9556353cf583c36ac248b33a57eaa8a510e7b81f64b40f14678a94692157"}
Jul 14 13:56:50.637760 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:50.637656    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk"
Jul 14 13:56:50.660526 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:50.660482    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk" podStartSLOduration=2.254959253 podStartE2EDuration="2.660467064s" totalImagesPullingTime="405.507811ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 13:56:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 13:56:50.65910275 +0000 UTC m=+1775.985422678" watchObservedRunningTime="2026-07-14 13:56:50.660467064 +0000 UTC m=+1775.986786991"
Jul 14 13:56:56.647376 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:56:56.647348    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk"
Jul 14 13:57:15.206091 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:57:15.205985    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vq8tk_e33a489f-7d68-4e6c-9ffe-e7252049e15d/ovn-acl-logging/0.log"
Jul 14 13:57:15.213845 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:57:15.213829    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vq8tk_e33a489f-7d68-4e6c-9ffe-e7252049e15d/ovn-acl-logging/0.log"
Jul 14 13:58:03.539473 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:58:03.539443    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk"]
Jul 14 13:58:03.539924 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:58:03.539633    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk" podUID="88032d4b-5ada-4861-9a68-047a62ef806e" containerName="maas-api" containerID="cri-o://177e9c24a42da43cb27bcba32466333157158e1415f0f1c9e80842ec867d86e7" gracePeriod=30
Jul 14 13:58:03.776912 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:58:03.776888    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk"
Jul 14 13:58:03.844613 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:58:03.844552    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88032d4b-5ada-4861-9a68-047a62ef806e-maas-api-tls\") pod \"88032d4b-5ada-4861-9a68-047a62ef806e\" (UID: \"88032d4b-5ada-4861-9a68-047a62ef806e\") "
Jul 14 13:58:03.844613 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:58:03.844586    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88032d4b-5ada-4861-9a68-047a62ef806e-openshift-service-ca\") pod \"88032d4b-5ada-4861-9a68-047a62ef806e\" (UID: \"88032d4b-5ada-4861-9a68-047a62ef806e\") "
Jul 14 13:58:03.844807 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:58:03.844636    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdpxq\" (UniqueName: \"kubernetes.io/projected/88032d4b-5ada-4861-9a68-047a62ef806e-kube-api-access-gdpxq\") pod \"88032d4b-5ada-4861-9a68-047a62ef806e\" (UID: \"88032d4b-5ada-4861-9a68-047a62ef806e\") "
Jul 14 13:58:03.844961 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:58:03.844935    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88032d4b-5ada-4861-9a68-047a62ef806e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "88032d4b-5ada-4861-9a68-047a62ef806e" (UID: "88032d4b-5ada-4861-9a68-047a62ef806e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 13:58:03.846633 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:58:03.846611    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88032d4b-5ada-4861-9a68-047a62ef806e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "88032d4b-5ada-4861-9a68-047a62ef806e" (UID: "88032d4b-5ada-4861-9a68-047a62ef806e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 13:58:03.846768 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:58:03.846746    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88032d4b-5ada-4861-9a68-047a62ef806e-kube-api-access-gdpxq" (OuterVolumeSpecName: "kube-api-access-gdpxq") pod "88032d4b-5ada-4861-9a68-047a62ef806e" (UID: "88032d4b-5ada-4861-9a68-047a62ef806e"). InnerVolumeSpecName "kube-api-access-gdpxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 13:58:03.945332 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:58:03.945306    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88032d4b-5ada-4861-9a68-047a62ef806e-maas-api-tls\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:58:03.945332 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:58:03.945331    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88032d4b-5ada-4861-9a68-047a62ef806e-openshift-service-ca\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:58:03.945454 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:58:03.945345    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdpxq\" (UniqueName: \"kubernetes.io/projected/88032d4b-5ada-4861-9a68-047a62ef806e-kube-api-access-gdpxq\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 13:58:04.000728 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:58:04.000704    2569 generic.go:356] "Generic (PLEG): container finished" podID="88032d4b-5ada-4861-9a68-047a62ef806e" containerID="177e9c24a42da43cb27bcba32466333157158e1415f0f1c9e80842ec867d86e7" exitCode=0
Jul 14 13:58:04.000811 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:58:04.000747    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk"
Jul 14 13:58:04.000811 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:58:04.000757    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk" event={"ID":"88032d4b-5ada-4861-9a68-047a62ef806e","Type":"ContainerDied","Data":"177e9c24a42da43cb27bcba32466333157158e1415f0f1c9e80842ec867d86e7"}
Jul 14 13:58:04.000811 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:58:04.000797    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk" event={"ID":"88032d4b-5ada-4861-9a68-047a62ef806e","Type":"ContainerDied","Data":"3aeb9556353cf583c36ac248b33a57eaa8a510e7b81f64b40f14678a94692157"}
Jul 14 13:58:04.000916 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:58:04.000816    2569 scope.go:117] "RemoveContainer" containerID="177e9c24a42da43cb27bcba32466333157158e1415f0f1c9e80842ec867d86e7"
Jul 14 13:58:04.009504 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:58:04.009487    2569 scope.go:117] "RemoveContainer" containerID="177e9c24a42da43cb27bcba32466333157158e1415f0f1c9e80842ec867d86e7"
Jul 14 13:58:04.009740 ip-10-0-140-78 kubenswrapper[2569]: E0714 13:58:04.009721    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"177e9c24a42da43cb27bcba32466333157158e1415f0f1c9e80842ec867d86e7\": container with ID starting with 177e9c24a42da43cb27bcba32466333157158e1415f0f1c9e80842ec867d86e7 not found: ID does not exist" containerID="177e9c24a42da43cb27bcba32466333157158e1415f0f1c9e80842ec867d86e7"
Jul 14 13:58:04.009804 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:58:04.009753    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"177e9c24a42da43cb27bcba32466333157158e1415f0f1c9e80842ec867d86e7"} err="failed to get container status \"177e9c24a42da43cb27bcba32466333157158e1415f0f1c9e80842ec867d86e7\": rpc error: code = NotFound desc = could not find container \"177e9c24a42da43cb27bcba32466333157158e1415f0f1c9e80842ec867d86e7\": container with ID starting with 177e9c24a42da43cb27bcba32466333157158e1415f0f1c9e80842ec867d86e7 not found: ID does not exist"
Jul 14 13:58:04.026134 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:58:04.026114    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk"]
Jul 14 13:58:04.032022 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:58:04.031992    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fe2b25e2-6c594d77dc-vhfjk"]
Jul 14 13:58:05.192290 ip-10-0-140-78 kubenswrapper[2569]: I0714 13:58:05.192261    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88032d4b-5ada-4861-9a68-047a62ef806e" path="/var/lib/kubelet/pods/88032d4b-5ada-4861-9a68-047a62ef806e/volumes"
Jul 14 14:00:00.094435 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:00.094403    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv"]
Jul 14 14:00:00.094919 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:00.094704    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88032d4b-5ada-4861-9a68-047a62ef806e" containerName="maas-api"
Jul 14 14:00:00.094919 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:00.094714    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="88032d4b-5ada-4861-9a68-047a62ef806e" containerName="maas-api"
Jul 14 14:00:00.094919 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:00.094785    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="88032d4b-5ada-4861-9a68-047a62ef806e" containerName="maas-api"
Jul 14 14:00:00.097627 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:00.097608    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv"
Jul 14 14:00:00.100302 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:00.100283    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-50aeb779-serving-cert\""
Jul 14 14:00:00.100420 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:00.100376    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wkxqn\""
Jul 14 14:00:00.110098 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:00.110076    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv"]
Jul 14 14:00:00.237169 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:00.237136    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-842l4\" (UniqueName: \"kubernetes.io/projected/593f875f-a4d9-47cc-859d-b07bac776964-kube-api-access-842l4\") pod \"maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv\" (UID: \"593f875f-a4d9-47cc-859d-b07bac776964\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv"
Jul 14 14:00:00.237328 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:00.237245    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/593f875f-a4d9-47cc-859d-b07bac776964-openshift-service-ca\") pod \"maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv\" (UID: \"593f875f-a4d9-47cc-859d-b07bac776964\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv"
Jul 14 14:00:00.237328 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:00.237286    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/593f875f-a4d9-47cc-859d-b07bac776964-maas-api-tls\") pod \"maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv\" (UID: \"593f875f-a4d9-47cc-859d-b07bac776964\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv"
Jul 14 14:00:00.338552 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:00.338520    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/593f875f-a4d9-47cc-859d-b07bac776964-maas-api-tls\") pod \"maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv\" (UID: \"593f875f-a4d9-47cc-859d-b07bac776964\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv"
Jul 14 14:00:00.338701 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:00.338600    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-842l4\" (UniqueName: \"kubernetes.io/projected/593f875f-a4d9-47cc-859d-b07bac776964-kube-api-access-842l4\") pod \"maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv\" (UID: \"593f875f-a4d9-47cc-859d-b07bac776964\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv"
Jul 14 14:00:00.338701 ip-10-0-140-78 kubenswrapper[2569]: E0714 14:00:00.338670    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-50aeb779-serving-cert: secret "maas-api-e2e-mt-50aeb779-serving-cert" not found
Jul 14 14:00:00.338701 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:00.338677    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/593f875f-a4d9-47cc-859d-b07bac776964-openshift-service-ca\") pod \"maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv\" (UID: \"593f875f-a4d9-47cc-859d-b07bac776964\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv"
Jul 14 14:00:00.338861 ip-10-0-140-78 kubenswrapper[2569]: E0714 14:00:00.338738    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/593f875f-a4d9-47cc-859d-b07bac776964-maas-api-tls podName:593f875f-a4d9-47cc-859d-b07bac776964 nodeName:}" failed. No retries permitted until 2026-07-14 14:00:00.838719879 +0000 UTC m=+1966.165039796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/593f875f-a4d9-47cc-859d-b07bac776964-maas-api-tls") pod "maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv" (UID: "593f875f-a4d9-47cc-859d-b07bac776964") : secret "maas-api-e2e-mt-50aeb779-serving-cert" not found
Jul 14 14:00:00.339307 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:00.339287    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/593f875f-a4d9-47cc-859d-b07bac776964-openshift-service-ca\") pod \"maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv\" (UID: \"593f875f-a4d9-47cc-859d-b07bac776964\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv"
Jul 14 14:00:00.349649 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:00.349593    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-842l4\" (UniqueName: \"kubernetes.io/projected/593f875f-a4d9-47cc-859d-b07bac776964-kube-api-access-842l4\") pod \"maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv\" (UID: \"593f875f-a4d9-47cc-859d-b07bac776964\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv"
Jul 14 14:00:00.842929 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:00.842890    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/593f875f-a4d9-47cc-859d-b07bac776964-maas-api-tls\") pod \"maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv\" (UID: \"593f875f-a4d9-47cc-859d-b07bac776964\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv"
Jul 14 14:00:00.845296 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:00.845272    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/593f875f-a4d9-47cc-859d-b07bac776964-maas-api-tls\") pod \"maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv\" (UID: \"593f875f-a4d9-47cc-859d-b07bac776964\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv"
Jul 14 14:00:01.007445 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:01.007414    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv"
Jul 14 14:00:01.146351 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:01.146330    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv"]
Jul 14 14:00:01.147862 ip-10-0-140-78 kubenswrapper[2569]: W0714 14:00:01.147840    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod593f875f_a4d9_47cc_859d_b07bac776964.slice/crio-4953daf38a38decd6a3421c80dea8e06b42ed969a2c1dc5a313a874fc748474e WatchSource:0}: Error finding container 4953daf38a38decd6a3421c80dea8e06b42ed969a2c1dc5a313a874fc748474e: Status 404 returned error can't find the container with id 4953daf38a38decd6a3421c80dea8e06b42ed969a2c1dc5a313a874fc748474e
Jul 14 14:00:01.149170 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:01.149149    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 14:00:01.572082 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:01.572048    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv" event={"ID":"593f875f-a4d9-47cc-859d-b07bac776964","Type":"ContainerStarted","Data":"4953daf38a38decd6a3421c80dea8e06b42ed969a2c1dc5a313a874fc748474e"}
Jul 14 14:00:02.578568 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:02.578537    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv" event={"ID":"593f875f-a4d9-47cc-859d-b07bac776964","Type":"ContainerStarted","Data":"5806d9c02e545fc348c7553e7dcafbfe628cbb9a0290bf9abc6c91a7461eea8b"}
Jul 14 14:00:02.578837 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:02.578621    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv"
Jul 14 14:00:02.599523 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:02.599480    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv" podStartSLOduration=1.251270928 podStartE2EDuration="2.599467425s" totalImagesPullingTime="1.348196497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:00:02.597223275 +0000 UTC m=+1967.923543204" watchObservedRunningTime="2026-07-14 14:00:02.599467425 +0000 UTC m=+1967.925787352"
Jul 14 14:00:08.590204 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:08.590175    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv"
Jul 14 14:00:22.783281 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:22.783247    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv"]
Jul 14 14:00:22.784137 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:22.784082    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv" podUID="593f875f-a4d9-47cc-859d-b07bac776964" containerName="maas-api" containerID="cri-o://5806d9c02e545fc348c7553e7dcafbfe628cbb9a0290bf9abc6c91a7461eea8b" gracePeriod=30
Jul 14 14:00:23.034328 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:23.034276    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv"
Jul 14 14:00:23.121569 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:23.121545    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/593f875f-a4d9-47cc-859d-b07bac776964-openshift-service-ca\") pod \"593f875f-a4d9-47cc-859d-b07bac776964\" (UID: \"593f875f-a4d9-47cc-859d-b07bac776964\") "
Jul 14 14:00:23.121704 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:23.121649    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-842l4\" (UniqueName: \"kubernetes.io/projected/593f875f-a4d9-47cc-859d-b07bac776964-kube-api-access-842l4\") pod \"593f875f-a4d9-47cc-859d-b07bac776964\" (UID: \"593f875f-a4d9-47cc-859d-b07bac776964\") "
Jul 14 14:00:23.121704 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:23.121668    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/593f875f-a4d9-47cc-859d-b07bac776964-maas-api-tls\") pod \"593f875f-a4d9-47cc-859d-b07bac776964\" (UID: \"593f875f-a4d9-47cc-859d-b07bac776964\") "
Jul 14 14:00:23.121867 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:23.121844    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/593f875f-a4d9-47cc-859d-b07bac776964-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "593f875f-a4d9-47cc-859d-b07bac776964" (UID: "593f875f-a4d9-47cc-859d-b07bac776964"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 14:00:23.123732 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:23.123710    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/593f875f-a4d9-47cc-859d-b07bac776964-kube-api-access-842l4" (OuterVolumeSpecName: "kube-api-access-842l4") pod "593f875f-a4d9-47cc-859d-b07bac776964" (UID: "593f875f-a4d9-47cc-859d-b07bac776964"). InnerVolumeSpecName "kube-api-access-842l4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 14:00:23.123816 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:23.123773    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/593f875f-a4d9-47cc-859d-b07bac776964-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "593f875f-a4d9-47cc-859d-b07bac776964" (UID: "593f875f-a4d9-47cc-859d-b07bac776964"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 14:00:23.222486 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:23.222460    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-842l4\" (UniqueName: \"kubernetes.io/projected/593f875f-a4d9-47cc-859d-b07bac776964-kube-api-access-842l4\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:00:23.222486 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:23.222482    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/593f875f-a4d9-47cc-859d-b07bac776964-maas-api-tls\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:00:23.222486 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:23.222491    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/593f875f-a4d9-47cc-859d-b07bac776964-openshift-service-ca\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:00:23.684235 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:23.684199    2569 generic.go:356] "Generic (PLEG): container finished" podID="593f875f-a4d9-47cc-859d-b07bac776964" containerID="5806d9c02e545fc348c7553e7dcafbfe628cbb9a0290bf9abc6c91a7461eea8b" exitCode=0
Jul 14 14:00:23.684425 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:23.684242    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv"
Jul 14 14:00:23.684425 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:23.684244    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv" event={"ID":"593f875f-a4d9-47cc-859d-b07bac776964","Type":"ContainerDied","Data":"5806d9c02e545fc348c7553e7dcafbfe628cbb9a0290bf9abc6c91a7461eea8b"}
Jul 14 14:00:23.684425 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:23.684358    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv" event={"ID":"593f875f-a4d9-47cc-859d-b07bac776964","Type":"ContainerDied","Data":"4953daf38a38decd6a3421c80dea8e06b42ed969a2c1dc5a313a874fc748474e"}
Jul 14 14:00:23.684425 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:23.684387    2569 scope.go:117] "RemoveContainer" containerID="5806d9c02e545fc348c7553e7dcafbfe628cbb9a0290bf9abc6c91a7461eea8b"
Jul 14 14:00:23.692925 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:23.692908    2569 scope.go:117] "RemoveContainer" containerID="5806d9c02e545fc348c7553e7dcafbfe628cbb9a0290bf9abc6c91a7461eea8b"
Jul 14 14:00:23.693282 ip-10-0-140-78 kubenswrapper[2569]: E0714 14:00:23.693263    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5806d9c02e545fc348c7553e7dcafbfe628cbb9a0290bf9abc6c91a7461eea8b\": container with ID starting with 5806d9c02e545fc348c7553e7dcafbfe628cbb9a0290bf9abc6c91a7461eea8b not found: ID does not exist" containerID="5806d9c02e545fc348c7553e7dcafbfe628cbb9a0290bf9abc6c91a7461eea8b"
Jul 14 14:00:23.693335 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:23.693290    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5806d9c02e545fc348c7553e7dcafbfe628cbb9a0290bf9abc6c91a7461eea8b"} err="failed to get container status \"5806d9c02e545fc348c7553e7dcafbfe628cbb9a0290bf9abc6c91a7461eea8b\": rpc error: code = NotFound desc = could not find container \"5806d9c02e545fc348c7553e7dcafbfe628cbb9a0290bf9abc6c91a7461eea8b\": container with ID starting with 5806d9c02e545fc348c7553e7dcafbfe628cbb9a0290bf9abc6c91a7461eea8b not found: ID does not exist"
Jul 14 14:00:23.703826 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:23.703806    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv"]
Jul 14 14:00:23.708298 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:23.708279    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50aeb779-57d8c7fdfd-49swv"]
Jul 14 14:00:25.192555 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:00:25.192525    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="593f875f-a4d9-47cc-859d-b07bac776964" path="/var/lib/kubelet/pods/593f875f-a4d9-47cc-859d-b07bac776964/volumes"
Jul 14 14:01:02.598658 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:02.598629    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5"]
Jul 14 14:01:02.599145 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:02.598936    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="593f875f-a4d9-47cc-859d-b07bac776964" containerName="maas-api"
Jul 14 14:01:02.599145 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:02.598946    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="593f875f-a4d9-47cc-859d-b07bac776964" containerName="maas-api"
Jul 14 14:01:02.599145 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:02.599038    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="593f875f-a4d9-47cc-859d-b07bac776964" containerName="maas-api"
Jul 14 14:01:02.601536 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:02.601520    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5"
Jul 14 14:01:02.603999 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:02.603978    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-4ca3631f-serving-cert\""
Jul 14 14:01:02.603999 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:02.604028    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wkxqn\""
Jul 14 14:01:02.613773 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:02.613704    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5"]
Jul 14 14:01:02.732360 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:02.732330    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt87v\" (UniqueName: \"kubernetes.io/projected/6c5c758c-61d1-4d8a-a676-540bfaa90c42-kube-api-access-gt87v\") pod \"maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5\" (UID: \"6c5c758c-61d1-4d8a-a676-540bfaa90c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5"
Jul 14 14:01:02.732527 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:02.732374    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c5c758c-61d1-4d8a-a676-540bfaa90c42-openshift-service-ca\") pod \"maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5\" (UID: \"6c5c758c-61d1-4d8a-a676-540bfaa90c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5"
Jul 14 14:01:02.732527 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:02.732396    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c5c758c-61d1-4d8a-a676-540bfaa90c42-maas-api-tls\") pod \"maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5\" (UID: \"6c5c758c-61d1-4d8a-a676-540bfaa90c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5"
Jul 14 14:01:02.833733 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:02.833698    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c5c758c-61d1-4d8a-a676-540bfaa90c42-openshift-service-ca\") pod \"maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5\" (UID: \"6c5c758c-61d1-4d8a-a676-540bfaa90c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5"
Jul 14 14:01:02.833885 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:02.833737    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c5c758c-61d1-4d8a-a676-540bfaa90c42-maas-api-tls\") pod \"maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5\" (UID: \"6c5c758c-61d1-4d8a-a676-540bfaa90c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5"
Jul 14 14:01:02.833956 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:02.833916    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt87v\" (UniqueName: \"kubernetes.io/projected/6c5c758c-61d1-4d8a-a676-540bfaa90c42-kube-api-access-gt87v\") pod \"maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5\" (UID: \"6c5c758c-61d1-4d8a-a676-540bfaa90c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5"
Jul 14 14:01:02.834380 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:02.834360    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c5c758c-61d1-4d8a-a676-540bfaa90c42-openshift-service-ca\") pod \"maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5\" (UID: \"6c5c758c-61d1-4d8a-a676-540bfaa90c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5"
Jul 14 14:01:02.836094 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:02.836075    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c5c758c-61d1-4d8a-a676-540bfaa90c42-maas-api-tls\") pod \"maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5\" (UID: \"6c5c758c-61d1-4d8a-a676-540bfaa90c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5"
Jul 14 14:01:02.845134 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:02.845113    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt87v\" (UniqueName: \"kubernetes.io/projected/6c5c758c-61d1-4d8a-a676-540bfaa90c42-kube-api-access-gt87v\") pod \"maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5\" (UID: \"6c5c758c-61d1-4d8a-a676-540bfaa90c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5"
Jul 14 14:01:02.911824 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:02.911768    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5"
Jul 14 14:01:03.041506 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:03.041483    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5"]
Jul 14 14:01:03.045149 ip-10-0-140-78 kubenswrapper[2569]: W0714 14:01:03.045100    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c5c758c_61d1_4d8a_a676_540bfaa90c42.slice/crio-b107fab8e2c7066f55bde0ccc976c1affeffb8e7e588a40653ab27d24417422e WatchSource:0}: Error finding container b107fab8e2c7066f55bde0ccc976c1affeffb8e7e588a40653ab27d24417422e: Status 404 returned error can't find the container with id b107fab8e2c7066f55bde0ccc976c1affeffb8e7e588a40653ab27d24417422e
Jul 14 14:01:03.877071 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:03.877034    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5" event={"ID":"6c5c758c-61d1-4d8a-a676-540bfaa90c42","Type":"ContainerStarted","Data":"b107fab8e2c7066f55bde0ccc976c1affeffb8e7e588a40653ab27d24417422e"}
Jul 14 14:01:04.883322 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:04.883287    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5" event={"ID":"6c5c758c-61d1-4d8a-a676-540bfaa90c42","Type":"ContainerStarted","Data":"12dab4322fd89696637e40a3e12ae094752be5570bb5beb3734e7ddb944a4346"}
Jul 14 14:01:04.883691 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:04.883353    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5"
Jul 14 14:01:04.904211 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:04.904163    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5" podStartSLOduration=1.609975522 podStartE2EDuration="2.904148664s" totalImagesPullingTime="1.294173142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:01:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:01:04.901839493 +0000 UTC m=+2030.228159433" watchObservedRunningTime="2026-07-14 14:01:04.904148664 +0000 UTC m=+2030.230468591"
Jul 14 14:01:10.892681 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:10.892651    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5"
Jul 14 14:01:19.814429 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:19.814393    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2"]
Jul 14 14:01:19.817543 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:19.817525    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2"
Jul 14 14:01:19.820163 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:19.820136    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-4ca3631f\"/\"openshift-service-ca.crt\""
Jul 14 14:01:19.821568 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:19.821547    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-4ca3631f\"/\"e2e-lifecycle-model-4ca3631f-kserve-self-signed-certs\""
Jul 14 14:01:19.821659 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:19.821567    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-4ca3631f\"/\"default-dockercfg-8qpjz\""
Jul 14 14:01:19.821659 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:19.821634    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-4ca3631f\"/\"kube-root-ca.crt\""
Jul 14 14:01:19.832196 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:19.832174    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2"]
Jul 14 14:01:19.967005 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:19.966976    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-home\") pod \"e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2\" (UID: \"f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a\") " pod="ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2"
Jul 14 14:01:19.967005 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:19.967004    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-tls-certs\") pod \"e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2\" (UID: \"f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a\") " pod="ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2"
Jul 14 14:01:19.967243 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:19.967105    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmmzx\" (UniqueName: \"kubernetes.io/projected/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-kube-api-access-cmmzx\") pod \"e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2\" (UID: \"f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a\") " pod="ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2"
Jul 14 14:01:19.967243 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:19.967144    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-dshm\") pod \"e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2\" (UID: \"f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a\") " pod="ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2"
Jul 14 14:01:19.967358 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:19.967261    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-model-cache\") pod \"e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2\" (UID: \"f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a\") " pod="ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2"
Jul 14 14:01:19.967358 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:19.967295    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-tmp-dir\") pod \"e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2\" (UID: \"f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a\") " pod="ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2"
Jul 14 14:01:20.068044 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:20.067960    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-model-cache\") pod \"e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2\" (UID: \"f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a\") " pod="ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2"
Jul 14 14:01:20.068044 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:20.067996    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-tmp-dir\") pod \"e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2\" (UID: \"f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a\") " pod="ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2"
Jul 14 14:01:20.068225 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:20.068063    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-home\") pod \"e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2\" (UID: \"f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a\") " pod="ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2"
Jul 14 14:01:20.068225 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:20.068086    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-tls-certs\") pod \"e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2\" (UID: \"f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a\") " pod="ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2"
Jul 14 14:01:20.068225 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:20.068126    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmmzx\" (UniqueName: \"kubernetes.io/projected/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-kube-api-access-cmmzx\") pod \"e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2\" (UID: \"f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a\") " pod="ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2"
Jul 14 14:01:20.068225 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:20.068158    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-dshm\") pod \"e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2\" (UID: \"f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a\") " pod="ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2"
Jul 14 14:01:20.068438 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:20.068330    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-model-cache\") pod \"e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2\" (UID: \"f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a\") " pod="ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2"
Jul 14 14:01:20.068438 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:20.068407    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-home\") pod \"e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2\" (UID: \"f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a\") " pod="ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2"
Jul 14 14:01:20.068538 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:20.068440    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-tmp-dir\") pod \"e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2\" (UID: \"f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a\") " pod="ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2"
Jul 14 14:01:20.070389 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:20.070370    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-dshm\") pod \"e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2\" (UID: \"f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a\") " pod="ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2"
Jul 14 14:01:20.070565 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:20.070549    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-tls-certs\") pod \"e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2\" (UID: \"f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a\") " pod="ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2"
Jul 14 14:01:20.076662 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:20.076641    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmmzx\" (UniqueName: \"kubernetes.io/projected/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-kube-api-access-cmmzx\") pod \"e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2\" (UID: \"f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a\") " pod="ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2"
Jul 14 14:01:20.127585 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:20.127566    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2"
Jul 14 14:01:20.258141 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:20.258116    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2"]
Jul 14 14:01:20.259481 ip-10-0-140-78 kubenswrapper[2569]: W0714 14:01:20.259452    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7ee12fb_78c7_4f42_9d72_c3abb5c1df1a.slice/crio-fba77d169315c2b0696af0c63631ea62d9c1daaf90a4c68248f1ee68199cf7d3 WatchSource:0}: Error finding container fba77d169315c2b0696af0c63631ea62d9c1daaf90a4c68248f1ee68199cf7d3: Status 404 returned error can't find the container with id fba77d169315c2b0696af0c63631ea62d9c1daaf90a4c68248f1ee68199cf7d3
Jul 14 14:01:20.961507 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:20.961472    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2" event={"ID":"f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a","Type":"ContainerStarted","Data":"e896a23cbdd6e105cbacc95a2c64ea6f57b5938ab5d51d804253b3ec45168422"}
Jul 14 14:01:20.961887 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:20.961515    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2" event={"ID":"f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a","Type":"ContainerStarted","Data":"fba77d169315c2b0696af0c63631ea62d9c1daaf90a4c68248f1ee68199cf7d3"}
Jul 14 14:01:20.994205 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:20.994156    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2" podStartSLOduration=1.994139424 podStartE2EDuration="1.994139424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:01:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:01:20.992130901 +0000 UTC m=+2046.318450831" watchObservedRunningTime="2026-07-14 14:01:20.994139424 +0000 UTC m=+2046.320459352"
Jul 14 14:01:30.127922 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:30.127887    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2"
Jul 14 14:01:30.128370 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:30.127955    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2"
Jul 14 14:01:30.140270 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:30.140247    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2"
Jul 14 14:01:31.026838 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:31.026813    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2"
Jul 14 14:01:55.446257 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:55.446214    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5"]
Jul 14 14:01:55.447207 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:55.446474    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5" podUID="6c5c758c-61d1-4d8a-a676-540bfaa90c42" containerName="maas-api" containerID="cri-o://12dab4322fd89696637e40a3e12ae094752be5570bb5beb3734e7ddb944a4346" gracePeriod=30
Jul 14 14:01:55.695059 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:55.695036    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5"
Jul 14 14:01:55.745039 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:55.745003    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c5c758c-61d1-4d8a-a676-540bfaa90c42-maas-api-tls\") pod \"6c5c758c-61d1-4d8a-a676-540bfaa90c42\" (UID: \"6c5c758c-61d1-4d8a-a676-540bfaa90c42\") "
Jul 14 14:01:55.745143 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:55.745089    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gt87v\" (UniqueName: \"kubernetes.io/projected/6c5c758c-61d1-4d8a-a676-540bfaa90c42-kube-api-access-gt87v\") pod \"6c5c758c-61d1-4d8a-a676-540bfaa90c42\" (UID: \"6c5c758c-61d1-4d8a-a676-540bfaa90c42\") "
Jul 14 14:01:55.745143 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:55.745118    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c5c758c-61d1-4d8a-a676-540bfaa90c42-openshift-service-ca\") pod \"6c5c758c-61d1-4d8a-a676-540bfaa90c42\" (UID: \"6c5c758c-61d1-4d8a-a676-540bfaa90c42\") "
Jul 14 14:01:55.745507 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:55.745483    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c5c758c-61d1-4d8a-a676-540bfaa90c42-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6c5c758c-61d1-4d8a-a676-540bfaa90c42" (UID: "6c5c758c-61d1-4d8a-a676-540bfaa90c42"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 14:01:55.746997 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:55.746973    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c5c758c-61d1-4d8a-a676-540bfaa90c42-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6c5c758c-61d1-4d8a-a676-540bfaa90c42" (UID: "6c5c758c-61d1-4d8a-a676-540bfaa90c42"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 14:01:55.747147 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:55.747126    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c5c758c-61d1-4d8a-a676-540bfaa90c42-kube-api-access-gt87v" (OuterVolumeSpecName: "kube-api-access-gt87v") pod "6c5c758c-61d1-4d8a-a676-540bfaa90c42" (UID: "6c5c758c-61d1-4d8a-a676-540bfaa90c42"). InnerVolumeSpecName "kube-api-access-gt87v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 14:01:55.846075 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:55.846053    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c5c758c-61d1-4d8a-a676-540bfaa90c42-maas-api-tls\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:01:55.846075 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:55.846074    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt87v\" (UniqueName: \"kubernetes.io/projected/6c5c758c-61d1-4d8a-a676-540bfaa90c42-kube-api-access-gt87v\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:01:55.846222 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:55.846085    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c5c758c-61d1-4d8a-a676-540bfaa90c42-openshift-service-ca\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:01:56.145940 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:56.145848    2569 generic.go:356] "Generic (PLEG): container finished" podID="6c5c758c-61d1-4d8a-a676-540bfaa90c42" containerID="12dab4322fd89696637e40a3e12ae094752be5570bb5beb3734e7ddb944a4346" exitCode=0
Jul 14 14:01:56.145940 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:56.145888    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5"
Jul 14 14:01:56.145940 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:56.145907    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5" event={"ID":"6c5c758c-61d1-4d8a-a676-540bfaa90c42","Type":"ContainerDied","Data":"12dab4322fd89696637e40a3e12ae094752be5570bb5beb3734e7ddb944a4346"}
Jul 14 14:01:56.146201 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:56.145955    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5" event={"ID":"6c5c758c-61d1-4d8a-a676-540bfaa90c42","Type":"ContainerDied","Data":"b107fab8e2c7066f55bde0ccc976c1affeffb8e7e588a40653ab27d24417422e"}
Jul 14 14:01:56.146201 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:56.145980    2569 scope.go:117] "RemoveContainer" containerID="12dab4322fd89696637e40a3e12ae094752be5570bb5beb3734e7ddb944a4346"
Jul 14 14:01:56.154989 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:56.154975    2569 scope.go:117] "RemoveContainer" containerID="12dab4322fd89696637e40a3e12ae094752be5570bb5beb3734e7ddb944a4346"
Jul 14 14:01:56.155286 ip-10-0-140-78 kubenswrapper[2569]: E0714 14:01:56.155266    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12dab4322fd89696637e40a3e12ae094752be5570bb5beb3734e7ddb944a4346\": container with ID starting with 12dab4322fd89696637e40a3e12ae094752be5570bb5beb3734e7ddb944a4346 not found: ID does not exist" containerID="12dab4322fd89696637e40a3e12ae094752be5570bb5beb3734e7ddb944a4346"
Jul 14 14:01:56.155340 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:56.155295    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12dab4322fd89696637e40a3e12ae094752be5570bb5beb3734e7ddb944a4346"} err="failed to get container status \"12dab4322fd89696637e40a3e12ae094752be5570bb5beb3734e7ddb944a4346\": rpc error: code = NotFound desc = could not find container \"12dab4322fd89696637e40a3e12ae094752be5570bb5beb3734e7ddb944a4346\": container with ID starting with 12dab4322fd89696637e40a3e12ae094752be5570bb5beb3734e7ddb944a4346 not found: ID does not exist"
Jul 14 14:01:56.169544 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:56.169522    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5"]
Jul 14 14:01:56.176523 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:56.176503    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4ca3631f-7b8f54f76c-9m8h5"]
Jul 14 14:01:57.194148 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:01:57.194114    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c5c758c-61d1-4d8a-a676-540bfaa90c42" path="/var/lib/kubelet/pods/6c5c758c-61d1-4d8a-a676-540bfaa90c42/volumes"
Jul 14 14:02:05.079124 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:02:05.079092    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2"]
Jul 14 14:02:15.237854 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:02:15.237828    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vq8tk_e33a489f-7d68-4e6c-9ffe-e7252049e15d/ovn-acl-logging/0.log"
Jul 14 14:02:15.248235 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:02:15.248213    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vq8tk_e33a489f-7d68-4e6c-9ffe-e7252049e15d/ovn-acl-logging/0.log"
Jul 14 14:02:20.100555 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:02:20.100494    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2" podUID="f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a" containerName="main" containerID="cri-o://e896a23cbdd6e105cbacc95a2c64ea6f57b5938ab5d51d804253b3ec45168422" gracePeriod=45
Jul 14 14:02:20.263842 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:02:20.263815    2569 generic.go:356] "Generic (PLEG): container finished" podID="f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a" containerID="e896a23cbdd6e105cbacc95a2c64ea6f57b5938ab5d51d804253b3ec45168422" exitCode=0
Jul 14 14:02:20.263968 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:02:20.263895    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2" event={"ID":"f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a","Type":"ContainerDied","Data":"e896a23cbdd6e105cbacc95a2c64ea6f57b5938ab5d51d804253b3ec45168422"}
Jul 14 14:02:20.339362 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:02:20.339341    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2"
Jul 14 14:02:20.442847 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:02:20.442769    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-model-cache\") pod \"f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a\" (UID: \"f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a\") "
Jul 14 14:02:20.442847 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:02:20.442832    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-home\") pod \"f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a\" (UID: \"f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a\") "
Jul 14 14:02:20.443091 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:02:20.442858    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmmzx\" (UniqueName: \"kubernetes.io/projected/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-kube-api-access-cmmzx\") pod \"f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a\" (UID: \"f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a\") "
Jul 14 14:02:20.443091 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:02:20.442877    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-tmp-dir\") pod \"f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a\" (UID: \"f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a\") "
Jul 14 14:02:20.443091 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:02:20.442918    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-tls-certs\") pod \"f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a\" (UID: \"f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a\") "
Jul 14 14:02:20.443091 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:02:20.442962    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-dshm\") pod \"f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a\" (UID: \"f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a\") "
Jul 14 14:02:20.443091 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:02:20.443059    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-model-cache" (OuterVolumeSpecName: "model-cache") pod "f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a" (UID: "f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:02:20.443299 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:02:20.443133    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-home" (OuterVolumeSpecName: "home") pod "f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a" (UID: "f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:02:20.443299 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:02:20.443181    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a" (UID: "f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:02:20.443372 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:02:20.443302    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-home\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:02:20.443372 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:02:20.443316    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-tmp-dir\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:02:20.443372 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:02:20.443326    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-model-cache\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:02:20.445136 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:02:20.445104    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-dshm" (OuterVolumeSpecName: "dshm") pod "f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a" (UID: "f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:02:20.445239 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:02:20.445168    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a" (UID: "f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 14:02:20.445239 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:02:20.445192    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-kube-api-access-cmmzx" (OuterVolumeSpecName: "kube-api-access-cmmzx") pod "f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a" (UID: "f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a"). InnerVolumeSpecName "kube-api-access-cmmzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 14:02:20.544113 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:02:20.544084    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-tls-certs\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:02:20.544113 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:02:20.544106    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-dshm\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:02:20.544113 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:02:20.544116    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmmzx\" (UniqueName: \"kubernetes.io/projected/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a-kube-api-access-cmmzx\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:02:21.270080 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:02:21.270054    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2"
Jul 14 14:02:21.270080 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:02:21.270069    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2" event={"ID":"f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a","Type":"ContainerDied","Data":"fba77d169315c2b0696af0c63631ea62d9c1daaf90a4c68248f1ee68199cf7d3"}
Jul 14 14:02:21.270488 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:02:21.270119    2569 scope.go:117] "RemoveContainer" containerID="e896a23cbdd6e105cbacc95a2c64ea6f57b5938ab5d51d804253b3ec45168422"
Jul 14 14:02:21.289344 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:02:21.289323    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2"]
Jul 14 14:02:21.293914 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:02:21.293893    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-4ca3631f/e2e-lifecycle-model-4ca3631f-kserve-6979b6484c-lc7c2"]
Jul 14 14:02:23.193478 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:02:23.193443    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a" path="/var/lib/kubelet/pods/f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a/volumes"
Jul 14 14:04:02.277802 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:02.277769    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf"]
Jul 14 14:04:02.278304 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:02.278239    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a" containerName="main"
Jul 14 14:04:02.278304 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:02.278252    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a" containerName="main"
Jul 14 14:04:02.278304 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:02.278270    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c5c758c-61d1-4d8a-a676-540bfaa90c42" containerName="maas-api"
Jul 14 14:04:02.278304 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:02.278276    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c5c758c-61d1-4d8a-a676-540bfaa90c42" containerName="maas-api"
Jul 14 14:04:02.278439 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:02.278324    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7ee12fb-78c7-4f42-9d72-c3abb5c1df1a" containerName="main"
Jul 14 14:04:02.278439 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:02.278333    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c5c758c-61d1-4d8a-a676-540bfaa90c42" containerName="maas-api"
Jul 14 14:04:02.281473 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:02.281457    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf"
Jul 14 14:04:02.284186 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:02.284167    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wkxqn\""
Jul 14 14:04:02.284291 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:02.284234    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-370ee8-serving-cert\""
Jul 14 14:04:02.291232 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:02.291208    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf"]
Jul 14 14:04:02.303770 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:02.303725    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c685ffa1-b678-45ad-9f8e-95caeb1de05d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf\" (UID: \"c685ffa1-b678-45ad-9f8e-95caeb1de05d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf"
Jul 14 14:04:02.303877 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:02.303795    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62w8q\" (UniqueName: \"kubernetes.io/projected/c685ffa1-b678-45ad-9f8e-95caeb1de05d-kube-api-access-62w8q\") pod \"maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf\" (UID: \"c685ffa1-b678-45ad-9f8e-95caeb1de05d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf"
Jul 14 14:04:02.303877 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:02.303834    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c685ffa1-b678-45ad-9f8e-95caeb1de05d-maas-api-tls\") pod \"maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf\" (UID: \"c685ffa1-b678-45ad-9f8e-95caeb1de05d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf"
Jul 14 14:04:02.404953 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:02.404927    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c685ffa1-b678-45ad-9f8e-95caeb1de05d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf\" (UID: \"c685ffa1-b678-45ad-9f8e-95caeb1de05d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf"
Jul 14 14:04:02.405069 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:02.404966    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62w8q\" (UniqueName: \"kubernetes.io/projected/c685ffa1-b678-45ad-9f8e-95caeb1de05d-kube-api-access-62w8q\") pod \"maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf\" (UID: \"c685ffa1-b678-45ad-9f8e-95caeb1de05d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf"
Jul 14 14:04:02.405069 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:02.404989    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c685ffa1-b678-45ad-9f8e-95caeb1de05d-maas-api-tls\") pod \"maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf\" (UID: \"c685ffa1-b678-45ad-9f8e-95caeb1de05d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf"
Jul 14 14:04:02.405203 ip-10-0-140-78 kubenswrapper[2569]: E0714 14:04:02.405095    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-370ee8-serving-cert: secret "maas-api-e2e-inf-a-370ee8-serving-cert" not found
Jul 14 14:04:02.405203 ip-10-0-140-78 kubenswrapper[2569]: E0714 14:04:02.405160    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c685ffa1-b678-45ad-9f8e-95caeb1de05d-maas-api-tls podName:c685ffa1-b678-45ad-9f8e-95caeb1de05d nodeName:}" failed. No retries permitted until 2026-07-14 14:04:02.90514411 +0000 UTC m=+2208.231464017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c685ffa1-b678-45ad-9f8e-95caeb1de05d-maas-api-tls") pod "maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf" (UID: "c685ffa1-b678-45ad-9f8e-95caeb1de05d") : secret "maas-api-e2e-inf-a-370ee8-serving-cert" not found
Jul 14 14:04:02.405626 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:02.405603    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c685ffa1-b678-45ad-9f8e-95caeb1de05d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf\" (UID: \"c685ffa1-b678-45ad-9f8e-95caeb1de05d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf"
Jul 14 14:04:02.417325 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:02.417301    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62w8q\" (UniqueName: \"kubernetes.io/projected/c685ffa1-b678-45ad-9f8e-95caeb1de05d-kube-api-access-62w8q\") pod \"maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf\" (UID: \"c685ffa1-b678-45ad-9f8e-95caeb1de05d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf"
Jul 14 14:04:02.908500 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:02.908472    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c685ffa1-b678-45ad-9f8e-95caeb1de05d-maas-api-tls\") pod \"maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf\" (UID: \"c685ffa1-b678-45ad-9f8e-95caeb1de05d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf"
Jul 14 14:04:02.910884 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:02.910858    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c685ffa1-b678-45ad-9f8e-95caeb1de05d-maas-api-tls\") pod \"maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf\" (UID: \"c685ffa1-b678-45ad-9f8e-95caeb1de05d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf"
Jul 14 14:04:03.192639 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:03.192551    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf"
Jul 14 14:04:03.323665 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:03.323643    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf"]
Jul 14 14:04:03.325553 ip-10-0-140-78 kubenswrapper[2569]: W0714 14:04:03.325528    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc685ffa1_b678_45ad_9f8e_95caeb1de05d.slice/crio-ed78a6563153e0690b68ad4b00c62dd3d84746110d7777f936959e56d313d03d WatchSource:0}: Error finding container ed78a6563153e0690b68ad4b00c62dd3d84746110d7777f936959e56d313d03d: Status 404 returned error can't find the container with id ed78a6563153e0690b68ad4b00c62dd3d84746110d7777f936959e56d313d03d
Jul 14 14:04:03.771521 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:03.771490    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf" event={"ID":"c685ffa1-b678-45ad-9f8e-95caeb1de05d","Type":"ContainerStarted","Data":"ed78a6563153e0690b68ad4b00c62dd3d84746110d7777f936959e56d313d03d"}
Jul 14 14:04:04.778799 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:04.778754    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf" event={"ID":"c685ffa1-b678-45ad-9f8e-95caeb1de05d","Type":"ContainerStarted","Data":"c1554372b341983b9b24e29fe557fbb82d45e9c13613d5cd553f5fab701db133"}
Jul 14 14:04:04.779184 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:04.778817    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf"
Jul 14 14:04:04.798288 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:04.798238    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf" podStartSLOduration=2.343757118 podStartE2EDuration="2.798222495s" totalImagesPullingTime="454.465377ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:04:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:04:04.797173198 +0000 UTC m=+2210.123493130" watchObservedRunningTime="2026-07-14 14:04:04.798222495 +0000 UTC m=+2210.124542422"
Jul 14 14:04:10.219815 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:10.219786    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f1dea9-b9d479885-782z5"]
Jul 14 14:04:10.223346 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:10.223325    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1dea9-b9d479885-782z5"
Jul 14 14:04:10.227935 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:10.227915    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-f1dea9-serving-cert\""
Jul 14 14:04:10.239426 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:10.239406    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f1dea9-b9d479885-782z5"]
Jul 14 14:04:10.265227 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:10.265205    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ff4bf3-5f48-4625-a9ea-e4ed733f2270-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f1dea9-b9d479885-782z5\" (UID: \"68ff4bf3-5f48-4625-a9ea-e4ed733f2270\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1dea9-b9d479885-782z5"
Jul 14 14:04:10.265315 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:10.265247    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68ff4bf3-5f48-4625-a9ea-e4ed733f2270-maas-api-tls\") pod \"maas-api-e2e-inf-b-f1dea9-b9d479885-782z5\" (UID: \"68ff4bf3-5f48-4625-a9ea-e4ed733f2270\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1dea9-b9d479885-782z5"
Jul 14 14:04:10.265360 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:10.265350    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fthjn\" (UniqueName: \"kubernetes.io/projected/68ff4bf3-5f48-4625-a9ea-e4ed733f2270-kube-api-access-fthjn\") pod \"maas-api-e2e-inf-b-f1dea9-b9d479885-782z5\" (UID: \"68ff4bf3-5f48-4625-a9ea-e4ed733f2270\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1dea9-b9d479885-782z5"
Jul 14 14:04:10.366388 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:10.366364    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fthjn\" (UniqueName: \"kubernetes.io/projected/68ff4bf3-5f48-4625-a9ea-e4ed733f2270-kube-api-access-fthjn\") pod \"maas-api-e2e-inf-b-f1dea9-b9d479885-782z5\" (UID: \"68ff4bf3-5f48-4625-a9ea-e4ed733f2270\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1dea9-b9d479885-782z5"
Jul 14 14:04:10.366485 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:10.366404    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ff4bf3-5f48-4625-a9ea-e4ed733f2270-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f1dea9-b9d479885-782z5\" (UID: \"68ff4bf3-5f48-4625-a9ea-e4ed733f2270\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1dea9-b9d479885-782z5"
Jul 14 14:04:10.366485 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:10.366459    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68ff4bf3-5f48-4625-a9ea-e4ed733f2270-maas-api-tls\") pod \"maas-api-e2e-inf-b-f1dea9-b9d479885-782z5\" (UID: \"68ff4bf3-5f48-4625-a9ea-e4ed733f2270\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1dea9-b9d479885-782z5"
Jul 14 14:04:10.366581 ip-10-0-140-78 kubenswrapper[2569]: E0714 14:04:10.366566    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-f1dea9-serving-cert: secret "maas-api-e2e-inf-b-f1dea9-serving-cert" not found
Jul 14 14:04:10.366637 ip-10-0-140-78 kubenswrapper[2569]: E0714 14:04:10.366627    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68ff4bf3-5f48-4625-a9ea-e4ed733f2270-maas-api-tls podName:68ff4bf3-5f48-4625-a9ea-e4ed733f2270 nodeName:}" failed. No retries permitted until 2026-07-14 14:04:10.866607728 +0000 UTC m=+2216.192927638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/68ff4bf3-5f48-4625-a9ea-e4ed733f2270-maas-api-tls") pod "maas-api-e2e-inf-b-f1dea9-b9d479885-782z5" (UID: "68ff4bf3-5f48-4625-a9ea-e4ed733f2270") : secret "maas-api-e2e-inf-b-f1dea9-serving-cert" not found
Jul 14 14:04:10.366961 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:10.366944    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ff4bf3-5f48-4625-a9ea-e4ed733f2270-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f1dea9-b9d479885-782z5\" (UID: \"68ff4bf3-5f48-4625-a9ea-e4ed733f2270\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1dea9-b9d479885-782z5"
Jul 14 14:04:10.377870 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:10.377849    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fthjn\" (UniqueName: \"kubernetes.io/projected/68ff4bf3-5f48-4625-a9ea-e4ed733f2270-kube-api-access-fthjn\") pod \"maas-api-e2e-inf-b-f1dea9-b9d479885-782z5\" (UID: \"68ff4bf3-5f48-4625-a9ea-e4ed733f2270\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1dea9-b9d479885-782z5"
Jul 14 14:04:10.788807 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:10.788781    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf"
Jul 14 14:04:10.869993 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:10.869964    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68ff4bf3-5f48-4625-a9ea-e4ed733f2270-maas-api-tls\") pod \"maas-api-e2e-inf-b-f1dea9-b9d479885-782z5\" (UID: \"68ff4bf3-5f48-4625-a9ea-e4ed733f2270\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1dea9-b9d479885-782z5"
Jul 14 14:04:10.872354 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:10.872326    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68ff4bf3-5f48-4625-a9ea-e4ed733f2270-maas-api-tls\") pod \"maas-api-e2e-inf-b-f1dea9-b9d479885-782z5\" (UID: \"68ff4bf3-5f48-4625-a9ea-e4ed733f2270\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1dea9-b9d479885-782z5"
Jul 14 14:04:11.134442 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:11.134354    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1dea9-b9d479885-782z5"
Jul 14 14:04:11.263515 ip-10-0-140-78 kubenswrapper[2569]: W0714 14:04:11.263487    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ff4bf3_5f48_4625_a9ea_e4ed733f2270.slice/crio-8362d82e62aeb4d23554a7dae54c219c1f5d6d126f3bf2a3d760c6e5b3c1ba9c WatchSource:0}: Error finding container 8362d82e62aeb4d23554a7dae54c219c1f5d6d126f3bf2a3d760c6e5b3c1ba9c: Status 404 returned error can't find the container with id 8362d82e62aeb4d23554a7dae54c219c1f5d6d126f3bf2a3d760c6e5b3c1ba9c
Jul 14 14:04:11.263916 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:11.263891    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f1dea9-b9d479885-782z5"]
Jul 14 14:04:11.813206 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:11.813178    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1dea9-b9d479885-782z5" event={"ID":"68ff4bf3-5f48-4625-a9ea-e4ed733f2270","Type":"ContainerStarted","Data":"3e61136911ce6f461ace7929f460eee929714b63cbe0237c5b8d4b029ff890dc"}
Jul 14 14:04:11.813332 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:11.813230    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1dea9-b9d479885-782z5" event={"ID":"68ff4bf3-5f48-4625-a9ea-e4ed733f2270","Type":"ContainerStarted","Data":"8362d82e62aeb4d23554a7dae54c219c1f5d6d126f3bf2a3d760c6e5b3c1ba9c"}
Jul 14 14:04:11.813332 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:11.813256    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1dea9-b9d479885-782z5"
Jul 14 14:04:11.837524 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:11.837473    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1dea9-b9d479885-782z5" podStartSLOduration=1.482158836 podStartE2EDuration="1.837458018s" totalImagesPullingTime="355.299182ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:04:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:04:11.835173262 +0000 UTC m=+2217.161493190" watchObservedRunningTime="2026-07-14 14:04:11.837458018 +0000 UTC m=+2217.163777945"
Jul 14 14:04:17.824731 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:17.824704    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1dea9-b9d479885-782z5"
Jul 14 14:04:37.481196 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:37.481125    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z"]
Jul 14 14:04:37.484625 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:37.484609    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z"
Jul 14 14:04:37.488915 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:37.488888    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-370ee8\"/\"openshift-service-ca.crt\""
Jul 14 14:04:37.492533 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:37.492510    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-370ee8\"/\"default-dockercfg-mz8mx\""
Jul 14 14:04:37.492786 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:37.492578    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-370ee8\"/\"kube-root-ca.crt\""
Jul 14 14:04:37.492914 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:37.492892    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-370ee8\"/\"test-model-370ee8-kserve-self-signed-certs\""
Jul 14 14:04:37.495076 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:37.495056    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z"]
Jul 14 14:04:37.571906 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:37.571884    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c85b3c51-a26d-404d-a4aa-3f75f3832f75-dshm\") pod \"test-model-370ee8-kserve-674b56cc69-n689z\" (UID: \"c85b3c51-a26d-404d-a4aa-3f75f3832f75\") " pod="ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z"
Jul 14 14:04:37.572061 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:37.571909    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c85b3c51-a26d-404d-a4aa-3f75f3832f75-tmp-dir\") pod \"test-model-370ee8-kserve-674b56cc69-n689z\" (UID: \"c85b3c51-a26d-404d-a4aa-3f75f3832f75\") " pod="ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z"
Jul 14 14:04:37.572061 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:37.571934    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c85b3c51-a26d-404d-a4aa-3f75f3832f75-model-cache\") pod \"test-model-370ee8-kserve-674b56cc69-n689z\" (UID: \"c85b3c51-a26d-404d-a4aa-3f75f3832f75\") " pod="ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z"
Jul 14 14:04:37.572199 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:37.572066    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c85b3c51-a26d-404d-a4aa-3f75f3832f75-home\") pod \"test-model-370ee8-kserve-674b56cc69-n689z\" (UID: \"c85b3c51-a26d-404d-a4aa-3f75f3832f75\") " pod="ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z"
Jul 14 14:04:37.572199 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:37.572120    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b4jb\" (UniqueName: \"kubernetes.io/projected/c85b3c51-a26d-404d-a4aa-3f75f3832f75-kube-api-access-4b4jb\") pod \"test-model-370ee8-kserve-674b56cc69-n689z\" (UID: \"c85b3c51-a26d-404d-a4aa-3f75f3832f75\") " pod="ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z"
Jul 14 14:04:37.572199 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:37.572151    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c85b3c51-a26d-404d-a4aa-3f75f3832f75-tls-certs\") pod \"test-model-370ee8-kserve-674b56cc69-n689z\" (UID: \"c85b3c51-a26d-404d-a4aa-3f75f3832f75\") " pod="ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z"
Jul 14 14:04:37.672595 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:37.672565    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c85b3c51-a26d-404d-a4aa-3f75f3832f75-tls-certs\") pod \"test-model-370ee8-kserve-674b56cc69-n689z\" (UID: \"c85b3c51-a26d-404d-a4aa-3f75f3832f75\") " pod="ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z"
Jul 14 14:04:37.672721 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:37.672618    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c85b3c51-a26d-404d-a4aa-3f75f3832f75-dshm\") pod \"test-model-370ee8-kserve-674b56cc69-n689z\" (UID: \"c85b3c51-a26d-404d-a4aa-3f75f3832f75\") " pod="ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z"
Jul 14 14:04:37.672721 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:37.672636    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c85b3c51-a26d-404d-a4aa-3f75f3832f75-tmp-dir\") pod \"test-model-370ee8-kserve-674b56cc69-n689z\" (UID: \"c85b3c51-a26d-404d-a4aa-3f75f3832f75\") " pod="ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z"
Jul 14 14:04:37.672721 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:37.672657    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c85b3c51-a26d-404d-a4aa-3f75f3832f75-model-cache\") pod \"test-model-370ee8-kserve-674b56cc69-n689z\" (UID: \"c85b3c51-a26d-404d-a4aa-3f75f3832f75\") " pod="ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z"
Jul 14 14:04:37.672873 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:37.672764    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c85b3c51-a26d-404d-a4aa-3f75f3832f75-home\") pod \"test-model-370ee8-kserve-674b56cc69-n689z\" (UID: \"c85b3c51-a26d-404d-a4aa-3f75f3832f75\") " pod="ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z"
Jul 14 14:04:37.672873 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:37.672814    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b4jb\" (UniqueName: \"kubernetes.io/projected/c85b3c51-a26d-404d-a4aa-3f75f3832f75-kube-api-access-4b4jb\") pod \"test-model-370ee8-kserve-674b56cc69-n689z\" (UID: \"c85b3c51-a26d-404d-a4aa-3f75f3832f75\") " pod="ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z"
Jul 14 14:04:37.673069 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:37.673044    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c85b3c51-a26d-404d-a4aa-3f75f3832f75-model-cache\") pod \"test-model-370ee8-kserve-674b56cc69-n689z\" (UID: \"c85b3c51-a26d-404d-a4aa-3f75f3832f75\") " pod="ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z"
Jul 14 14:04:37.673173 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:37.673082    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c85b3c51-a26d-404d-a4aa-3f75f3832f75-tmp-dir\") pod \"test-model-370ee8-kserve-674b56cc69-n689z\" (UID: \"c85b3c51-a26d-404d-a4aa-3f75f3832f75\") " pod="ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z"
Jul 14 14:04:37.673222 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:37.673175    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c85b3c51-a26d-404d-a4aa-3f75f3832f75-home\") pod \"test-model-370ee8-kserve-674b56cc69-n689z\" (UID: \"c85b3c51-a26d-404d-a4aa-3f75f3832f75\") " pod="ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z"
Jul 14 14:04:37.674932 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:37.674904    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c85b3c51-a26d-404d-a4aa-3f75f3832f75-dshm\") pod \"test-model-370ee8-kserve-674b56cc69-n689z\" (UID: \"c85b3c51-a26d-404d-a4aa-3f75f3832f75\") " pod="ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z"
Jul 14 14:04:37.675107 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:37.675089    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c85b3c51-a26d-404d-a4aa-3f75f3832f75-tls-certs\") pod \"test-model-370ee8-kserve-674b56cc69-n689z\" (UID: \"c85b3c51-a26d-404d-a4aa-3f75f3832f75\") " pod="ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z"
Jul 14 14:04:37.683501 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:37.683480    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b4jb\" (UniqueName: \"kubernetes.io/projected/c85b3c51-a26d-404d-a4aa-3f75f3832f75-kube-api-access-4b4jb\") pod \"test-model-370ee8-kserve-674b56cc69-n689z\" (UID: \"c85b3c51-a26d-404d-a4aa-3f75f3832f75\") " pod="ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z"
Jul 14 14:04:37.795168 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:37.795115    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z"
Jul 14 14:04:37.924418 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:37.924394    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z"]
Jul 14 14:04:37.925886 ip-10-0-140-78 kubenswrapper[2569]: W0714 14:04:37.925858    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc85b3c51_a26d_404d_a4aa_3f75f3832f75.slice/crio-1db0013e432ce158469ef9be5142853bdfb45ce5e7e4d482671da4e95db5be22 WatchSource:0}: Error finding container 1db0013e432ce158469ef9be5142853bdfb45ce5e7e4d482671da4e95db5be22: Status 404 returned error can't find the container with id 1db0013e432ce158469ef9be5142853bdfb45ce5e7e4d482671da4e95db5be22
Jul 14 14:04:37.948437 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:37.948412    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z" event={"ID":"c85b3c51-a26d-404d-a4aa-3f75f3832f75","Type":"ContainerStarted","Data":"1db0013e432ce158469ef9be5142853bdfb45ce5e7e4d482671da4e95db5be22"}
Jul 14 14:04:38.955136 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:38.955101    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z" event={"ID":"c85b3c51-a26d-404d-a4aa-3f75f3832f75","Type":"ContainerStarted","Data":"eea973b145793d7afe9867d75f3f45c991cd653d9708d35400d4dfe140ddab25"}
Jul 14 14:04:38.973685 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:38.973638    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z" podStartSLOduration=1.973622848 podStartE2EDuration="1.973622848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:04:38.972459966 +0000 UTC m=+2244.298779895" watchObservedRunningTime="2026-07-14 14:04:38.973622848 +0000 UTC m=+2244.299942776"
Jul 14 14:04:47.795704 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:47.795665    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z"
Jul 14 14:04:47.796170 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:47.795804    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z"
Jul 14 14:04:47.808136 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:47.808109    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z"
Jul 14 14:04:48.014703 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:04:48.014677    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z"
Jul 14 14:05:29.487781 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:05:29.487751    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp"]
Jul 14 14:05:29.491107 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:05:29.491088    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp"
Jul 14 14:05:29.495436 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:05:29.495416    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f1dea9\"/\"kube-root-ca.crt\""
Jul 14 14:05:29.495571 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:05:29.495435    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f1dea9\"/\"default-dockercfg-9cx5p\""
Jul 14 14:05:29.495571 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:05:29.495449    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f1dea9\"/\"openshift-service-ca.crt\""
Jul 14 14:05:29.495571 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:05:29.495457    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f1dea9\"/\"test-model-f1dea9-kserve-self-signed-certs\""
Jul 14 14:05:29.502592 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:05:29.502572    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp"]
Jul 14 14:05:29.559309 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:05:29.559284    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1de62093-468d-44a1-9948-0bec84811146-model-cache\") pod \"test-model-f1dea9-kserve-68db9cf48f-2v9kp\" (UID: \"1de62093-468d-44a1-9948-0bec84811146\") " pod="ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp"
Jul 14 14:05:29.559414 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:05:29.559348    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1de62093-468d-44a1-9948-0bec84811146-dshm\") pod \"test-model-f1dea9-kserve-68db9cf48f-2v9kp\" (UID: \"1de62093-468d-44a1-9948-0bec84811146\") " pod="ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp"
Jul 14 14:05:29.559414 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:05:29.559381    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1de62093-468d-44a1-9948-0bec84811146-tls-certs\") pod \"test-model-f1dea9-kserve-68db9cf48f-2v9kp\" (UID: \"1de62093-468d-44a1-9948-0bec84811146\") " pod="ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp"
Jul 14 14:05:29.559414 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:05:29.559409    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq8x2\" (UniqueName: \"kubernetes.io/projected/1de62093-468d-44a1-9948-0bec84811146-kube-api-access-gq8x2\") pod \"test-model-f1dea9-kserve-68db9cf48f-2v9kp\" (UID: \"1de62093-468d-44a1-9948-0bec84811146\") " pod="ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp"
Jul 14 14:05:29.559567 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:05:29.559440    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1de62093-468d-44a1-9948-0bec84811146-home\") pod \"test-model-f1dea9-kserve-68db9cf48f-2v9kp\" (UID: \"1de62093-468d-44a1-9948-0bec84811146\") " pod="ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp"
Jul 14 14:05:29.559567 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:05:29.559468    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1de62093-468d-44a1-9948-0bec84811146-tmp-dir\") pod \"test-model-f1dea9-kserve-68db9cf48f-2v9kp\" (UID: \"1de62093-468d-44a1-9948-0bec84811146\") " pod="ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp"
Jul 14 14:05:29.660625 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:05:29.660595    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1de62093-468d-44a1-9948-0bec84811146-dshm\") pod \"test-model-f1dea9-kserve-68db9cf48f-2v9kp\" (UID: \"1de62093-468d-44a1-9948-0bec84811146\") " pod="ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp"
Jul 14 14:05:29.660625 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:05:29.660631    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1de62093-468d-44a1-9948-0bec84811146-tls-certs\") pod \"test-model-f1dea9-kserve-68db9cf48f-2v9kp\" (UID: \"1de62093-468d-44a1-9948-0bec84811146\") " pod="ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp"
Jul 14 14:05:29.660834 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:05:29.660650    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq8x2\" (UniqueName: \"kubernetes.io/projected/1de62093-468d-44a1-9948-0bec84811146-kube-api-access-gq8x2\") pod \"test-model-f1dea9-kserve-68db9cf48f-2v9kp\" (UID: \"1de62093-468d-44a1-9948-0bec84811146\") " pod="ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp"
Jul 14 14:05:29.660834 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:05:29.660761    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1de62093-468d-44a1-9948-0bec84811146-home\") pod \"test-model-f1dea9-kserve-68db9cf48f-2v9kp\" (UID: \"1de62093-468d-44a1-9948-0bec84811146\") " pod="ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp"
Jul 14 14:05:29.660834 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:05:29.660801    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1de62093-468d-44a1-9948-0bec84811146-tmp-dir\") pod \"test-model-f1dea9-kserve-68db9cf48f-2v9kp\" (UID: \"1de62093-468d-44a1-9948-0bec84811146\") " pod="ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp"
Jul 14 14:05:29.661031 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:05:29.660872    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1de62093-468d-44a1-9948-0bec84811146-model-cache\") pod \"test-model-f1dea9-kserve-68db9cf48f-2v9kp\" (UID: \"1de62093-468d-44a1-9948-0bec84811146\") " pod="ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp"
Jul 14 14:05:29.661204 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:05:29.661185    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1de62093-468d-44a1-9948-0bec84811146-home\") pod \"test-model-f1dea9-kserve-68db9cf48f-2v9kp\" (UID: \"1de62093-468d-44a1-9948-0bec84811146\") " pod="ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp"
Jul 14 14:05:29.661270 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:05:29.661218    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1de62093-468d-44a1-9948-0bec84811146-model-cache\") pod \"test-model-f1dea9-kserve-68db9cf48f-2v9kp\" (UID: \"1de62093-468d-44a1-9948-0bec84811146\") " pod="ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp"
Jul 14 14:05:29.661310 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:05:29.661282    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1de62093-468d-44a1-9948-0bec84811146-tmp-dir\") pod \"test-model-f1dea9-kserve-68db9cf48f-2v9kp\" (UID: \"1de62093-468d-44a1-9948-0bec84811146\") " pod="ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp"
Jul 14 14:05:29.662933 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:05:29.662907    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1de62093-468d-44a1-9948-0bec84811146-dshm\") pod \"test-model-f1dea9-kserve-68db9cf48f-2v9kp\" (UID: \"1de62093-468d-44a1-9948-0bec84811146\") " pod="ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp"
Jul 14 14:05:29.663070 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:05:29.663054    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1de62093-468d-44a1-9948-0bec84811146-tls-certs\") pod \"test-model-f1dea9-kserve-68db9cf48f-2v9kp\" (UID: \"1de62093-468d-44a1-9948-0bec84811146\") " pod="ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp"
Jul 14 14:05:29.670639 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:05:29.670613    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq8x2\" (UniqueName: \"kubernetes.io/projected/1de62093-468d-44a1-9948-0bec84811146-kube-api-access-gq8x2\") pod \"test-model-f1dea9-kserve-68db9cf48f-2v9kp\" (UID: \"1de62093-468d-44a1-9948-0bec84811146\") " pod="ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp"
Jul 14 14:05:29.802719 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:05:29.802668    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp"
Jul 14 14:05:29.933349 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:05:29.933318    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp"]
Jul 14 14:05:29.934586 ip-10-0-140-78 kubenswrapper[2569]: W0714 14:05:29.934557    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1de62093_468d_44a1_9948_0bec84811146.slice/crio-fa93e72baad97a2f6d81264493d61617c18d57a332d544a044e7a5d395700311 WatchSource:0}: Error finding container fa93e72baad97a2f6d81264493d61617c18d57a332d544a044e7a5d395700311: Status 404 returned error can't find the container with id fa93e72baad97a2f6d81264493d61617c18d57a332d544a044e7a5d395700311
Jul 14 14:05:30.223729 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:05:30.223645    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp" event={"ID":"1de62093-468d-44a1-9948-0bec84811146","Type":"ContainerStarted","Data":"0facbc2c31c59104d2b8f1cacc36d34972e5798d06699833b4eb30a0971956e9"}
Jul 14 14:05:30.223729 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:05:30.223688    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp" event={"ID":"1de62093-468d-44a1-9948-0bec84811146","Type":"ContainerStarted","Data":"fa93e72baad97a2f6d81264493d61617c18d57a332d544a044e7a5d395700311"}
Jul 14 14:05:39.803693 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:05:39.803654    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp"
Jul 14 14:05:39.804183 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:05:39.803959    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp"
Jul 14 14:05:39.816480 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:05:39.816454    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp"
Jul 14 14:05:39.843779 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:05:39.837343    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp" podStartSLOduration=10.83731397 podStartE2EDuration="10.83731397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:05:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:05:30.243036574 +0000 UTC m=+2295.569356500" watchObservedRunningTime="2026-07-14 14:05:39.83731397 +0000 UTC m=+2305.163633899"
Jul 14 14:05:40.284837 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:05:40.284814    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp"
Jul 14 14:06:05.221651 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:05.221619    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z"]
Jul 14 14:06:06.215760 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:06.215726    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp"]
Jul 14 14:06:16.873760 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:16.873728    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf"]
Jul 14 14:06:16.874182 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:16.873952    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf" podUID="c685ffa1-b678-45ad-9f8e-95caeb1de05d" containerName="maas-api" containerID="cri-o://c1554372b341983b9b24e29fe557fbb82d45e9c13613d5cd553f5fab701db133" gracePeriod=30
Jul 14 14:06:17.115954 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:17.115932    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf"
Jul 14 14:06:17.133503 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:17.133444    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c685ffa1-b678-45ad-9f8e-95caeb1de05d-openshift-service-ca\") pod \"c685ffa1-b678-45ad-9f8e-95caeb1de05d\" (UID: \"c685ffa1-b678-45ad-9f8e-95caeb1de05d\") "
Jul 14 14:06:17.133607 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:17.133513    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c685ffa1-b678-45ad-9f8e-95caeb1de05d-maas-api-tls\") pod \"c685ffa1-b678-45ad-9f8e-95caeb1de05d\" (UID: \"c685ffa1-b678-45ad-9f8e-95caeb1de05d\") "
Jul 14 14:06:17.133607 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:17.133563    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62w8q\" (UniqueName: \"kubernetes.io/projected/c685ffa1-b678-45ad-9f8e-95caeb1de05d-kube-api-access-62w8q\") pod \"c685ffa1-b678-45ad-9f8e-95caeb1de05d\" (UID: \"c685ffa1-b678-45ad-9f8e-95caeb1de05d\") "
Jul 14 14:06:17.133743 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:17.133720    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c685ffa1-b678-45ad-9f8e-95caeb1de05d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c685ffa1-b678-45ad-9f8e-95caeb1de05d" (UID: "c685ffa1-b678-45ad-9f8e-95caeb1de05d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 14:06:17.133857 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:17.133842    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c685ffa1-b678-45ad-9f8e-95caeb1de05d-openshift-service-ca\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:06:17.135527 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:17.135502    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c685ffa1-b678-45ad-9f8e-95caeb1de05d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c685ffa1-b678-45ad-9f8e-95caeb1de05d" (UID: "c685ffa1-b678-45ad-9f8e-95caeb1de05d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 14:06:17.135729 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:17.135674    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c685ffa1-b678-45ad-9f8e-95caeb1de05d-kube-api-access-62w8q" (OuterVolumeSpecName: "kube-api-access-62w8q") pod "c685ffa1-b678-45ad-9f8e-95caeb1de05d" (UID: "c685ffa1-b678-45ad-9f8e-95caeb1de05d"). InnerVolumeSpecName "kube-api-access-62w8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 14:06:17.234719 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:17.234695    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c685ffa1-b678-45ad-9f8e-95caeb1de05d-maas-api-tls\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:06:17.234846 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:17.234723    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62w8q\" (UniqueName: \"kubernetes.io/projected/c685ffa1-b678-45ad-9f8e-95caeb1de05d-kube-api-access-62w8q\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:06:17.463880 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:17.463812    2569 generic.go:356] "Generic (PLEG): container finished" podID="c685ffa1-b678-45ad-9f8e-95caeb1de05d" containerID="c1554372b341983b9b24e29fe557fbb82d45e9c13613d5cd553f5fab701db133" exitCode=0
Jul 14 14:06:17.463880 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:17.463857    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf"
Jul 14 14:06:17.463880 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:17.463861    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf" event={"ID":"c685ffa1-b678-45ad-9f8e-95caeb1de05d","Type":"ContainerDied","Data":"c1554372b341983b9b24e29fe557fbb82d45e9c13613d5cd553f5fab701db133"}
Jul 14 14:06:17.464116 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:17.463902    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf" event={"ID":"c685ffa1-b678-45ad-9f8e-95caeb1de05d","Type":"ContainerDied","Data":"ed78a6563153e0690b68ad4b00c62dd3d84746110d7777f936959e56d313d03d"}
Jul 14 14:06:17.464116 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:17.463922    2569 scope.go:117] "RemoveContainer" containerID="c1554372b341983b9b24e29fe557fbb82d45e9c13613d5cd553f5fab701db133"
Jul 14 14:06:17.472662 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:17.472641    2569 scope.go:117] "RemoveContainer" containerID="c1554372b341983b9b24e29fe557fbb82d45e9c13613d5cd553f5fab701db133"
Jul 14 14:06:17.472932 ip-10-0-140-78 kubenswrapper[2569]: E0714 14:06:17.472915    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1554372b341983b9b24e29fe557fbb82d45e9c13613d5cd553f5fab701db133\": container with ID starting with c1554372b341983b9b24e29fe557fbb82d45e9c13613d5cd553f5fab701db133 not found: ID does not exist" containerID="c1554372b341983b9b24e29fe557fbb82d45e9c13613d5cd553f5fab701db133"
Jul 14 14:06:17.472987 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:17.472939    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1554372b341983b9b24e29fe557fbb82d45e9c13613d5cd553f5fab701db133"} err="failed to get container status \"c1554372b341983b9b24e29fe557fbb82d45e9c13613d5cd553f5fab701db133\": rpc error: code = NotFound desc = could not find container \"c1554372b341983b9b24e29fe557fbb82d45e9c13613d5cd553f5fab701db133\": container with ID starting with c1554372b341983b9b24e29fe557fbb82d45e9c13613d5cd553f5fab701db133 not found: ID does not exist"
Jul 14 14:06:17.484648 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:17.484623    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf"]
Jul 14 14:06:17.489002 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:17.488981    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-370ee8-594d59f4b-qn6kf"]
Jul 14 14:06:19.193285 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:19.193244    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c685ffa1-b678-45ad-9f8e-95caeb1de05d" path="/var/lib/kubelet/pods/c685ffa1-b678-45ad-9f8e-95caeb1de05d/volumes"
Jul 14 14:06:20.242302 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:20.242237    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z" podUID="c85b3c51-a26d-404d-a4aa-3f75f3832f75" containerName="main" containerID="cri-o://eea973b145793d7afe9867d75f3f45c991cd653d9708d35400d4dfe140ddab25" gracePeriod=45
Jul 14 14:06:20.482165 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:20.482139    2569 generic.go:356] "Generic (PLEG): container finished" podID="c85b3c51-a26d-404d-a4aa-3f75f3832f75" containerID="eea973b145793d7afe9867d75f3f45c991cd653d9708d35400d4dfe140ddab25" exitCode=0
Jul 14 14:06:20.482295 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:20.482220    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z" event={"ID":"c85b3c51-a26d-404d-a4aa-3f75f3832f75","Type":"ContainerDied","Data":"eea973b145793d7afe9867d75f3f45c991cd653d9708d35400d4dfe140ddab25"}
Jul 14 14:06:20.482295 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:20.482265    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z" event={"ID":"c85b3c51-a26d-404d-a4aa-3f75f3832f75","Type":"ContainerDied","Data":"1db0013e432ce158469ef9be5142853bdfb45ce5e7e4d482671da4e95db5be22"}
Jul 14 14:06:20.482295 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:20.482283    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1db0013e432ce158469ef9be5142853bdfb45ce5e7e4d482671da4e95db5be22"
Jul 14 14:06:20.488570 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:20.488551    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z"
Jul 14 14:06:20.561169 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:20.561116    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c85b3c51-a26d-404d-a4aa-3f75f3832f75-dshm\") pod \"c85b3c51-a26d-404d-a4aa-3f75f3832f75\" (UID: \"c85b3c51-a26d-404d-a4aa-3f75f3832f75\") "
Jul 14 14:06:20.561169 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:20.561146    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c85b3c51-a26d-404d-a4aa-3f75f3832f75-tls-certs\") pod \"c85b3c51-a26d-404d-a4aa-3f75f3832f75\" (UID: \"c85b3c51-a26d-404d-a4aa-3f75f3832f75\") "
Jul 14 14:06:20.561295 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:20.561183    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4b4jb\" (UniqueName: \"kubernetes.io/projected/c85b3c51-a26d-404d-a4aa-3f75f3832f75-kube-api-access-4b4jb\") pod \"c85b3c51-a26d-404d-a4aa-3f75f3832f75\" (UID: \"c85b3c51-a26d-404d-a4aa-3f75f3832f75\") "
Jul 14 14:06:20.561295 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:20.561206    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c85b3c51-a26d-404d-a4aa-3f75f3832f75-tmp-dir\") pod \"c85b3c51-a26d-404d-a4aa-3f75f3832f75\" (UID: \"c85b3c51-a26d-404d-a4aa-3f75f3832f75\") "
Jul 14 14:06:20.561295 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:20.561226    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c85b3c51-a26d-404d-a4aa-3f75f3832f75-model-cache\") pod \"c85b3c51-a26d-404d-a4aa-3f75f3832f75\" (UID: \"c85b3c51-a26d-404d-a4aa-3f75f3832f75\") "
Jul 14 14:06:20.561295 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:20.561254    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c85b3c51-a26d-404d-a4aa-3f75f3832f75-home\") pod \"c85b3c51-a26d-404d-a4aa-3f75f3832f75\" (UID: \"c85b3c51-a26d-404d-a4aa-3f75f3832f75\") "
Jul 14 14:06:20.561545 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:20.561508    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c85b3c51-a26d-404d-a4aa-3f75f3832f75-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c85b3c51-a26d-404d-a4aa-3f75f3832f75" (UID: "c85b3c51-a26d-404d-a4aa-3f75f3832f75"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:06:20.561545 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:20.561535    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c85b3c51-a26d-404d-a4aa-3f75f3832f75-model-cache" (OuterVolumeSpecName: "model-cache") pod "c85b3c51-a26d-404d-a4aa-3f75f3832f75" (UID: "c85b3c51-a26d-404d-a4aa-3f75f3832f75"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:06:20.561682 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:20.561619    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c85b3c51-a26d-404d-a4aa-3f75f3832f75-home" (OuterVolumeSpecName: "home") pod "c85b3c51-a26d-404d-a4aa-3f75f3832f75" (UID: "c85b3c51-a26d-404d-a4aa-3f75f3832f75"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:06:20.563341 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:20.563319    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c85b3c51-a26d-404d-a4aa-3f75f3832f75-dshm" (OuterVolumeSpecName: "dshm") pod "c85b3c51-a26d-404d-a4aa-3f75f3832f75" (UID: "c85b3c51-a26d-404d-a4aa-3f75f3832f75"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:06:20.563423 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:20.563387    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c85b3c51-a26d-404d-a4aa-3f75f3832f75-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c85b3c51-a26d-404d-a4aa-3f75f3832f75" (UID: "c85b3c51-a26d-404d-a4aa-3f75f3832f75"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 14:06:20.563423 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:20.563396    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c85b3c51-a26d-404d-a4aa-3f75f3832f75-kube-api-access-4b4jb" (OuterVolumeSpecName: "kube-api-access-4b4jb") pod "c85b3c51-a26d-404d-a4aa-3f75f3832f75" (UID: "c85b3c51-a26d-404d-a4aa-3f75f3832f75"). InnerVolumeSpecName "kube-api-access-4b4jb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 14:06:20.662596 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:20.662564    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c85b3c51-a26d-404d-a4aa-3f75f3832f75-dshm\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:06:20.662596 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:20.662594    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c85b3c51-a26d-404d-a4aa-3f75f3832f75-tls-certs\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:06:20.662596 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:20.662603    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b4jb\" (UniqueName: \"kubernetes.io/projected/c85b3c51-a26d-404d-a4aa-3f75f3832f75-kube-api-access-4b4jb\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:06:20.662780 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:20.662611    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c85b3c51-a26d-404d-a4aa-3f75f3832f75-tmp-dir\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:06:20.662780 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:20.662620    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c85b3c51-a26d-404d-a4aa-3f75f3832f75-model-cache\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:06:20.662780 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:20.662628    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c85b3c51-a26d-404d-a4aa-3f75f3832f75-home\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:06:21.236081 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:21.236050    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp" podUID="1de62093-468d-44a1-9948-0bec84811146" containerName="main" containerID="cri-o://0facbc2c31c59104d2b8f1cacc36d34972e5798d06699833b4eb30a0971956e9" gracePeriod=45
Jul 14 14:06:21.477458 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:21.477434    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp"
Jul 14 14:06:21.487566 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:21.487539    2569 generic.go:356] "Generic (PLEG): container finished" podID="1de62093-468d-44a1-9948-0bec84811146" containerID="0facbc2c31c59104d2b8f1cacc36d34972e5798d06699833b4eb30a0971956e9" exitCode=0
Jul 14 14:06:21.487663 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:21.487571    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp"
Jul 14 14:06:21.487663 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:21.487619    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp" event={"ID":"1de62093-468d-44a1-9948-0bec84811146","Type":"ContainerDied","Data":"0facbc2c31c59104d2b8f1cacc36d34972e5798d06699833b4eb30a0971956e9"}
Jul 14 14:06:21.487779 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:21.487666    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp" event={"ID":"1de62093-468d-44a1-9948-0bec84811146","Type":"ContainerDied","Data":"fa93e72baad97a2f6d81264493d61617c18d57a332d544a044e7a5d395700311"}
Jul 14 14:06:21.487779 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:21.487692    2569 scope.go:117] "RemoveContainer" containerID="0facbc2c31c59104d2b8f1cacc36d34972e5798d06699833b4eb30a0971956e9"
Jul 14 14:06:21.487779 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:21.487703    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z"
Jul 14 14:06:21.499815 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:21.499792    2569 scope.go:117] "RemoveContainer" containerID="0facbc2c31c59104d2b8f1cacc36d34972e5798d06699833b4eb30a0971956e9"
Jul 14 14:06:21.500331 ip-10-0-140-78 kubenswrapper[2569]: E0714 14:06:21.500307    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0facbc2c31c59104d2b8f1cacc36d34972e5798d06699833b4eb30a0971956e9\": container with ID starting with 0facbc2c31c59104d2b8f1cacc36d34972e5798d06699833b4eb30a0971956e9 not found: ID does not exist" containerID="0facbc2c31c59104d2b8f1cacc36d34972e5798d06699833b4eb30a0971956e9"
Jul 14 14:06:21.500418 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:21.500339    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0facbc2c31c59104d2b8f1cacc36d34972e5798d06699833b4eb30a0971956e9"} err="failed to get container status \"0facbc2c31c59104d2b8f1cacc36d34972e5798d06699833b4eb30a0971956e9\": rpc error: code = NotFound desc = could not find container \"0facbc2c31c59104d2b8f1cacc36d34972e5798d06699833b4eb30a0971956e9\": container with ID starting with 0facbc2c31c59104d2b8f1cacc36d34972e5798d06699833b4eb30a0971956e9 not found: ID does not exist"
Jul 14 14:06:21.514351 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:21.514330    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z"]
Jul 14 14:06:21.522720 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:21.522699    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-370ee8/test-model-370ee8-kserve-674b56cc69-n689z"]
Jul 14 14:06:21.567274 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:21.567248    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1de62093-468d-44a1-9948-0bec84811146-tls-certs\") pod \"1de62093-468d-44a1-9948-0bec84811146\" (UID: \"1de62093-468d-44a1-9948-0bec84811146\") "
Jul 14 14:06:21.567373 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:21.567308    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1de62093-468d-44a1-9948-0bec84811146-tmp-dir\") pod \"1de62093-468d-44a1-9948-0bec84811146\" (UID: \"1de62093-468d-44a1-9948-0bec84811146\") "
Jul 14 14:06:21.567438 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:21.567375    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1de62093-468d-44a1-9948-0bec84811146-home\") pod \"1de62093-468d-44a1-9948-0bec84811146\" (UID: \"1de62093-468d-44a1-9948-0bec84811146\") "
Jul 14 14:06:21.567438 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:21.567409    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1de62093-468d-44a1-9948-0bec84811146-model-cache\") pod \"1de62093-468d-44a1-9948-0bec84811146\" (UID: \"1de62093-468d-44a1-9948-0bec84811146\") "
Jul 14 14:06:21.567438 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:21.567432    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1de62093-468d-44a1-9948-0bec84811146-dshm\") pod \"1de62093-468d-44a1-9948-0bec84811146\" (UID: \"1de62093-468d-44a1-9948-0bec84811146\") "
Jul 14 14:06:21.567640 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:21.567486    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gq8x2\" (UniqueName: \"kubernetes.io/projected/1de62093-468d-44a1-9948-0bec84811146-kube-api-access-gq8x2\") pod \"1de62093-468d-44a1-9948-0bec84811146\" (UID: \"1de62093-468d-44a1-9948-0bec84811146\") "
Jul 14 14:06:21.567640 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:21.567497    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1de62093-468d-44a1-9948-0bec84811146-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1de62093-468d-44a1-9948-0bec84811146" (UID: "1de62093-468d-44a1-9948-0bec84811146"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:06:21.567730 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:21.567630    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1de62093-468d-44a1-9948-0bec84811146-home" (OuterVolumeSpecName: "home") pod "1de62093-468d-44a1-9948-0bec84811146" (UID: "1de62093-468d-44a1-9948-0bec84811146"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:06:21.567730 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:21.567660    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1de62093-468d-44a1-9948-0bec84811146-model-cache" (OuterVolumeSpecName: "model-cache") pod "1de62093-468d-44a1-9948-0bec84811146" (UID: "1de62093-468d-44a1-9948-0bec84811146"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:06:21.567811 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:21.567775    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1de62093-468d-44a1-9948-0bec84811146-tmp-dir\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:06:21.567811 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:21.567792    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1de62093-468d-44a1-9948-0bec84811146-home\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:06:21.567811 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:21.567805    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1de62093-468d-44a1-9948-0bec84811146-model-cache\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:06:21.569496 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:21.569471    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1de62093-468d-44a1-9948-0bec84811146-dshm" (OuterVolumeSpecName: "dshm") pod "1de62093-468d-44a1-9948-0bec84811146" (UID: "1de62093-468d-44a1-9948-0bec84811146"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:06:21.569587 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:21.569500    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1de62093-468d-44a1-9948-0bec84811146-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1de62093-468d-44a1-9948-0bec84811146" (UID: "1de62093-468d-44a1-9948-0bec84811146"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 14:06:21.569587 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:21.569538    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1de62093-468d-44a1-9948-0bec84811146-kube-api-access-gq8x2" (OuterVolumeSpecName: "kube-api-access-gq8x2") pod "1de62093-468d-44a1-9948-0bec84811146" (UID: "1de62093-468d-44a1-9948-0bec84811146"). InnerVolumeSpecName "kube-api-access-gq8x2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 14:06:21.668367 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:21.668332    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1de62093-468d-44a1-9948-0bec84811146-dshm\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:06:21.668367 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:21.668364    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq8x2\" (UniqueName: \"kubernetes.io/projected/1de62093-468d-44a1-9948-0bec84811146-kube-api-access-gq8x2\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:06:21.668478 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:21.668379    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1de62093-468d-44a1-9948-0bec84811146-tls-certs\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:06:21.810723 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:21.810697    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp"]
Jul 14 14:06:21.817377 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:21.817358    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f1dea9/test-model-f1dea9-kserve-68db9cf48f-2v9kp"]
Jul 14 14:06:23.198161 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:23.194444    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1de62093-468d-44a1-9948-0bec84811146" path="/var/lib/kubelet/pods/1de62093-468d-44a1-9948-0bec84811146/volumes"
Jul 14 14:06:23.198161 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:23.195050    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c85b3c51-a26d-404d-a4aa-3f75f3832f75" path="/var/lib/kubelet/pods/c85b3c51-a26d-404d-a4aa-3f75f3832f75/volumes"
Jul 14 14:06:42.969394 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:42.969362    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f1dea9-b9d479885-782z5"]
Jul 14 14:06:42.969964 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:42.969570    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1dea9-b9d479885-782z5" podUID="68ff4bf3-5f48-4625-a9ea-e4ed733f2270" containerName="maas-api" containerID="cri-o://3e61136911ce6f461ace7929f460eee929714b63cbe0237c5b8d4b029ff890dc" gracePeriod=30
Jul 14 14:06:43.208920 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:43.208896    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1dea9-b9d479885-782z5"
Jul 14 14:06:43.324750 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:43.324724    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fthjn\" (UniqueName: \"kubernetes.io/projected/68ff4bf3-5f48-4625-a9ea-e4ed733f2270-kube-api-access-fthjn\") pod \"68ff4bf3-5f48-4625-a9ea-e4ed733f2270\" (UID: \"68ff4bf3-5f48-4625-a9ea-e4ed733f2270\") "
Jul 14 14:06:43.324880 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:43.324782    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68ff4bf3-5f48-4625-a9ea-e4ed733f2270-maas-api-tls\") pod \"68ff4bf3-5f48-4625-a9ea-e4ed733f2270\" (UID: \"68ff4bf3-5f48-4625-a9ea-e4ed733f2270\") "
Jul 14 14:06:43.324880 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:43.324825    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ff4bf3-5f48-4625-a9ea-e4ed733f2270-openshift-service-ca\") pod \"68ff4bf3-5f48-4625-a9ea-e4ed733f2270\" (UID: \"68ff4bf3-5f48-4625-a9ea-e4ed733f2270\") "
Jul 14 14:06:43.325254 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:43.325221    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68ff4bf3-5f48-4625-a9ea-e4ed733f2270-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "68ff4bf3-5f48-4625-a9ea-e4ed733f2270" (UID: "68ff4bf3-5f48-4625-a9ea-e4ed733f2270"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 14:06:43.326725 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:43.326705    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68ff4bf3-5f48-4625-a9ea-e4ed733f2270-kube-api-access-fthjn" (OuterVolumeSpecName: "kube-api-access-fthjn") pod "68ff4bf3-5f48-4625-a9ea-e4ed733f2270" (UID: "68ff4bf3-5f48-4625-a9ea-e4ed733f2270"). InnerVolumeSpecName "kube-api-access-fthjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 14:06:43.326844 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:43.326824    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68ff4bf3-5f48-4625-a9ea-e4ed733f2270-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "68ff4bf3-5f48-4625-a9ea-e4ed733f2270" (UID: "68ff4bf3-5f48-4625-a9ea-e4ed733f2270"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 14:06:43.426040 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:43.425988    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fthjn\" (UniqueName: \"kubernetes.io/projected/68ff4bf3-5f48-4625-a9ea-e4ed733f2270-kube-api-access-fthjn\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:06:43.426040 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:43.426039    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68ff4bf3-5f48-4625-a9ea-e4ed733f2270-maas-api-tls\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:06:43.426040 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:43.426049    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ff4bf3-5f48-4625-a9ea-e4ed733f2270-openshift-service-ca\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:06:43.599701 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:43.599644    2569 generic.go:356] "Generic (PLEG): container finished" podID="68ff4bf3-5f48-4625-a9ea-e4ed733f2270" containerID="3e61136911ce6f461ace7929f460eee929714b63cbe0237c5b8d4b029ff890dc" exitCode=0
Jul 14 14:06:43.599701 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:43.599686    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1dea9-b9d479885-782z5"
Jul 14 14:06:43.599819 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:43.599726    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1dea9-b9d479885-782z5" event={"ID":"68ff4bf3-5f48-4625-a9ea-e4ed733f2270","Type":"ContainerDied","Data":"3e61136911ce6f461ace7929f460eee929714b63cbe0237c5b8d4b029ff890dc"}
Jul 14 14:06:43.599819 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:43.599776    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f1dea9-b9d479885-782z5" event={"ID":"68ff4bf3-5f48-4625-a9ea-e4ed733f2270","Type":"ContainerDied","Data":"8362d82e62aeb4d23554a7dae54c219c1f5d6d126f3bf2a3d760c6e5b3c1ba9c"}
Jul 14 14:06:43.599819 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:43.599798    2569 scope.go:117] "RemoveContainer" containerID="3e61136911ce6f461ace7929f460eee929714b63cbe0237c5b8d4b029ff890dc"
Jul 14 14:06:43.608782 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:43.608767    2569 scope.go:117] "RemoveContainer" containerID="3e61136911ce6f461ace7929f460eee929714b63cbe0237c5b8d4b029ff890dc"
Jul 14 14:06:43.609061 ip-10-0-140-78 kubenswrapper[2569]: E0714 14:06:43.609035    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e61136911ce6f461ace7929f460eee929714b63cbe0237c5b8d4b029ff890dc\": container with ID starting with 3e61136911ce6f461ace7929f460eee929714b63cbe0237c5b8d4b029ff890dc not found: ID does not exist" containerID="3e61136911ce6f461ace7929f460eee929714b63cbe0237c5b8d4b029ff890dc"
Jul 14 14:06:43.609142 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:43.609069    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e61136911ce6f461ace7929f460eee929714b63cbe0237c5b8d4b029ff890dc"} err="failed to get container status \"3e61136911ce6f461ace7929f460eee929714b63cbe0237c5b8d4b029ff890dc\": rpc error: code = NotFound desc = could not find container \"3e61136911ce6f461ace7929f460eee929714b63cbe0237c5b8d4b029ff890dc\": container with ID starting with 3e61136911ce6f461ace7929f460eee929714b63cbe0237c5b8d4b029ff890dc not found: ID does not exist"
Jul 14 14:06:43.622184 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:43.622157    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f1dea9-b9d479885-782z5"]
Jul 14 14:06:43.630434 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:43.630416    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f1dea9-b9d479885-782z5"]
Jul 14 14:06:45.193470 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:06:45.193429    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68ff4bf3-5f48-4625-a9ea-e4ed733f2270" path="/var/lib/kubelet/pods/68ff4bf3-5f48-4625-a9ea-e4ed733f2270/volumes"
Jul 14 14:07:02.095462 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:02.095431    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97"]
Jul 14 14:07:02.095925 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:02.095744    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c685ffa1-b678-45ad-9f8e-95caeb1de05d" containerName="maas-api"
Jul 14 14:07:02.095925 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:02.095755    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c685ffa1-b678-45ad-9f8e-95caeb1de05d" containerName="maas-api"
Jul 14 14:07:02.095925 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:02.095766    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68ff4bf3-5f48-4625-a9ea-e4ed733f2270" containerName="maas-api"
Jul 14 14:07:02.095925 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:02.095771    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="68ff4bf3-5f48-4625-a9ea-e4ed733f2270" containerName="maas-api"
Jul 14 14:07:02.095925 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:02.095785    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1de62093-468d-44a1-9948-0bec84811146" containerName="main"
Jul 14 14:07:02.095925 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:02.095791    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1de62093-468d-44a1-9948-0bec84811146" containerName="main"
Jul 14 14:07:02.095925 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:02.095800    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c85b3c51-a26d-404d-a4aa-3f75f3832f75" containerName="main"
Jul 14 14:07:02.095925 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:02.095805    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c85b3c51-a26d-404d-a4aa-3f75f3832f75" containerName="main"
Jul 14 14:07:02.095925 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:02.095858    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c685ffa1-b678-45ad-9f8e-95caeb1de05d" containerName="maas-api"
Jul 14 14:07:02.095925 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:02.095867    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1de62093-468d-44a1-9948-0bec84811146" containerName="main"
Jul 14 14:07:02.095925 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:02.095873    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="68ff4bf3-5f48-4625-a9ea-e4ed733f2270" containerName="maas-api"
Jul 14 14:07:02.095925 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:02.095879    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c85b3c51-a26d-404d-a4aa-3f75f3832f75" containerName="main"
Jul 14 14:07:02.098642 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:02.098623    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97"
Jul 14 14:07:02.101146 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:02.101127    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f2a083-serving-cert\""
Jul 14 14:07:02.101242 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:02.101220    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wkxqn\""
Jul 14 14:07:02.109133 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:02.109113    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97"]
Jul 14 14:07:02.166834 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:02.166815    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwpqz\" (UniqueName: \"kubernetes.io/projected/814ea20a-9d95-46c8-933d-e9de4bf0ae65-kube-api-access-gwpqz\") pod \"maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97\" (UID: \"814ea20a-9d95-46c8-933d-e9de4bf0ae65\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97"
Jul 14 14:07:02.166925 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:02.166858    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/814ea20a-9d95-46c8-933d-e9de4bf0ae65-maas-api-tls\") pod \"maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97\" (UID: \"814ea20a-9d95-46c8-933d-e9de4bf0ae65\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97"
Jul 14 14:07:02.166925 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:02.166920    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/814ea20a-9d95-46c8-933d-e9de4bf0ae65-openshift-service-ca\") pod \"maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97\" (UID: \"814ea20a-9d95-46c8-933d-e9de4bf0ae65\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97"
Jul 14 14:07:02.267898 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:02.267874    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/814ea20a-9d95-46c8-933d-e9de4bf0ae65-openshift-service-ca\") pod \"maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97\" (UID: \"814ea20a-9d95-46c8-933d-e9de4bf0ae65\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97"
Jul 14 14:07:02.267981 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:02.267914    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwpqz\" (UniqueName: \"kubernetes.io/projected/814ea20a-9d95-46c8-933d-e9de4bf0ae65-kube-api-access-gwpqz\") pod \"maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97\" (UID: \"814ea20a-9d95-46c8-933d-e9de4bf0ae65\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97"
Jul 14 14:07:02.267981 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:02.267972    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/814ea20a-9d95-46c8-933d-e9de4bf0ae65-maas-api-tls\") pod \"maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97\" (UID: \"814ea20a-9d95-46c8-933d-e9de4bf0ae65\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97"
Jul 14 14:07:02.268116 ip-10-0-140-78 kubenswrapper[2569]: E0714 14:07:02.268101    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-f2a083-serving-cert: secret "maas-api-e2e-api-a-f2a083-serving-cert" not found
Jul 14 14:07:02.268175 ip-10-0-140-78 kubenswrapper[2569]: E0714 14:07:02.268164    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/814ea20a-9d95-46c8-933d-e9de4bf0ae65-maas-api-tls podName:814ea20a-9d95-46c8-933d-e9de4bf0ae65 nodeName:}" failed. No retries permitted until 2026-07-14 14:07:02.768143014 +0000 UTC m=+2388.094462922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/814ea20a-9d95-46c8-933d-e9de4bf0ae65-maas-api-tls") pod "maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97" (UID: "814ea20a-9d95-46c8-933d-e9de4bf0ae65") : secret "maas-api-e2e-api-a-f2a083-serving-cert" not found
Jul 14 14:07:02.268555 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:02.268539    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/814ea20a-9d95-46c8-933d-e9de4bf0ae65-openshift-service-ca\") pod \"maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97\" (UID: \"814ea20a-9d95-46c8-933d-e9de4bf0ae65\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97"
Jul 14 14:07:02.279340 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:02.279320    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwpqz\" (UniqueName: \"kubernetes.io/projected/814ea20a-9d95-46c8-933d-e9de4bf0ae65-kube-api-access-gwpqz\") pod \"maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97\" (UID: \"814ea20a-9d95-46c8-933d-e9de4bf0ae65\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97"
Jul 14 14:07:02.772453 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:02.772414    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/814ea20a-9d95-46c8-933d-e9de4bf0ae65-maas-api-tls\") pod \"maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97\" (UID: \"814ea20a-9d95-46c8-933d-e9de4bf0ae65\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97"
Jul 14 14:07:02.774693 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:02.774665    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/814ea20a-9d95-46c8-933d-e9de4bf0ae65-maas-api-tls\") pod \"maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97\" (UID: \"814ea20a-9d95-46c8-933d-e9de4bf0ae65\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97"
Jul 14 14:07:03.010890 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:03.010860    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97"
Jul 14 14:07:03.144982 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:03.144954    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97"]
Jul 14 14:07:03.146818 ip-10-0-140-78 kubenswrapper[2569]: W0714 14:07:03.146788    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod814ea20a_9d95_46c8_933d_e9de4bf0ae65.slice/crio-6f3f42da5cc1753e474f1b21ee3ca621c66c4dabaedf9a28e60eff2283b9cb8c WatchSource:0}: Error finding container 6f3f42da5cc1753e474f1b21ee3ca621c66c4dabaedf9a28e60eff2283b9cb8c: Status 404 returned error can't find the container with id 6f3f42da5cc1753e474f1b21ee3ca621c66c4dabaedf9a28e60eff2283b9cb8c
Jul 14 14:07:03.148242 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:03.148225    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 14:07:03.704655 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:03.704605    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97" event={"ID":"814ea20a-9d95-46c8-933d-e9de4bf0ae65","Type":"ContainerStarted","Data":"757483e09c6d3e77b08c98a90e85cd9efc0c54b92e705a0a086e2b4d90af43ec"}
Jul 14 14:07:03.704768 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:03.704674    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97" event={"ID":"814ea20a-9d95-46c8-933d-e9de4bf0ae65","Type":"ContainerStarted","Data":"6f3f42da5cc1753e474f1b21ee3ca621c66c4dabaedf9a28e60eff2283b9cb8c"}
Jul 14 14:07:03.704768 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:03.704705    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97"
Jul 14 14:07:03.722282 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:03.722239    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97" podStartSLOduration=1.293939198 podStartE2EDuration="1.722226568s" totalImagesPullingTime="428.28737ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:07:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:07:03.720090747 +0000 UTC m=+2389.046410675" watchObservedRunningTime="2026-07-14 14:07:03.722226568 +0000 UTC m=+2389.048546497"
Jul 14 14:07:09.716236 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:09.716208    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97"
Jul 14 14:07:10.328704 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:10.328674    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1816dc-5864df795-scs8b"]
Jul 14 14:07:10.332062 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:10.332042    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1816dc-5864df795-scs8b"
Jul 14 14:07:10.334652 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:10.334634    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-1816dc-serving-cert\""
Jul 14 14:07:10.344253 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:10.344232    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1816dc-5864df795-scs8b"]
Jul 14 14:07:10.429851 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:10.429825    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42804fe3-7b29-4012-a00b-4afcbbe5a481-maas-api-tls\") pod \"maas-api-e2e-api-b-1816dc-5864df795-scs8b\" (UID: \"42804fe3-7b29-4012-a00b-4afcbbe5a481\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1816dc-5864df795-scs8b"
Jul 14 14:07:10.429956 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:10.429855    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42804fe3-7b29-4012-a00b-4afcbbe5a481-openshift-service-ca\") pod \"maas-api-e2e-api-b-1816dc-5864df795-scs8b\" (UID: \"42804fe3-7b29-4012-a00b-4afcbbe5a481\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1816dc-5864df795-scs8b"
Jul 14 14:07:10.429956 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:10.429898    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kprsm\" (UniqueName: \"kubernetes.io/projected/42804fe3-7b29-4012-a00b-4afcbbe5a481-kube-api-access-kprsm\") pod \"maas-api-e2e-api-b-1816dc-5864df795-scs8b\" (UID: \"42804fe3-7b29-4012-a00b-4afcbbe5a481\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1816dc-5864df795-scs8b"
Jul 14 14:07:10.530836 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:10.530810    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kprsm\" (UniqueName: \"kubernetes.io/projected/42804fe3-7b29-4012-a00b-4afcbbe5a481-kube-api-access-kprsm\") pod \"maas-api-e2e-api-b-1816dc-5864df795-scs8b\" (UID: \"42804fe3-7b29-4012-a00b-4afcbbe5a481\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1816dc-5864df795-scs8b"
Jul 14 14:07:10.530950 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:10.530861    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42804fe3-7b29-4012-a00b-4afcbbe5a481-maas-api-tls\") pod \"maas-api-e2e-api-b-1816dc-5864df795-scs8b\" (UID: \"42804fe3-7b29-4012-a00b-4afcbbe5a481\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1816dc-5864df795-scs8b"
Jul 14 14:07:10.530950 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:10.530881    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42804fe3-7b29-4012-a00b-4afcbbe5a481-openshift-service-ca\") pod \"maas-api-e2e-api-b-1816dc-5864df795-scs8b\" (UID: \"42804fe3-7b29-4012-a00b-4afcbbe5a481\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1816dc-5864df795-scs8b"
Jul 14 14:07:10.531475 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:10.531457    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42804fe3-7b29-4012-a00b-4afcbbe5a481-openshift-service-ca\") pod \"maas-api-e2e-api-b-1816dc-5864df795-scs8b\" (UID: \"42804fe3-7b29-4012-a00b-4afcbbe5a481\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1816dc-5864df795-scs8b"
Jul 14 14:07:10.533217 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:10.533198    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42804fe3-7b29-4012-a00b-4afcbbe5a481-maas-api-tls\") pod \"maas-api-e2e-api-b-1816dc-5864df795-scs8b\" (UID: \"42804fe3-7b29-4012-a00b-4afcbbe5a481\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1816dc-5864df795-scs8b"
Jul 14 14:07:10.541838 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:10.541815    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kprsm\" (UniqueName: \"kubernetes.io/projected/42804fe3-7b29-4012-a00b-4afcbbe5a481-kube-api-access-kprsm\") pod \"maas-api-e2e-api-b-1816dc-5864df795-scs8b\" (UID: \"42804fe3-7b29-4012-a00b-4afcbbe5a481\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1816dc-5864df795-scs8b"
Jul 14 14:07:10.643258 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:10.643196    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1816dc-5864df795-scs8b"
Jul 14 14:07:10.775349 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:10.775327    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1816dc-5864df795-scs8b"]
Jul 14 14:07:10.777288 ip-10-0-140-78 kubenswrapper[2569]: W0714 14:07:10.777261    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42804fe3_7b29_4012_a00b_4afcbbe5a481.slice/crio-629dabcf7e369d8e254dfe57e392bb2d747e848a281acf5183f6718978ce2340 WatchSource:0}: Error finding container 629dabcf7e369d8e254dfe57e392bb2d747e848a281acf5183f6718978ce2340: Status 404 returned error can't find the container with id 629dabcf7e369d8e254dfe57e392bb2d747e848a281acf5183f6718978ce2340
Jul 14 14:07:11.749432 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:11.749403    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1816dc-5864df795-scs8b" event={"ID":"42804fe3-7b29-4012-a00b-4afcbbe5a481","Type":"ContainerStarted","Data":"4148de84b88c9f14d3617a20dfd8196579ca1cc7e29d4ccd72772d8c8e2a6471"}
Jul 14 14:07:11.749572 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:11.749452    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1816dc-5864df795-scs8b" event={"ID":"42804fe3-7b29-4012-a00b-4afcbbe5a481","Type":"ContainerStarted","Data":"629dabcf7e369d8e254dfe57e392bb2d747e848a281acf5183f6718978ce2340"}
Jul 14 14:07:11.749572 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:11.749523    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1816dc-5864df795-scs8b"
Jul 14 14:07:11.771407 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:11.771363    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1816dc-5864df795-scs8b" podStartSLOduration=1.215535819 podStartE2EDuration="1.771349887s" totalImagesPullingTime="555.814068ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:07:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:07:11.769612948 +0000 UTC m=+2397.095932877" watchObservedRunningTime="2026-07-14 14:07:11.771349887 +0000 UTC m=+2397.097669815"
Jul 14 14:07:15.271824 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:15.271737    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vq8tk_e33a489f-7d68-4e6c-9ffe-e7252049e15d/ovn-acl-logging/0.log"
Jul 14 14:07:15.279899 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:15.279879    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vq8tk_e33a489f-7d68-4e6c-9ffe-e7252049e15d/ovn-acl-logging/0.log"
Jul 14 14:07:17.759281 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:17.759254    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1816dc-5864df795-scs8b"
Jul 14 14:07:32.862132 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:32.862099    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97"]
Jul 14 14:07:32.862633 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:32.862303    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97" podUID="814ea20a-9d95-46c8-933d-e9de4bf0ae65" containerName="maas-api" containerID="cri-o://757483e09c6d3e77b08c98a90e85cd9efc0c54b92e705a0a086e2b4d90af43ec" gracePeriod=30
Jul 14 14:07:33.101781 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:33.101759    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97"
Jul 14 14:07:33.200284 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:33.200230    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/814ea20a-9d95-46c8-933d-e9de4bf0ae65-openshift-service-ca\") pod \"814ea20a-9d95-46c8-933d-e9de4bf0ae65\" (UID: \"814ea20a-9d95-46c8-933d-e9de4bf0ae65\") "
Jul 14 14:07:33.200284 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:33.200259    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwpqz\" (UniqueName: \"kubernetes.io/projected/814ea20a-9d95-46c8-933d-e9de4bf0ae65-kube-api-access-gwpqz\") pod \"814ea20a-9d95-46c8-933d-e9de4bf0ae65\" (UID: \"814ea20a-9d95-46c8-933d-e9de4bf0ae65\") "
Jul 14 14:07:33.200454 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:33.200289    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/814ea20a-9d95-46c8-933d-e9de4bf0ae65-maas-api-tls\") pod \"814ea20a-9d95-46c8-933d-e9de4bf0ae65\" (UID: \"814ea20a-9d95-46c8-933d-e9de4bf0ae65\") "
Jul 14 14:07:33.200578 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:33.200554    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/814ea20a-9d95-46c8-933d-e9de4bf0ae65-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "814ea20a-9d95-46c8-933d-e9de4bf0ae65" (UID: "814ea20a-9d95-46c8-933d-e9de4bf0ae65"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 14:07:33.202221 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:33.202195    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/814ea20a-9d95-46c8-933d-e9de4bf0ae65-kube-api-access-gwpqz" (OuterVolumeSpecName: "kube-api-access-gwpqz") pod "814ea20a-9d95-46c8-933d-e9de4bf0ae65" (UID: "814ea20a-9d95-46c8-933d-e9de4bf0ae65"). InnerVolumeSpecName "kube-api-access-gwpqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 14:07:33.202311 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:33.202266    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/814ea20a-9d95-46c8-933d-e9de4bf0ae65-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "814ea20a-9d95-46c8-933d-e9de4bf0ae65" (UID: "814ea20a-9d95-46c8-933d-e9de4bf0ae65"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 14:07:33.300980 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:33.300955    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/814ea20a-9d95-46c8-933d-e9de4bf0ae65-openshift-service-ca\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:07:33.300980 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:33.300976    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwpqz\" (UniqueName: \"kubernetes.io/projected/814ea20a-9d95-46c8-933d-e9de4bf0ae65-kube-api-access-gwpqz\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:07:33.301134 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:33.300985    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/814ea20a-9d95-46c8-933d-e9de4bf0ae65-maas-api-tls\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:07:33.869212 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:33.869182    2569 generic.go:356] "Generic (PLEG): container finished" podID="814ea20a-9d95-46c8-933d-e9de4bf0ae65" containerID="757483e09c6d3e77b08c98a90e85cd9efc0c54b92e705a0a086e2b4d90af43ec" exitCode=0
Jul 14 14:07:33.869626 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:33.869226    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97"
Jul 14 14:07:33.869626 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:33.869274    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97" event={"ID":"814ea20a-9d95-46c8-933d-e9de4bf0ae65","Type":"ContainerDied","Data":"757483e09c6d3e77b08c98a90e85cd9efc0c54b92e705a0a086e2b4d90af43ec"}
Jul 14 14:07:33.869626 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:33.869338    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97" event={"ID":"814ea20a-9d95-46c8-933d-e9de4bf0ae65","Type":"ContainerDied","Data":"6f3f42da5cc1753e474f1b21ee3ca621c66c4dabaedf9a28e60eff2283b9cb8c"}
Jul 14 14:07:33.869626 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:33.869369    2569 scope.go:117] "RemoveContainer" containerID="757483e09c6d3e77b08c98a90e85cd9efc0c54b92e705a0a086e2b4d90af43ec"
Jul 14 14:07:33.879164 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:33.879150    2569 scope.go:117] "RemoveContainer" containerID="757483e09c6d3e77b08c98a90e85cd9efc0c54b92e705a0a086e2b4d90af43ec"
Jul 14 14:07:33.879429 ip-10-0-140-78 kubenswrapper[2569]: E0714 14:07:33.879402    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"757483e09c6d3e77b08c98a90e85cd9efc0c54b92e705a0a086e2b4d90af43ec\": container with ID starting with 757483e09c6d3e77b08c98a90e85cd9efc0c54b92e705a0a086e2b4d90af43ec not found: ID does not exist" containerID="757483e09c6d3e77b08c98a90e85cd9efc0c54b92e705a0a086e2b4d90af43ec"
Jul 14 14:07:33.879526 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:33.879432    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"757483e09c6d3e77b08c98a90e85cd9efc0c54b92e705a0a086e2b4d90af43ec"} err="failed to get container status \"757483e09c6d3e77b08c98a90e85cd9efc0c54b92e705a0a086e2b4d90af43ec\": rpc error: code = NotFound desc = could not find container \"757483e09c6d3e77b08c98a90e85cd9efc0c54b92e705a0a086e2b4d90af43ec\": container with ID starting with 757483e09c6d3e77b08c98a90e85cd9efc0c54b92e705a0a086e2b4d90af43ec not found: ID does not exist"
Jul 14 14:07:33.892747 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:33.892722    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97"]
Jul 14 14:07:33.897707 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:33.897684    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f2a083-6b9c4b9bcf-k6z97"]
Jul 14 14:07:35.192875 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:35.192842    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="814ea20a-9d95-46c8-933d-e9de4bf0ae65" path="/var/lib/kubelet/pods/814ea20a-9d95-46c8-933d-e9de4bf0ae65/volumes"
Jul 14 14:07:59.236214 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:59.236178    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1816dc-5864df795-scs8b"]
Jul 14 14:07:59.236779 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:59.236389    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1816dc-5864df795-scs8b" podUID="42804fe3-7b29-4012-a00b-4afcbbe5a481" containerName="maas-api" containerID="cri-o://4148de84b88c9f14d3617a20dfd8196579ca1cc7e29d4ccd72772d8c8e2a6471" gracePeriod=30
Jul 14 14:07:59.487652 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:59.487593    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1816dc-5864df795-scs8b"
Jul 14 14:07:59.589273 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:59.589246    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kprsm\" (UniqueName: \"kubernetes.io/projected/42804fe3-7b29-4012-a00b-4afcbbe5a481-kube-api-access-kprsm\") pod \"42804fe3-7b29-4012-a00b-4afcbbe5a481\" (UID: \"42804fe3-7b29-4012-a00b-4afcbbe5a481\") "
Jul 14 14:07:59.589386 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:59.589286    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42804fe3-7b29-4012-a00b-4afcbbe5a481-maas-api-tls\") pod \"42804fe3-7b29-4012-a00b-4afcbbe5a481\" (UID: \"42804fe3-7b29-4012-a00b-4afcbbe5a481\") "
Jul 14 14:07:59.589386 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:59.589329    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42804fe3-7b29-4012-a00b-4afcbbe5a481-openshift-service-ca\") pod \"42804fe3-7b29-4012-a00b-4afcbbe5a481\" (UID: \"42804fe3-7b29-4012-a00b-4afcbbe5a481\") "
Jul 14 14:07:59.589750 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:59.589724    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42804fe3-7b29-4012-a00b-4afcbbe5a481-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "42804fe3-7b29-4012-a00b-4afcbbe5a481" (UID: "42804fe3-7b29-4012-a00b-4afcbbe5a481"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 14:07:59.591310 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:59.591290    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42804fe3-7b29-4012-a00b-4afcbbe5a481-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "42804fe3-7b29-4012-a00b-4afcbbe5a481" (UID: "42804fe3-7b29-4012-a00b-4afcbbe5a481"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 14:07:59.591400 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:59.591349    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42804fe3-7b29-4012-a00b-4afcbbe5a481-kube-api-access-kprsm" (OuterVolumeSpecName: "kube-api-access-kprsm") pod "42804fe3-7b29-4012-a00b-4afcbbe5a481" (UID: "42804fe3-7b29-4012-a00b-4afcbbe5a481"). InnerVolumeSpecName "kube-api-access-kprsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 14:07:59.689827 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:59.689806    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42804fe3-7b29-4012-a00b-4afcbbe5a481-maas-api-tls\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:07:59.689827 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:59.689827    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42804fe3-7b29-4012-a00b-4afcbbe5a481-openshift-service-ca\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:07:59.689955 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:07:59.689837    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kprsm\" (UniqueName: \"kubernetes.io/projected/42804fe3-7b29-4012-a00b-4afcbbe5a481-kube-api-access-kprsm\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:08:00.022254 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:00.022226    2569 generic.go:356] "Generic (PLEG): container finished" podID="42804fe3-7b29-4012-a00b-4afcbbe5a481" containerID="4148de84b88c9f14d3617a20dfd8196579ca1cc7e29d4ccd72772d8c8e2a6471" exitCode=0
Jul 14 14:08:00.022407 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:00.022262    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1816dc-5864df795-scs8b" event={"ID":"42804fe3-7b29-4012-a00b-4afcbbe5a481","Type":"ContainerDied","Data":"4148de84b88c9f14d3617a20dfd8196579ca1cc7e29d4ccd72772d8c8e2a6471"}
Jul 14 14:08:00.022407 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:00.022270    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1816dc-5864df795-scs8b"
Jul 14 14:08:00.022407 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:00.022300    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1816dc-5864df795-scs8b" event={"ID":"42804fe3-7b29-4012-a00b-4afcbbe5a481","Type":"ContainerDied","Data":"629dabcf7e369d8e254dfe57e392bb2d747e848a281acf5183f6718978ce2340"}
Jul 14 14:08:00.022407 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:00.022321    2569 scope.go:117] "RemoveContainer" containerID="4148de84b88c9f14d3617a20dfd8196579ca1cc7e29d4ccd72772d8c8e2a6471"
Jul 14 14:08:00.031702 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:00.031685    2569 scope.go:117] "RemoveContainer" containerID="4148de84b88c9f14d3617a20dfd8196579ca1cc7e29d4ccd72772d8c8e2a6471"
Jul 14 14:08:00.031954 ip-10-0-140-78 kubenswrapper[2569]: E0714 14:08:00.031937    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4148de84b88c9f14d3617a20dfd8196579ca1cc7e29d4ccd72772d8c8e2a6471\": container with ID starting with 4148de84b88c9f14d3617a20dfd8196579ca1cc7e29d4ccd72772d8c8e2a6471 not found: ID does not exist" containerID="4148de84b88c9f14d3617a20dfd8196579ca1cc7e29d4ccd72772d8c8e2a6471"
Jul 14 14:08:00.032026 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:00.031962    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4148de84b88c9f14d3617a20dfd8196579ca1cc7e29d4ccd72772d8c8e2a6471"} err="failed to get container status \"4148de84b88c9f14d3617a20dfd8196579ca1cc7e29d4ccd72772d8c8e2a6471\": rpc error: code = NotFound desc = could not find container \"4148de84b88c9f14d3617a20dfd8196579ca1cc7e29d4ccd72772d8c8e2a6471\": container with ID starting with 4148de84b88c9f14d3617a20dfd8196579ca1cc7e29d4ccd72772d8c8e2a6471 not found: ID does not exist"
Jul 14 14:08:00.045730 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:00.045708    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1816dc-5864df795-scs8b"]
Jul 14 14:08:00.049715 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:00.049699    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1816dc-5864df795-scs8b"]
Jul 14 14:08:01.193809 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:01.193774    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42804fe3-7b29-4012-a00b-4afcbbe5a481" path="/var/lib/kubelet/pods/42804fe3-7b29-4012-a00b-4afcbbe5a481/volumes"
Jul 14 14:08:18.706862 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:18.706786    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm"]
Jul 14 14:08:18.707358 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:18.707200    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="814ea20a-9d95-46c8-933d-e9de4bf0ae65" containerName="maas-api"
Jul 14 14:08:18.707358 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:18.707214    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="814ea20a-9d95-46c8-933d-e9de4bf0ae65" containerName="maas-api"
Jul 14 14:08:18.707358 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:18.707228    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42804fe3-7b29-4012-a00b-4afcbbe5a481" containerName="maas-api"
Jul 14 14:08:18.707358 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:18.707234    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="42804fe3-7b29-4012-a00b-4afcbbe5a481" containerName="maas-api"
Jul 14 14:08:18.707358 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:18.707291    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="814ea20a-9d95-46c8-933d-e9de4bf0ae65" containerName="maas-api"
Jul 14 14:08:18.707358 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:18.707300    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="42804fe3-7b29-4012-a00b-4afcbbe5a481" containerName="maas-api"
Jul 14 14:08:18.712408 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:18.712388    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm"
Jul 14 14:08:18.714969 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:18.714950    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-9c0e14-serving-cert\""
Jul 14 14:08:18.715101 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:18.715077    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wkxqn\""
Jul 14 14:08:18.719447 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:18.719426    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm"]
Jul 14 14:08:18.829667 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:18.829642    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ca80e71-3c4f-4462-8e26-85295439fe1b-maas-api-tls\") pod \"maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm\" (UID: \"3ca80e71-3c4f-4462-8e26-85295439fe1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm"
Jul 14 14:08:18.829770 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:18.829689    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ca80e71-3c4f-4462-8e26-85295439fe1b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm\" (UID: \"3ca80e71-3c4f-4462-8e26-85295439fe1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm"
Jul 14 14:08:18.829770 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:18.829715    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvnvp\" (UniqueName: \"kubernetes.io/projected/3ca80e71-3c4f-4462-8e26-85295439fe1b-kube-api-access-tvnvp\") pod \"maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm\" (UID: \"3ca80e71-3c4f-4462-8e26-85295439fe1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm"
Jul 14 14:08:18.931064 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:18.931040    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ca80e71-3c4f-4462-8e26-85295439fe1b-maas-api-tls\") pod \"maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm\" (UID: \"3ca80e71-3c4f-4462-8e26-85295439fe1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm"
Jul 14 14:08:18.931205 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:18.931100    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ca80e71-3c4f-4462-8e26-85295439fe1b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm\" (UID: \"3ca80e71-3c4f-4462-8e26-85295439fe1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm"
Jul 14 14:08:18.931205 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:18.931126    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvnvp\" (UniqueName: \"kubernetes.io/projected/3ca80e71-3c4f-4462-8e26-85295439fe1b-kube-api-access-tvnvp\") pod \"maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm\" (UID: \"3ca80e71-3c4f-4462-8e26-85295439fe1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm"
Jul 14 14:08:18.931332 ip-10-0-140-78 kubenswrapper[2569]: E0714 14:08:18.931202    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-9c0e14-serving-cert: secret "maas-api-e2e-shared-a-9c0e14-serving-cert" not found
Jul 14 14:08:18.931332 ip-10-0-140-78 kubenswrapper[2569]: E0714 14:08:18.931291    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca80e71-3c4f-4462-8e26-85295439fe1b-maas-api-tls podName:3ca80e71-3c4f-4462-8e26-85295439fe1b nodeName:}" failed. No retries permitted until 2026-07-14 14:08:19.431268021 +0000 UTC m=+2464.757587942 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3ca80e71-3c4f-4462-8e26-85295439fe1b-maas-api-tls") pod "maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm" (UID: "3ca80e71-3c4f-4462-8e26-85295439fe1b") : secret "maas-api-e2e-shared-a-9c0e14-serving-cert" not found
Jul 14 14:08:18.931693 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:18.931675    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ca80e71-3c4f-4462-8e26-85295439fe1b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm\" (UID: \"3ca80e71-3c4f-4462-8e26-85295439fe1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm"
Jul 14 14:08:18.945169 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:18.945144    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvnvp\" (UniqueName: \"kubernetes.io/projected/3ca80e71-3c4f-4462-8e26-85295439fe1b-kube-api-access-tvnvp\") pod \"maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm\" (UID: \"3ca80e71-3c4f-4462-8e26-85295439fe1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm"
Jul 14 14:08:19.435434 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:19.435406    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ca80e71-3c4f-4462-8e26-85295439fe1b-maas-api-tls\") pod \"maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm\" (UID: \"3ca80e71-3c4f-4462-8e26-85295439fe1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm"
Jul 14 14:08:19.437771 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:19.437752    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ca80e71-3c4f-4462-8e26-85295439fe1b-maas-api-tls\") pod \"maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm\" (UID: \"3ca80e71-3c4f-4462-8e26-85295439fe1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm"
Jul 14 14:08:19.624669 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:19.624642    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm"
Jul 14 14:08:19.760524 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:19.760498    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm"]
Jul 14 14:08:19.762072 ip-10-0-140-78 kubenswrapper[2569]: W0714 14:08:19.762040    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca80e71_3c4f_4462_8e26_85295439fe1b.slice/crio-c038fec0b3d3d709a423ff72ffb9d966a00da3522f74a0d170be83b8c5faed0d WatchSource:0}: Error finding container c038fec0b3d3d709a423ff72ffb9d966a00da3522f74a0d170be83b8c5faed0d: Status 404 returned error can't find the container with id c038fec0b3d3d709a423ff72ffb9d966a00da3522f74a0d170be83b8c5faed0d
Jul 14 14:08:20.127255 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:20.127223    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm" event={"ID":"3ca80e71-3c4f-4462-8e26-85295439fe1b","Type":"ContainerStarted","Data":"c038fec0b3d3d709a423ff72ffb9d966a00da3522f74a0d170be83b8c5faed0d"}
Jul 14 14:08:21.133807 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:21.133776    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm" event={"ID":"3ca80e71-3c4f-4462-8e26-85295439fe1b","Type":"ContainerStarted","Data":"11055ad07adb8f7ef186ca0e7bd0021d627bd8be55356f70e1d0d261dad2006c"}
Jul 14 14:08:21.134299 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:21.133881    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm"
Jul 14 14:08:21.152027 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:21.151966    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm" podStartSLOduration=2.72001659 podStartE2EDuration="3.151951942s" totalImagesPullingTime="431.935352ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:08:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:08:21.150307773 +0000 UTC m=+2466.476627734" watchObservedRunningTime="2026-07-14 14:08:21.151951942 +0000 UTC m=+2466.478271870"
Jul 14 14:08:26.811106 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:26.811053    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv"]
Jul 14 14:08:26.814885 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:26.814865    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv"
Jul 14 14:08:26.817428 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:26.817405    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-ff9f03-serving-cert\""
Jul 14 14:08:26.825552 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:26.825531    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv"]
Jul 14 14:08:26.901300 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:26.901274    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d9fs\" (UniqueName: \"kubernetes.io/projected/12480401-c287-4035-a99e-717966dda4e9-kube-api-access-8d9fs\") pod \"maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv\" (UID: \"12480401-c287-4035-a99e-717966dda4e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv"
Jul 14 14:08:26.901405 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:26.901318    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12480401-c287-4035-a99e-717966dda4e9-maas-api-tls\") pod \"maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv\" (UID: \"12480401-c287-4035-a99e-717966dda4e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv"
Jul 14 14:08:26.901405 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:26.901348    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12480401-c287-4035-a99e-717966dda4e9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv\" (UID: \"12480401-c287-4035-a99e-717966dda4e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv"
Jul 14 14:08:27.002439 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:27.002418    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d9fs\" (UniqueName: \"kubernetes.io/projected/12480401-c287-4035-a99e-717966dda4e9-kube-api-access-8d9fs\") pod \"maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv\" (UID: \"12480401-c287-4035-a99e-717966dda4e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv"
Jul 14 14:08:27.002536 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:27.002456    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12480401-c287-4035-a99e-717966dda4e9-maas-api-tls\") pod \"maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv\" (UID: \"12480401-c287-4035-a99e-717966dda4e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv"
Jul 14 14:08:27.002536 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:27.002476    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12480401-c287-4035-a99e-717966dda4e9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv\" (UID: \"12480401-c287-4035-a99e-717966dda4e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv"
Jul 14 14:08:27.003207 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:27.003190    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12480401-c287-4035-a99e-717966dda4e9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv\" (UID: \"12480401-c287-4035-a99e-717966dda4e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv"
Jul 14 14:08:27.004867 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:27.004849    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12480401-c287-4035-a99e-717966dda4e9-maas-api-tls\") pod \"maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv\" (UID: \"12480401-c287-4035-a99e-717966dda4e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv"
Jul 14 14:08:27.011039 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:27.010997    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d9fs\" (UniqueName: \"kubernetes.io/projected/12480401-c287-4035-a99e-717966dda4e9-kube-api-access-8d9fs\") pod \"maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv\" (UID: \"12480401-c287-4035-a99e-717966dda4e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv"
Jul 14 14:08:27.127413 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:27.127335    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv"
Jul 14 14:08:27.145086 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:27.145065    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm"
Jul 14 14:08:27.262792 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:27.262768    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv"]
Jul 14 14:08:27.264600 ip-10-0-140-78 kubenswrapper[2569]: W0714 14:08:27.264574    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12480401_c287_4035_a99e_717966dda4e9.slice/crio-c8c12161e1cfa6a9757bb3f38f06c48d21e6ffd2692f5d306b9932427e71458c WatchSource:0}: Error finding container c8c12161e1cfa6a9757bb3f38f06c48d21e6ffd2692f5d306b9932427e71458c: Status 404 returned error can't find the container with id c8c12161e1cfa6a9757bb3f38f06c48d21e6ffd2692f5d306b9932427e71458c
Jul 14 14:08:28.173302 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:28.173221    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv" event={"ID":"12480401-c287-4035-a99e-717966dda4e9","Type":"ContainerStarted","Data":"14f03bdeb92b69a5aa6a67ac2505c0cc9a44a117c155fce874200dd7128ab388"}
Jul 14 14:08:28.173302 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:28.173280    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv" event={"ID":"12480401-c287-4035-a99e-717966dda4e9","Type":"ContainerStarted","Data":"c8c12161e1cfa6a9757bb3f38f06c48d21e6ffd2692f5d306b9932427e71458c"}
Jul 14 14:08:28.173697 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:28.173349    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv"
Jul 14 14:08:28.192890 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:28.192848    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv" podStartSLOduration=1.722026465 podStartE2EDuration="2.192834853s" totalImagesPullingTime="470.808388ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:08:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:08:28.190931441 +0000 UTC m=+2473.517251369" watchObservedRunningTime="2026-07-14 14:08:28.192834853 +0000 UTC m=+2473.519154781"
Jul 14 14:08:34.183795 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:34.183762    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv"
Jul 14 14:08:56.992356 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:56.992319    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq"]
Jul 14 14:08:56.995692 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:56.995675    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq"
Jul 14 14:08:56.999552 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:56.999526    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9c0e14\"/\"openshift-service-ca.crt\""
Jul 14 14:08:56.999684 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:56.999561    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9c0e14\"/\"kube-root-ca.crt\""
Jul 14 14:08:56.999684 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:56.999530    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9c0e14\"/\"default-dockercfg-6lvpd\""
Jul 14 14:08:56.999684 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:56.999526    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9c0e14\"/\"auth-test-model-9c0e14-kserve-self-signed-certs\""
Jul 14 14:08:57.005368 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:57.005343    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq"]
Jul 14 14:08:57.121166 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:57.121134    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-tmp-dir\") pod \"auth-test-model-9c0e14-kserve-54d458bff5-5spxq\" (UID: \"03f432c1-7a05-4eed-a428-3f15d6a9e3c0\") " pod="ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq"
Jul 14 14:08:57.121290 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:57.121183    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-tls-certs\") pod \"auth-test-model-9c0e14-kserve-54d458bff5-5spxq\" (UID: \"03f432c1-7a05-4eed-a428-3f15d6a9e3c0\") " pod="ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq"
Jul 14 14:08:57.121290 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:57.121209    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-home\") pod \"auth-test-model-9c0e14-kserve-54d458bff5-5spxq\" (UID: \"03f432c1-7a05-4eed-a428-3f15d6a9e3c0\") " pod="ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq"
Jul 14 14:08:57.121290 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:57.121234    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-dshm\") pod \"auth-test-model-9c0e14-kserve-54d458bff5-5spxq\" (UID: \"03f432c1-7a05-4eed-a428-3f15d6a9e3c0\") " pod="ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq"
Jul 14 14:08:57.121290 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:57.121282    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzbwl\" (UniqueName: \"kubernetes.io/projected/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-kube-api-access-kzbwl\") pod \"auth-test-model-9c0e14-kserve-54d458bff5-5spxq\" (UID: \"03f432c1-7a05-4eed-a428-3f15d6a9e3c0\") " pod="ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq"
Jul 14 14:08:57.121455 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:57.121350    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-model-cache\") pod \"auth-test-model-9c0e14-kserve-54d458bff5-5spxq\" (UID: \"03f432c1-7a05-4eed-a428-3f15d6a9e3c0\") " pod="ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq"
Jul 14 14:08:57.222513 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:57.222493    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzbwl\" (UniqueName: \"kubernetes.io/projected/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-kube-api-access-kzbwl\") pod \"auth-test-model-9c0e14-kserve-54d458bff5-5spxq\" (UID: \"03f432c1-7a05-4eed-a428-3f15d6a9e3c0\") " pod="ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq"
Jul 14 14:08:57.222648 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:57.222535    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-model-cache\") pod \"auth-test-model-9c0e14-kserve-54d458bff5-5spxq\" (UID: \"03f432c1-7a05-4eed-a428-3f15d6a9e3c0\") " pod="ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq"
Jul 14 14:08:57.222648 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:57.222566    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-tmp-dir\") pod \"auth-test-model-9c0e14-kserve-54d458bff5-5spxq\" (UID: \"03f432c1-7a05-4eed-a428-3f15d6a9e3c0\") " pod="ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq"
Jul 14 14:08:57.222648 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:57.222602    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-tls-certs\") pod \"auth-test-model-9c0e14-kserve-54d458bff5-5spxq\" (UID: \"03f432c1-7a05-4eed-a428-3f15d6a9e3c0\") " pod="ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq"
Jul 14 14:08:57.222648 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:57.222619    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-home\") pod \"auth-test-model-9c0e14-kserve-54d458bff5-5spxq\" (UID: \"03f432c1-7a05-4eed-a428-3f15d6a9e3c0\") " pod="ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq"
Jul 14 14:08:57.222648 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:57.222635    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-dshm\") pod \"auth-test-model-9c0e14-kserve-54d458bff5-5spxq\" (UID: \"03f432c1-7a05-4eed-a428-3f15d6a9e3c0\") " pod="ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq"
Jul 14 14:08:57.222958 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:57.222938    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-home\") pod \"auth-test-model-9c0e14-kserve-54d458bff5-5spxq\" (UID: \"03f432c1-7a05-4eed-a428-3f15d6a9e3c0\") " pod="ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq"
Jul 14 14:08:57.223057 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:57.222968    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-model-cache\") pod \"auth-test-model-9c0e14-kserve-54d458bff5-5spxq\" (UID: \"03f432c1-7a05-4eed-a428-3f15d6a9e3c0\") " pod="ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq"
Jul 14 14:08:57.223057 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:57.223032    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-tmp-dir\") pod \"auth-test-model-9c0e14-kserve-54d458bff5-5spxq\" (UID: \"03f432c1-7a05-4eed-a428-3f15d6a9e3c0\") " pod="ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq"
Jul 14 14:08:57.224922 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:57.224903    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-dshm\") pod \"auth-test-model-9c0e14-kserve-54d458bff5-5spxq\" (UID: \"03f432c1-7a05-4eed-a428-3f15d6a9e3c0\") " pod="ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq"
Jul 14 14:08:57.225252 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:57.225233    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-tls-certs\") pod \"auth-test-model-9c0e14-kserve-54d458bff5-5spxq\" (UID: \"03f432c1-7a05-4eed-a428-3f15d6a9e3c0\") " pod="ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq"
Jul 14 14:08:57.232984 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:57.232963    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzbwl\" (UniqueName: \"kubernetes.io/projected/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-kube-api-access-kzbwl\") pod \"auth-test-model-9c0e14-kserve-54d458bff5-5spxq\" (UID: \"03f432c1-7a05-4eed-a428-3f15d6a9e3c0\") " pod="ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq"
Jul 14 14:08:57.306409 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:57.306354    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq"
Jul 14 14:08:57.443390 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:57.443365    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq"]
Jul 14 14:08:57.444735 ip-10-0-140-78 kubenswrapper[2569]: W0714 14:08:57.444705    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03f432c1_7a05_4eed_a428_3f15d6a9e3c0.slice/crio-26d43e88c06df9ae31b7210b6fa3a85c7decaee4bc1dedd4f234fbf3acbfe127 WatchSource:0}: Error finding container 26d43e88c06df9ae31b7210b6fa3a85c7decaee4bc1dedd4f234fbf3acbfe127: Status 404 returned error can't find the container with id 26d43e88c06df9ae31b7210b6fa3a85c7decaee4bc1dedd4f234fbf3acbfe127
Jul 14 14:08:58.329712 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:58.329678    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq" event={"ID":"03f432c1-7a05-4eed-a428-3f15d6a9e3c0","Type":"ContainerStarted","Data":"e35ee255ee07cc0173af54fd5a195070ffcefd19616e86eabd67931503acd403"}
Jul 14 14:08:58.330181 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:58.329945    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq" event={"ID":"03f432c1-7a05-4eed-a428-3f15d6a9e3c0","Type":"ContainerStarted","Data":"26d43e88c06df9ae31b7210b6fa3a85c7decaee4bc1dedd4f234fbf3acbfe127"}
Jul 14 14:08:58.350484 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:08:58.350439    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq" podStartSLOduration=2.350425536 podStartE2EDuration="2.350425536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:08:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:08:58.349204211 +0000 UTC m=+2503.675524140" watchObservedRunningTime="2026-07-14 14:08:58.350425536 +0000 UTC m=+2503.676745493"
Jul 14 14:09:07.307101 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:07.307072    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq"
Jul 14 14:09:07.307430 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:07.307133    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq"
Jul 14 14:09:07.319409 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:07.319386    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq"
Jul 14 14:09:07.386778 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:07.386752    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq"
Jul 14 14:09:22.589694 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:22.589660    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6"]
Jul 14 14:09:22.593004 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:22.592985    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6"
Jul 14 14:09:22.597158 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:22.597137    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ff9f03\"/\"kube-root-ca.crt\""
Jul 14 14:09:22.597266 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:22.597171    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ff9f03\"/\"auth-test-model-ff9f03-kserve-self-signed-certs\""
Jul 14 14:09:22.597340 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:22.597283    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ff9f03\"/\"default-dockercfg-pvc6j\""
Jul 14 14:09:22.597340 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:22.597305    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ff9f03\"/\"openshift-service-ca.crt\""
Jul 14 14:09:22.598861 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:22.598829    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c910658c-5c03-4f92-a335-9e2296538f10-home\") pod \"auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6\" (UID: \"c910658c-5c03-4f92-a335-9e2296538f10\") " pod="ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6"
Jul 14 14:09:22.598970 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:22.598873    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c910658c-5c03-4f92-a335-9e2296538f10-dshm\") pod \"auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6\" (UID: \"c910658c-5c03-4f92-a335-9e2296538f10\") " pod="ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6"
Jul 14 14:09:22.599043 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:22.598996    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c910658c-5c03-4f92-a335-9e2296538f10-tls-certs\") pod \"auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6\" (UID: \"c910658c-5c03-4f92-a335-9e2296538f10\") " pod="ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6"
Jul 14 14:09:22.599110 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:22.599073    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c910658c-5c03-4f92-a335-9e2296538f10-model-cache\") pod \"auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6\" (UID: \"c910658c-5c03-4f92-a335-9e2296538f10\") " pod="ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6"
Jul 14 14:09:22.599166 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:22.599105    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxcpm\" (UniqueName: \"kubernetes.io/projected/c910658c-5c03-4f92-a335-9e2296538f10-kube-api-access-lxcpm\") pod \"auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6\" (UID: \"c910658c-5c03-4f92-a335-9e2296538f10\") " pod="ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6"
Jul 14 14:09:22.599227 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:22.599171    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c910658c-5c03-4f92-a335-9e2296538f10-tmp-dir\") pod \"auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6\" (UID: \"c910658c-5c03-4f92-a335-9e2296538f10\") " pod="ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6"
Jul 14 14:09:22.601548 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:22.601524    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6"]
Jul 14 14:09:22.699492 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:22.699464    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c910658c-5c03-4f92-a335-9e2296538f10-tmp-dir\") pod \"auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6\" (UID: \"c910658c-5c03-4f92-a335-9e2296538f10\") " pod="ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6"
Jul 14 14:09:22.699616 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:22.699507    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c910658c-5c03-4f92-a335-9e2296538f10-home\") pod \"auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6\" (UID: \"c910658c-5c03-4f92-a335-9e2296538f10\") " pod="ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6"
Jul 14 14:09:22.699616 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:22.699529    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c910658c-5c03-4f92-a335-9e2296538f10-dshm\") pod \"auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6\" (UID: \"c910658c-5c03-4f92-a335-9e2296538f10\") " pod="ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6"
Jul 14 14:09:22.699616 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:22.699604    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c910658c-5c03-4f92-a335-9e2296538f10-tls-certs\") pod \"auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6\" (UID: \"c910658c-5c03-4f92-a335-9e2296538f10\") " pod="ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6"
Jul 14 14:09:22.699776 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:22.699634    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c910658c-5c03-4f92-a335-9e2296538f10-model-cache\") pod \"auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6\" (UID: \"c910658c-5c03-4f92-a335-9e2296538f10\") " pod="ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6"
Jul 14 14:09:22.699776 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:22.699656    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxcpm\" (UniqueName: \"kubernetes.io/projected/c910658c-5c03-4f92-a335-9e2296538f10-kube-api-access-lxcpm\") pod \"auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6\" (UID: \"c910658c-5c03-4f92-a335-9e2296538f10\") " pod="ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6"
Jul 14 14:09:22.699932 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:22.699910    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c910658c-5c03-4f92-a335-9e2296538f10-home\") pod \"auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6\" (UID: \"c910658c-5c03-4f92-a335-9e2296538f10\") " pod="ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6"
Jul 14 14:09:22.700086 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:22.700067    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c910658c-5c03-4f92-a335-9e2296538f10-tmp-dir\") pod \"auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6\" (UID: \"c910658c-5c03-4f92-a335-9e2296538f10\") " pod="ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6"
Jul 14 14:09:22.700152 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:22.700113    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c910658c-5c03-4f92-a335-9e2296538f10-model-cache\") pod \"auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6\" (UID: \"c910658c-5c03-4f92-a335-9e2296538f10\") " pod="ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6"
Jul 14 14:09:22.701870 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:22.701851    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c910658c-5c03-4f92-a335-9e2296538f10-dshm\") pod \"auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6\" (UID: \"c910658c-5c03-4f92-a335-9e2296538f10\") " pod="ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6"
Jul 14 14:09:22.702276 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:22.702250    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c910658c-5c03-4f92-a335-9e2296538f10-tls-certs\") pod \"auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6\" (UID: \"c910658c-5c03-4f92-a335-9e2296538f10\") " pod="ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6"
Jul 14 14:09:22.709187 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:22.709163    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxcpm\" (UniqueName: \"kubernetes.io/projected/c910658c-5c03-4f92-a335-9e2296538f10-kube-api-access-lxcpm\") pod \"auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6\" (UID: \"c910658c-5c03-4f92-a335-9e2296538f10\") " pod="ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6"
Jul 14 14:09:22.903678 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:22.903612    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6"
Jul 14 14:09:23.037161 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:23.037138    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6"]
Jul 14 14:09:23.038574 ip-10-0-140-78 kubenswrapper[2569]: W0714 14:09:23.038551    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc910658c_5c03_4f92_a335_9e2296538f10.slice/crio-65dce91bf08863c102cbdb3d44b5608f860e43716b95ddc0e369bcef9ccd0ebc WatchSource:0}: Error finding container 65dce91bf08863c102cbdb3d44b5608f860e43716b95ddc0e369bcef9ccd0ebc: Status 404 returned error can't find the container with id 65dce91bf08863c102cbdb3d44b5608f860e43716b95ddc0e369bcef9ccd0ebc
Jul 14 14:09:23.460596 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:23.460560    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6" event={"ID":"c910658c-5c03-4f92-a335-9e2296538f10","Type":"ContainerStarted","Data":"ac34105fe72eb96a5e9a94bc17b6a3b3e484df0b8b0b8c8cf048a0280c4223b4"}
Jul 14 14:09:23.460788 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:23.460604    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6" event={"ID":"c910658c-5c03-4f92-a335-9e2296538f10","Type":"ContainerStarted","Data":"65dce91bf08863c102cbdb3d44b5608f860e43716b95ddc0e369bcef9ccd0ebc"}
Jul 14 14:09:23.481112 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:23.481056    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6" podStartSLOduration=1.481041126 podStartE2EDuration="1.481041126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:09:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:09:23.479506117 +0000 UTC m=+2528.805826045" watchObservedRunningTime="2026-07-14 14:09:23.481041126 +0000 UTC m=+2528.807361055"
Jul 14 14:09:32.904186 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:32.904149    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6"
Jul 14 14:09:32.904653 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:32.904198    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6"
Jul 14 14:09:32.916780 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:32.916757    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6"
Jul 14 14:09:33.525174 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:09:33.525142    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6"
Jul 14 14:11:18.425647 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:18.425616    2569 scope.go:117] "RemoveContainer" containerID="eea973b145793d7afe9867d75f3f45c991cd653d9708d35400d4dfe140ddab25"
Jul 14 14:11:28.786115 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:28.786079    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq"]
Jul 14 14:11:29.706808 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:29.706778    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6"]
Jul 14 14:11:43.806338 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:43.806296    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq" podUID="03f432c1-7a05-4eed-a428-3f15d6a9e3c0" containerName="main" containerID="cri-o://e35ee255ee07cc0173af54fd5a195070ffcefd19616e86eabd67931503acd403" gracePeriod=45
Jul 14 14:11:44.076308 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:44.076244    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq"
Jul 14 14:11:44.203836 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:44.203810    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzbwl\" (UniqueName: \"kubernetes.io/projected/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-kube-api-access-kzbwl\") pod \"03f432c1-7a05-4eed-a428-3f15d6a9e3c0\" (UID: \"03f432c1-7a05-4eed-a428-3f15d6a9e3c0\") "
Jul 14 14:11:44.204054 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:44.203873    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-home\") pod \"03f432c1-7a05-4eed-a428-3f15d6a9e3c0\" (UID: \"03f432c1-7a05-4eed-a428-3f15d6a9e3c0\") "
Jul 14 14:11:44.204054 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:44.203901    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-model-cache\") pod \"03f432c1-7a05-4eed-a428-3f15d6a9e3c0\" (UID: \"03f432c1-7a05-4eed-a428-3f15d6a9e3c0\") "
Jul 14 14:11:44.204054 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:44.203957    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-tls-certs\") pod \"03f432c1-7a05-4eed-a428-3f15d6a9e3c0\" (UID: \"03f432c1-7a05-4eed-a428-3f15d6a9e3c0\") "
Jul 14 14:11:44.204054 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:44.203990    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-dshm\") pod \"03f432c1-7a05-4eed-a428-3f15d6a9e3c0\" (UID: \"03f432c1-7a05-4eed-a428-3f15d6a9e3c0\") "
Jul 14 14:11:44.204054 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:44.204038    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-tmp-dir\") pod \"03f432c1-7a05-4eed-a428-3f15d6a9e3c0\" (UID: \"03f432c1-7a05-4eed-a428-3f15d6a9e3c0\") "
Jul 14 14:11:44.204321 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:44.204149    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-home" (OuterVolumeSpecName: "home") pod "03f432c1-7a05-4eed-a428-3f15d6a9e3c0" (UID: "03f432c1-7a05-4eed-a428-3f15d6a9e3c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:11:44.204321 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:44.204283    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-home\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:11:44.204402 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:44.204335    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "03f432c1-7a05-4eed-a428-3f15d6a9e3c0" (UID: "03f432c1-7a05-4eed-a428-3f15d6a9e3c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:11:44.204602 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:44.204559    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-model-cache" (OuterVolumeSpecName: "model-cache") pod "03f432c1-7a05-4eed-a428-3f15d6a9e3c0" (UID: "03f432c1-7a05-4eed-a428-3f15d6a9e3c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:11:44.206616 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:44.206589    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-dshm" (OuterVolumeSpecName: "dshm") pod "03f432c1-7a05-4eed-a428-3f15d6a9e3c0" (UID: "03f432c1-7a05-4eed-a428-3f15d6a9e3c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:11:44.206862 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:44.206817    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "03f432c1-7a05-4eed-a428-3f15d6a9e3c0" (UID: "03f432c1-7a05-4eed-a428-3f15d6a9e3c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 14:11:44.206862 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:44.206837    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-kube-api-access-kzbwl" (OuterVolumeSpecName: "kube-api-access-kzbwl") pod "03f432c1-7a05-4eed-a428-3f15d6a9e3c0" (UID: "03f432c1-7a05-4eed-a428-3f15d6a9e3c0"). InnerVolumeSpecName "kube-api-access-kzbwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 14:11:44.208303 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:44.208274    2569 generic.go:356] "Generic (PLEG): container finished" podID="03f432c1-7a05-4eed-a428-3f15d6a9e3c0" containerID="e35ee255ee07cc0173af54fd5a195070ffcefd19616e86eabd67931503acd403" exitCode=0
Jul 14 14:11:44.208412 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:44.208329    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq"
Jul 14 14:11:44.208412 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:44.208350    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq" event={"ID":"03f432c1-7a05-4eed-a428-3f15d6a9e3c0","Type":"ContainerDied","Data":"e35ee255ee07cc0173af54fd5a195070ffcefd19616e86eabd67931503acd403"}
Jul 14 14:11:44.208412 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:44.208391    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq" event={"ID":"03f432c1-7a05-4eed-a428-3f15d6a9e3c0","Type":"ContainerDied","Data":"26d43e88c06df9ae31b7210b6fa3a85c7decaee4bc1dedd4f234fbf3acbfe127"}
Jul 14 14:11:44.208412 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:44.208408    2569 scope.go:117] "RemoveContainer" containerID="e35ee255ee07cc0173af54fd5a195070ffcefd19616e86eabd67931503acd403"
Jul 14 14:11:44.225967 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:44.225946    2569 scope.go:117] "RemoveContainer" containerID="e35ee255ee07cc0173af54fd5a195070ffcefd19616e86eabd67931503acd403"
Jul 14 14:11:44.226299 ip-10-0-140-78 kubenswrapper[2569]: E0714 14:11:44.226279    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e35ee255ee07cc0173af54fd5a195070ffcefd19616e86eabd67931503acd403\": container with ID starting with e35ee255ee07cc0173af54fd5a195070ffcefd19616e86eabd67931503acd403 not found: ID does not exist" containerID="e35ee255ee07cc0173af54fd5a195070ffcefd19616e86eabd67931503acd403"
Jul 14 14:11:44.226374 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:44.226307    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e35ee255ee07cc0173af54fd5a195070ffcefd19616e86eabd67931503acd403"} err="failed to get container status \"e35ee255ee07cc0173af54fd5a195070ffcefd19616e86eabd67931503acd403\": rpc error: code = NotFound desc = could not find container \"e35ee255ee07cc0173af54fd5a195070ffcefd19616e86eabd67931503acd403\": container with ID starting with e35ee255ee07cc0173af54fd5a195070ffcefd19616e86eabd67931503acd403 not found: ID does not exist"
Jul 14 14:11:44.238723 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:44.238700    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq"]
Jul 14 14:11:44.246092 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:44.246070    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-9c0e14/auth-test-model-9c0e14-kserve-54d458bff5-5spxq"]
Jul 14 14:11:44.305433 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:44.305397    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-tls-certs\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:11:44.305433 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:44.305430    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-dshm\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:11:44.305694 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:44.305444    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-tmp-dir\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:11:44.305694 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:44.305457    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzbwl\" (UniqueName: \"kubernetes.io/projected/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-kube-api-access-kzbwl\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:11:44.305694 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:44.305470    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03f432c1-7a05-4eed-a428-3f15d6a9e3c0-model-cache\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:11:44.727244 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:44.727202    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6" podUID="c910658c-5c03-4f92-a335-9e2296538f10" containerName="main" containerID="cri-o://ac34105fe72eb96a5e9a94bc17b6a3b3e484df0b8b0b8c8cf048a0280c4223b4" gracePeriod=45
Jul 14 14:11:44.973137 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:44.973112    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6"
Jul 14 14:11:45.011742 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:45.011715    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c910658c-5c03-4f92-a335-9e2296538f10-home\") pod \"c910658c-5c03-4f92-a335-9e2296538f10\" (UID: \"c910658c-5c03-4f92-a335-9e2296538f10\") "
Jul 14 14:11:45.011929 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:45.011762    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxcpm\" (UniqueName: \"kubernetes.io/projected/c910658c-5c03-4f92-a335-9e2296538f10-kube-api-access-lxcpm\") pod \"c910658c-5c03-4f92-a335-9e2296538f10\" (UID: \"c910658c-5c03-4f92-a335-9e2296538f10\") "
Jul 14 14:11:45.011929 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:45.011843    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c910658c-5c03-4f92-a335-9e2296538f10-dshm\") pod \"c910658c-5c03-4f92-a335-9e2296538f10\" (UID: \"c910658c-5c03-4f92-a335-9e2296538f10\") "
Jul 14 14:11:45.011929 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:45.011866    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c910658c-5c03-4f92-a335-9e2296538f10-model-cache\") pod \"c910658c-5c03-4f92-a335-9e2296538f10\" (UID: \"c910658c-5c03-4f92-a335-9e2296538f10\") "
Jul 14 14:11:45.011929 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:45.011924    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c910658c-5c03-4f92-a335-9e2296538f10-tmp-dir\") pod \"c910658c-5c03-4f92-a335-9e2296538f10\" (UID: \"c910658c-5c03-4f92-a335-9e2296538f10\") "
Jul 14 14:11:45.012155 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:45.011949    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c910658c-5c03-4f92-a335-9e2296538f10-tls-certs\") pod \"c910658c-5c03-4f92-a335-9e2296538f10\" (UID: \"c910658c-5c03-4f92-a335-9e2296538f10\") "
Jul 14 14:11:45.012155 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:45.012055    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c910658c-5c03-4f92-a335-9e2296538f10-home" (OuterVolumeSpecName: "home") pod "c910658c-5c03-4f92-a335-9e2296538f10" (UID: "c910658c-5c03-4f92-a335-9e2296538f10"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:11:45.012155 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:45.012141    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c910658c-5c03-4f92-a335-9e2296538f10-model-cache" (OuterVolumeSpecName: "model-cache") pod "c910658c-5c03-4f92-a335-9e2296538f10" (UID: "c910658c-5c03-4f92-a335-9e2296538f10"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:11:45.012324 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:45.012184    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c910658c-5c03-4f92-a335-9e2296538f10-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c910658c-5c03-4f92-a335-9e2296538f10" (UID: "c910658c-5c03-4f92-a335-9e2296538f10"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:11:45.012324 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:45.012302    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c910658c-5c03-4f92-a335-9e2296538f10-model-cache\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:11:45.012324 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:45.012318    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c910658c-5c03-4f92-a335-9e2296538f10-tmp-dir\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:11:45.012489 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:45.012331    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c910658c-5c03-4f92-a335-9e2296538f10-home\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:11:45.014182 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:45.014152    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c910658c-5c03-4f92-a335-9e2296538f10-dshm" (OuterVolumeSpecName: "dshm") pod "c910658c-5c03-4f92-a335-9e2296538f10" (UID: "c910658c-5c03-4f92-a335-9e2296538f10"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:11:45.014182 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:45.014166    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c910658c-5c03-4f92-a335-9e2296538f10-kube-api-access-lxcpm" (OuterVolumeSpecName: "kube-api-access-lxcpm") pod "c910658c-5c03-4f92-a335-9e2296538f10" (UID: "c910658c-5c03-4f92-a335-9e2296538f10"). InnerVolumeSpecName "kube-api-access-lxcpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 14:11:45.014334 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:45.014305    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c910658c-5c03-4f92-a335-9e2296538f10-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c910658c-5c03-4f92-a335-9e2296538f10" (UID: "c910658c-5c03-4f92-a335-9e2296538f10"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 14:11:45.113226 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:45.113170    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c910658c-5c03-4f92-a335-9e2296538f10-tls-certs\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:11:45.113226 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:45.113217    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxcpm\" (UniqueName: \"kubernetes.io/projected/c910658c-5c03-4f92-a335-9e2296538f10-kube-api-access-lxcpm\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:11:45.113226 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:45.113226    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c910658c-5c03-4f92-a335-9e2296538f10-dshm\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:11:45.193191 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:45.193155    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03f432c1-7a05-4eed-a428-3f15d6a9e3c0" path="/var/lib/kubelet/pods/03f432c1-7a05-4eed-a428-3f15d6a9e3c0/volumes"
Jul 14 14:11:45.220721 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:45.220692    2569 generic.go:356] "Generic (PLEG): container finished" podID="c910658c-5c03-4f92-a335-9e2296538f10" containerID="ac34105fe72eb96a5e9a94bc17b6a3b3e484df0b8b0b8c8cf048a0280c4223b4" exitCode=0
Jul 14 14:11:45.220882 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:45.220743    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6"
Jul 14 14:11:45.220882 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:45.220783    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6" event={"ID":"c910658c-5c03-4f92-a335-9e2296538f10","Type":"ContainerDied","Data":"ac34105fe72eb96a5e9a94bc17b6a3b3e484df0b8b0b8c8cf048a0280c4223b4"}
Jul 14 14:11:45.220882 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:45.220824    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6" event={"ID":"c910658c-5c03-4f92-a335-9e2296538f10","Type":"ContainerDied","Data":"65dce91bf08863c102cbdb3d44b5608f860e43716b95ddc0e369bcef9ccd0ebc"}
Jul 14 14:11:45.220882 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:45.220846    2569 scope.go:117] "RemoveContainer" containerID="ac34105fe72eb96a5e9a94bc17b6a3b3e484df0b8b0b8c8cf048a0280c4223b4"
Jul 14 14:11:45.229539 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:45.229521    2569 scope.go:117] "RemoveContainer" containerID="ac34105fe72eb96a5e9a94bc17b6a3b3e484df0b8b0b8c8cf048a0280c4223b4"
Jul 14 14:11:45.229778 ip-10-0-140-78 kubenswrapper[2569]: E0714 14:11:45.229761    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac34105fe72eb96a5e9a94bc17b6a3b3e484df0b8b0b8c8cf048a0280c4223b4\": container with ID starting with ac34105fe72eb96a5e9a94bc17b6a3b3e484df0b8b0b8c8cf048a0280c4223b4 not found: ID does not exist" containerID="ac34105fe72eb96a5e9a94bc17b6a3b3e484df0b8b0b8c8cf048a0280c4223b4"
Jul 14 14:11:45.229824 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:45.229786    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac34105fe72eb96a5e9a94bc17b6a3b3e484df0b8b0b8c8cf048a0280c4223b4"} err="failed to get container status \"ac34105fe72eb96a5e9a94bc17b6a3b3e484df0b8b0b8c8cf048a0280c4223b4\": rpc error: code = NotFound desc = could not find container \"ac34105fe72eb96a5e9a94bc17b6a3b3e484df0b8b0b8c8cf048a0280c4223b4\": container with ID starting with ac34105fe72eb96a5e9a94bc17b6a3b3e484df0b8b0b8c8cf048a0280c4223b4 not found: ID does not exist"
Jul 14 14:11:45.240392 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:45.240370    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6"]
Jul 14 14:11:45.245132 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:45.245111    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-ff9f03/auth-test-model-ff9f03-kserve-799c5bcd67-9tkv6"]
Jul 14 14:11:47.193507 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:11:47.193469    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c910658c-5c03-4f92-a335-9e2296538f10" path="/var/lib/kubelet/pods/c910658c-5c03-4f92-a335-9e2296538f10/volumes"
Jul 14 14:12:00.983782 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:00.983737    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq"]
Jul 14 14:12:00.984326 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:00.984307    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c910658c-5c03-4f92-a335-9e2296538f10" containerName="main"
Jul 14 14:12:00.984375 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:00.984332    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c910658c-5c03-4f92-a335-9e2296538f10" containerName="main"
Jul 14 14:12:00.984375 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:00.984354    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03f432c1-7a05-4eed-a428-3f15d6a9e3c0" containerName="main"
Jul 14 14:12:00.984375 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:00.984362    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="03f432c1-7a05-4eed-a428-3f15d6a9e3c0" containerName="main"
Jul 14 14:12:00.984468 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:00.984445    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="03f432c1-7a05-4eed-a428-3f15d6a9e3c0" containerName="main"
Jul 14 14:12:00.984468 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:00.984459    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c910658c-5c03-4f92-a335-9e2296538f10" containerName="main"
Jul 14 14:12:00.987952 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:00.987926    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq"
Jul 14 14:12:00.992161 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:00.992128    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ff9f03\"/\"kube-root-ca.crt\""
Jul 14 14:12:00.992295 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:00.992169    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ff9f03\"/\"sub-test-model-ff9f03-kserve-self-signed-certs\""
Jul 14 14:12:00.992295 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:00.992232    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ff9f03\"/\"openshift-service-ca.crt\""
Jul 14 14:12:00.992415 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:00.992298    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ff9f03\"/\"default-dockercfg-pvc6j\""
Jul 14 14:12:00.996475 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:00.996448    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq"]
Jul 14 14:12:01.055247 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:01.055213    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63edc6a5-37eb-4456-8ac4-caaa2c255501-model-cache\") pod \"sub-test-model-ff9f03-kserve-d5d566c47-4tggq\" (UID: \"63edc6a5-37eb-4456-8ac4-caaa2c255501\") " pod="ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq"
Jul 14 14:12:01.055445 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:01.055266    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63edc6a5-37eb-4456-8ac4-caaa2c255501-dshm\") pod \"sub-test-model-ff9f03-kserve-d5d566c47-4tggq\" (UID: \"63edc6a5-37eb-4456-8ac4-caaa2c255501\") " pod="ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq"
Jul 14 14:12:01.055445 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:01.055293    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnd7m\" (UniqueName: \"kubernetes.io/projected/63edc6a5-37eb-4456-8ac4-caaa2c255501-kube-api-access-bnd7m\") pod \"sub-test-model-ff9f03-kserve-d5d566c47-4tggq\" (UID: \"63edc6a5-37eb-4456-8ac4-caaa2c255501\") " pod="ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq"
Jul 14 14:12:01.055445 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:01.055324    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63edc6a5-37eb-4456-8ac4-caaa2c255501-tmp-dir\") pod \"sub-test-model-ff9f03-kserve-d5d566c47-4tggq\" (UID: \"63edc6a5-37eb-4456-8ac4-caaa2c255501\") " pod="ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq"
Jul 14 14:12:01.055445 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:01.055350    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63edc6a5-37eb-4456-8ac4-caaa2c255501-tls-certs\") pod \"sub-test-model-ff9f03-kserve-d5d566c47-4tggq\" (UID: \"63edc6a5-37eb-4456-8ac4-caaa2c255501\") " pod="ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq"
Jul 14 14:12:01.055445 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:01.055422    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63edc6a5-37eb-4456-8ac4-caaa2c255501-home\") pod \"sub-test-model-ff9f03-kserve-d5d566c47-4tggq\" (UID: \"63edc6a5-37eb-4456-8ac4-caaa2c255501\") " pod="ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq"
Jul 14 14:12:01.155926 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:01.155896    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63edc6a5-37eb-4456-8ac4-caaa2c255501-model-cache\") pod \"sub-test-model-ff9f03-kserve-d5d566c47-4tggq\" (UID: \"63edc6a5-37eb-4456-8ac4-caaa2c255501\") " pod="ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq"
Jul 14 14:12:01.156160 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:01.155948    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63edc6a5-37eb-4456-8ac4-caaa2c255501-dshm\") pod \"sub-test-model-ff9f03-kserve-d5d566c47-4tggq\" (UID: \"63edc6a5-37eb-4456-8ac4-caaa2c255501\") " pod="ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq"
Jul 14 14:12:01.156160 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:01.155976    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnd7m\" (UniqueName: \"kubernetes.io/projected/63edc6a5-37eb-4456-8ac4-caaa2c255501-kube-api-access-bnd7m\") pod \"sub-test-model-ff9f03-kserve-d5d566c47-4tggq\" (UID: \"63edc6a5-37eb-4456-8ac4-caaa2c255501\") " pod="ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq"
Jul 14 14:12:01.156160 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:01.156003    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63edc6a5-37eb-4456-8ac4-caaa2c255501-tmp-dir\") pod \"sub-test-model-ff9f03-kserve-d5d566c47-4tggq\" (UID: \"63edc6a5-37eb-4456-8ac4-caaa2c255501\") " pod="ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq"
Jul 14 14:12:01.156160 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:01.156051    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63edc6a5-37eb-4456-8ac4-caaa2c255501-tls-certs\") pod \"sub-test-model-ff9f03-kserve-d5d566c47-4tggq\" (UID: \"63edc6a5-37eb-4456-8ac4-caaa2c255501\") " pod="ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq"
Jul 14 14:12:01.156160 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:01.156098    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63edc6a5-37eb-4456-8ac4-caaa2c255501-home\") pod \"sub-test-model-ff9f03-kserve-d5d566c47-4tggq\" (UID: \"63edc6a5-37eb-4456-8ac4-caaa2c255501\") " pod="ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq"
Jul 14 14:12:01.156530 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:01.156505    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63edc6a5-37eb-4456-8ac4-caaa2c255501-tmp-dir\") pod \"sub-test-model-ff9f03-kserve-d5d566c47-4tggq\" (UID: \"63edc6a5-37eb-4456-8ac4-caaa2c255501\") " pod="ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq"
Jul 14 14:12:01.156602 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:01.156541    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63edc6a5-37eb-4456-8ac4-caaa2c255501-home\") pod \"sub-test-model-ff9f03-kserve-d5d566c47-4tggq\" (UID: \"63edc6a5-37eb-4456-8ac4-caaa2c255501\") " pod="ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq"
Jul 14 14:12:01.156602 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:01.156511    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63edc6a5-37eb-4456-8ac4-caaa2c255501-model-cache\") pod \"sub-test-model-ff9f03-kserve-d5d566c47-4tggq\" (UID: \"63edc6a5-37eb-4456-8ac4-caaa2c255501\") " pod="ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq"
Jul 14 14:12:01.158253 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:01.158236    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63edc6a5-37eb-4456-8ac4-caaa2c255501-dshm\") pod \"sub-test-model-ff9f03-kserve-d5d566c47-4tggq\" (UID: \"63edc6a5-37eb-4456-8ac4-caaa2c255501\") " pod="ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq"
Jul 14 14:12:01.158508 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:01.158491    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63edc6a5-37eb-4456-8ac4-caaa2c255501-tls-certs\") pod \"sub-test-model-ff9f03-kserve-d5d566c47-4tggq\" (UID: \"63edc6a5-37eb-4456-8ac4-caaa2c255501\") " pod="ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq"
Jul 14 14:12:01.167257 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:01.167238    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnd7m\" (UniqueName: \"kubernetes.io/projected/63edc6a5-37eb-4456-8ac4-caaa2c255501-kube-api-access-bnd7m\") pod \"sub-test-model-ff9f03-kserve-d5d566c47-4tggq\" (UID: \"63edc6a5-37eb-4456-8ac4-caaa2c255501\") " pod="ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq"
Jul 14 14:12:01.299154 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:01.299076    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq"
Jul 14 14:12:01.436726 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:01.436700    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq"]
Jul 14 14:12:01.437862 ip-10-0-140-78 kubenswrapper[2569]: W0714 14:12:01.437836    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63edc6a5_37eb_4456_8ac4_caaa2c255501.slice/crio-f5e0b49cafe262fc4bf1e41e2851fe8085d937ba522fa1fa19c59d7f2b16a72d WatchSource:0}: Error finding container f5e0b49cafe262fc4bf1e41e2851fe8085d937ba522fa1fa19c59d7f2b16a72d: Status 404 returned error can't find the container with id f5e0b49cafe262fc4bf1e41e2851fe8085d937ba522fa1fa19c59d7f2b16a72d
Jul 14 14:12:02.310745 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:02.310710    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq" event={"ID":"63edc6a5-37eb-4456-8ac4-caaa2c255501","Type":"ContainerStarted","Data":"08e35bc318b2c2d5fd819163b677e613f42ce7ef97f0b175e3867b5a7f83e3e8"}
Jul 14 14:12:02.311256 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:02.310752    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq" event={"ID":"63edc6a5-37eb-4456-8ac4-caaa2c255501","Type":"ContainerStarted","Data":"f5e0b49cafe262fc4bf1e41e2851fe8085d937ba522fa1fa19c59d7f2b16a72d"}
Jul 14 14:12:02.330120 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:02.330070    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq" podStartSLOduration=2.330055585 podStartE2EDuration="2.330055585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:12:02.328347733 +0000 UTC m=+2687.654667662" watchObservedRunningTime="2026-07-14 14:12:02.330055585 +0000 UTC m=+2687.656375566"
Jul 14 14:12:11.299995 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:11.299961    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq"
Jul 14 14:12:11.300543 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:11.300229    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq"
Jul 14 14:12:11.312633 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:11.312606    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq"
Jul 14 14:12:11.367259 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:11.367229    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq"
Jul 14 14:12:15.302895 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:15.302782    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vq8tk_e33a489f-7d68-4e6c-9ffe-e7252049e15d/ovn-acl-logging/0.log"
Jul 14 14:12:15.311730 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:15.311707    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vq8tk_e33a489f-7d68-4e6c-9ffe-e7252049e15d/ovn-acl-logging/0.log"
Jul 14 14:12:38.597026 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:38.596972    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq"]
Jul 14 14:12:51.592570 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:51.592532    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj"]
Jul 14 14:12:51.596938 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:51.596914    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj"
Jul 14 14:12:51.607414 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:51.607392    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9c0e14\"/\"kube-root-ca.crt\""
Jul 14 14:12:51.607414 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:51.607400    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9c0e14\"/\"rate-test-model-9c0e14-kserve-self-signed-certs\""
Jul 14 14:12:51.607608 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:51.607392    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9c0e14\"/\"default-dockercfg-6lvpd\""
Jul 14 14:12:51.607608 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:51.607400    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9c0e14\"/\"openshift-service-ca.crt\""
Jul 14 14:12:51.609658 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:51.609637    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj"]
Jul 14 14:12:51.693612 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:51.693580    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3131c790-b6c7-467a-81dd-e5872757e92e-home\") pod \"rate-test-model-9c0e14-kserve-5657d875f5-knxrj\" (UID: \"3131c790-b6c7-467a-81dd-e5872757e92e\") " pod="ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj"
Jul 14 14:12:51.693612 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:51.693617    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3131c790-b6c7-467a-81dd-e5872757e92e-dshm\") pod \"rate-test-model-9c0e14-kserve-5657d875f5-knxrj\" (UID: \"3131c790-b6c7-467a-81dd-e5872757e92e\") " pod="ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj"
Jul 14 14:12:51.693830 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:51.693711    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ks8x\" (UniqueName: \"kubernetes.io/projected/3131c790-b6c7-467a-81dd-e5872757e92e-kube-api-access-6ks8x\") pod \"rate-test-model-9c0e14-kserve-5657d875f5-knxrj\" (UID: \"3131c790-b6c7-467a-81dd-e5872757e92e\") " pod="ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj"
Jul 14 14:12:51.693830 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:51.693742    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3131c790-b6c7-467a-81dd-e5872757e92e-tmp-dir\") pod \"rate-test-model-9c0e14-kserve-5657d875f5-knxrj\" (UID: \"3131c790-b6c7-467a-81dd-e5872757e92e\") " pod="ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj"
Jul 14 14:12:51.693830 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:51.693813    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3131c790-b6c7-467a-81dd-e5872757e92e-model-cache\") pod \"rate-test-model-9c0e14-kserve-5657d875f5-knxrj\" (UID: \"3131c790-b6c7-467a-81dd-e5872757e92e\") " pod="ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj"
Jul 14 14:12:51.693929 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:51.693873    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3131c790-b6c7-467a-81dd-e5872757e92e-tls-certs\") pod \"rate-test-model-9c0e14-kserve-5657d875f5-knxrj\" (UID: \"3131c790-b6c7-467a-81dd-e5872757e92e\") " pod="ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj"
Jul 14 14:12:51.794363 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:51.794316    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3131c790-b6c7-467a-81dd-e5872757e92e-model-cache\") pod \"rate-test-model-9c0e14-kserve-5657d875f5-knxrj\" (UID: \"3131c790-b6c7-467a-81dd-e5872757e92e\") " pod="ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj"
Jul 14 14:12:51.794581 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:51.794399    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3131c790-b6c7-467a-81dd-e5872757e92e-tls-certs\") pod \"rate-test-model-9c0e14-kserve-5657d875f5-knxrj\" (UID: \"3131c790-b6c7-467a-81dd-e5872757e92e\") " pod="ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj"
Jul 14 14:12:51.794581 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:51.794438    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3131c790-b6c7-467a-81dd-e5872757e92e-home\") pod \"rate-test-model-9c0e14-kserve-5657d875f5-knxrj\" (UID: \"3131c790-b6c7-467a-81dd-e5872757e92e\") " pod="ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj"
Jul 14 14:12:51.794581 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:51.794468    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3131c790-b6c7-467a-81dd-e5872757e92e-dshm\") pod \"rate-test-model-9c0e14-kserve-5657d875f5-knxrj\" (UID: \"3131c790-b6c7-467a-81dd-e5872757e92e\") " pod="ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj"
Jul 14 14:12:51.794581 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:51.794544    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ks8x\" (UniqueName: \"kubernetes.io/projected/3131c790-b6c7-467a-81dd-e5872757e92e-kube-api-access-6ks8x\") pod \"rate-test-model-9c0e14-kserve-5657d875f5-knxrj\" (UID: \"3131c790-b6c7-467a-81dd-e5872757e92e\") " pod="ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj"
Jul 14 14:12:51.794581 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:51.794579    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3131c790-b6c7-467a-81dd-e5872757e92e-tmp-dir\") pod \"rate-test-model-9c0e14-kserve-5657d875f5-knxrj\" (UID: \"3131c790-b6c7-467a-81dd-e5872757e92e\") " pod="ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj"
Jul 14 14:12:51.794879 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:51.794763    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3131c790-b6c7-467a-81dd-e5872757e92e-model-cache\") pod \"rate-test-model-9c0e14-kserve-5657d875f5-knxrj\" (UID: \"3131c790-b6c7-467a-81dd-e5872757e92e\") " pod="ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj"
Jul 14 14:12:51.794879 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:51.794805    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3131c790-b6c7-467a-81dd-e5872757e92e-home\") pod \"rate-test-model-9c0e14-kserve-5657d875f5-knxrj\" (UID: \"3131c790-b6c7-467a-81dd-e5872757e92e\") " pod="ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj"
Jul 14 14:12:51.794994 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:51.794879    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3131c790-b6c7-467a-81dd-e5872757e92e-tmp-dir\") pod \"rate-test-model-9c0e14-kserve-5657d875f5-knxrj\" (UID: \"3131c790-b6c7-467a-81dd-e5872757e92e\") " pod="ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj"
Jul 14 14:12:51.796868 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:51.796843    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3131c790-b6c7-467a-81dd-e5872757e92e-dshm\") pod \"rate-test-model-9c0e14-kserve-5657d875f5-knxrj\" (UID: \"3131c790-b6c7-467a-81dd-e5872757e92e\") " pod="ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj"
Jul 14 14:12:51.797151 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:51.797135    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3131c790-b6c7-467a-81dd-e5872757e92e-tls-certs\") pod \"rate-test-model-9c0e14-kserve-5657d875f5-knxrj\" (UID: \"3131c790-b6c7-467a-81dd-e5872757e92e\") " pod="ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj"
Jul 14 14:12:51.804526 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:51.804506    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ks8x\" (UniqueName: \"kubernetes.io/projected/3131c790-b6c7-467a-81dd-e5872757e92e-kube-api-access-6ks8x\") pod \"rate-test-model-9c0e14-kserve-5657d875f5-knxrj\" (UID: \"3131c790-b6c7-467a-81dd-e5872757e92e\") " pod="ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj"
Jul 14 14:12:51.906172 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:51.906081    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj"
Jul 14 14:12:52.039605 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:52.039525    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj"]
Jul 14 14:12:52.041330 ip-10-0-140-78 kubenswrapper[2569]: W0714 14:12:52.041299    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3131c790_b6c7_467a_81dd_e5872757e92e.slice/crio-f5c8fddc22fe976065ea79e70ab9f092b60929fc519d1718988c53ffd3a12a03 WatchSource:0}: Error finding container f5c8fddc22fe976065ea79e70ab9f092b60929fc519d1718988c53ffd3a12a03: Status 404 returned error can't find the container with id f5c8fddc22fe976065ea79e70ab9f092b60929fc519d1718988c53ffd3a12a03
Jul 14 14:12:52.567751 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:52.567713    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj" event={"ID":"3131c790-b6c7-467a-81dd-e5872757e92e","Type":"ContainerStarted","Data":"d4bb63a9bc67130062272592b5af8b32a1d4aa6b3b1d87a3a26070cb4891352d"}
Jul 14 14:12:52.567751 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:52.567756    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj" event={"ID":"3131c790-b6c7-467a-81dd-e5872757e92e","Type":"ContainerStarted","Data":"f5c8fddc22fe976065ea79e70ab9f092b60929fc519d1718988c53ffd3a12a03"}
Jul 14 14:12:52.585965 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:52.585918    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj" podStartSLOduration=1.585901481 podStartE2EDuration="1.585901481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:12:52.584467919 +0000 UTC m=+2737.910787848" watchObservedRunningTime="2026-07-14 14:12:52.585901481 +0000 UTC m=+2737.912221408"
Jul 14 14:12:53.616302 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:53.616266    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq" podUID="63edc6a5-37eb-4456-8ac4-caaa2c255501" containerName="main" containerID="cri-o://08e35bc318b2c2d5fd819163b677e613f42ce7ef97f0b175e3867b5a7f83e3e8" gracePeriod=45
Jul 14 14:12:53.865040 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:53.864993    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq"
Jul 14 14:12:54.017069 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:54.017038    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63edc6a5-37eb-4456-8ac4-caaa2c255501-tls-certs\") pod \"63edc6a5-37eb-4456-8ac4-caaa2c255501\" (UID: \"63edc6a5-37eb-4456-8ac4-caaa2c255501\") "
Jul 14 14:12:54.017233 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:54.017107    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63edc6a5-37eb-4456-8ac4-caaa2c255501-model-cache\") pod \"63edc6a5-37eb-4456-8ac4-caaa2c255501\" (UID: \"63edc6a5-37eb-4456-8ac4-caaa2c255501\") "
Jul 14 14:12:54.017233 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:54.017162    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63edc6a5-37eb-4456-8ac4-caaa2c255501-tmp-dir\") pod \"63edc6a5-37eb-4456-8ac4-caaa2c255501\" (UID: \"63edc6a5-37eb-4456-8ac4-caaa2c255501\") "
Jul 14 14:12:54.017233 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:54.017198    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63edc6a5-37eb-4456-8ac4-caaa2c255501-home\") pod \"63edc6a5-37eb-4456-8ac4-caaa2c255501\" (UID: \"63edc6a5-37eb-4456-8ac4-caaa2c255501\") "
Jul 14 14:12:54.017381 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:54.017241    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnd7m\" (UniqueName: \"kubernetes.io/projected/63edc6a5-37eb-4456-8ac4-caaa2c255501-kube-api-access-bnd7m\") pod \"63edc6a5-37eb-4456-8ac4-caaa2c255501\" (UID: \"63edc6a5-37eb-4456-8ac4-caaa2c255501\") "
Jul 14 14:12:54.017381 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:54.017260    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63edc6a5-37eb-4456-8ac4-caaa2c255501-dshm\") pod \"63edc6a5-37eb-4456-8ac4-caaa2c255501\" (UID: \"63edc6a5-37eb-4456-8ac4-caaa2c255501\") "
Jul 14 14:12:54.017489 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:54.017386    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63edc6a5-37eb-4456-8ac4-caaa2c255501-model-cache" (OuterVolumeSpecName: "model-cache") pod "63edc6a5-37eb-4456-8ac4-caaa2c255501" (UID: "63edc6a5-37eb-4456-8ac4-caaa2c255501"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:12:54.017489 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:54.017409    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63edc6a5-37eb-4456-8ac4-caaa2c255501-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "63edc6a5-37eb-4456-8ac4-caaa2c255501" (UID: "63edc6a5-37eb-4456-8ac4-caaa2c255501"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:12:54.017489 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:54.017440    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63edc6a5-37eb-4456-8ac4-caaa2c255501-home" (OuterVolumeSpecName: "home") pod "63edc6a5-37eb-4456-8ac4-caaa2c255501" (UID: "63edc6a5-37eb-4456-8ac4-caaa2c255501"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:12:54.017636 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:54.017530    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63edc6a5-37eb-4456-8ac4-caaa2c255501-tmp-dir\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:12:54.017636 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:54.017541    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63edc6a5-37eb-4456-8ac4-caaa2c255501-home\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:12:54.017636 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:54.017549    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63edc6a5-37eb-4456-8ac4-caaa2c255501-model-cache\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:12:54.019271 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:54.019251    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63edc6a5-37eb-4456-8ac4-caaa2c255501-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "63edc6a5-37eb-4456-8ac4-caaa2c255501" (UID: "63edc6a5-37eb-4456-8ac4-caaa2c255501"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 14:12:54.019346 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:54.019298    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63edc6a5-37eb-4456-8ac4-caaa2c255501-dshm" (OuterVolumeSpecName: "dshm") pod "63edc6a5-37eb-4456-8ac4-caaa2c255501" (UID: "63edc6a5-37eb-4456-8ac4-caaa2c255501"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:12:54.019436 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:54.019417    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63edc6a5-37eb-4456-8ac4-caaa2c255501-kube-api-access-bnd7m" (OuterVolumeSpecName: "kube-api-access-bnd7m") pod "63edc6a5-37eb-4456-8ac4-caaa2c255501" (UID: "63edc6a5-37eb-4456-8ac4-caaa2c255501"). InnerVolumeSpecName "kube-api-access-bnd7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 14:12:54.118034 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:54.117966    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63edc6a5-37eb-4456-8ac4-caaa2c255501-tls-certs\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:12:54.118034 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:54.118004    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnd7m\" (UniqueName: \"kubernetes.io/projected/63edc6a5-37eb-4456-8ac4-caaa2c255501-kube-api-access-bnd7m\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:12:54.118237 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:54.118046    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63edc6a5-37eb-4456-8ac4-caaa2c255501-dshm\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:12:54.578977 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:54.578943    2569 generic.go:356] "Generic (PLEG): container finished" podID="63edc6a5-37eb-4456-8ac4-caaa2c255501" containerID="08e35bc318b2c2d5fd819163b677e613f42ce7ef97f0b175e3867b5a7f83e3e8" exitCode=0
Jul 14 14:12:54.579199 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:54.578990    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq"
Jul 14 14:12:54.579199 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:54.579035    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq" event={"ID":"63edc6a5-37eb-4456-8ac4-caaa2c255501","Type":"ContainerDied","Data":"08e35bc318b2c2d5fd819163b677e613f42ce7ef97f0b175e3867b5a7f83e3e8"}
Jul 14 14:12:54.579199 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:54.579074    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq" event={"ID":"63edc6a5-37eb-4456-8ac4-caaa2c255501","Type":"ContainerDied","Data":"f5e0b49cafe262fc4bf1e41e2851fe8085d937ba522fa1fa19c59d7f2b16a72d"}
Jul 14 14:12:54.579199 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:54.579094    2569 scope.go:117] "RemoveContainer" containerID="08e35bc318b2c2d5fd819163b677e613f42ce7ef97f0b175e3867b5a7f83e3e8"
Jul 14 14:12:54.587910 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:54.587892    2569 scope.go:117] "RemoveContainer" containerID="08e35bc318b2c2d5fd819163b677e613f42ce7ef97f0b175e3867b5a7f83e3e8"
Jul 14 14:12:54.588176 ip-10-0-140-78 kubenswrapper[2569]: E0714 14:12:54.588158    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08e35bc318b2c2d5fd819163b677e613f42ce7ef97f0b175e3867b5a7f83e3e8\": container with ID starting with 08e35bc318b2c2d5fd819163b677e613f42ce7ef97f0b175e3867b5a7f83e3e8 not found: ID does not exist" containerID="08e35bc318b2c2d5fd819163b677e613f42ce7ef97f0b175e3867b5a7f83e3e8"
Jul 14 14:12:54.588223 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:54.588185    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08e35bc318b2c2d5fd819163b677e613f42ce7ef97f0b175e3867b5a7f83e3e8"} err="failed to get container status \"08e35bc318b2c2d5fd819163b677e613f42ce7ef97f0b175e3867b5a7f83e3e8\": rpc error: code = NotFound desc = could not find container \"08e35bc318b2c2d5fd819163b677e613f42ce7ef97f0b175e3867b5a7f83e3e8\": container with ID starting with 08e35bc318b2c2d5fd819163b677e613f42ce7ef97f0b175e3867b5a7f83e3e8 not found: ID does not exist"
Jul 14 14:12:54.601743 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:54.601717    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq"]
Jul 14 14:12:54.606060 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:54.606040    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-ff9f03/sub-test-model-ff9f03-kserve-d5d566c47-4tggq"]
Jul 14 14:12:55.193681 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:12:55.193640    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63edc6a5-37eb-4456-8ac4-caaa2c255501" path="/var/lib/kubelet/pods/63edc6a5-37eb-4456-8ac4-caaa2c255501/volumes"
Jul 14 14:13:01.906454 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:01.906414    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj"
Jul 14 14:13:01.906454 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:01.906466    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj"
Jul 14 14:13:01.919296 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:01.919269    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj"
Jul 14 14:13:02.630203 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:02.630174    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj"
Jul 14 14:13:09.394951 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:09.394922    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl"]
Jul 14 14:13:09.395371 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:09.395267    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63edc6a5-37eb-4456-8ac4-caaa2c255501" containerName="main"
Jul 14 14:13:09.395371 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:09.395278    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="63edc6a5-37eb-4456-8ac4-caaa2c255501" containerName="main"
Jul 14 14:13:09.395371 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:09.395354    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="63edc6a5-37eb-4456-8ac4-caaa2c255501" containerName="main"
Jul 14 14:13:09.399463 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:09.399447    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl"
Jul 14 14:13:09.402325 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:09.402296    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ff9f03\"/\"kube-root-ca.crt\""
Jul 14 14:13:09.403775 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:09.403752    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ff9f03\"/\"default-dockercfg-pvc6j\""
Jul 14 14:13:09.403895 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:09.403865    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ff9f03\"/\"rate-test-model-ff9f03-kserve-self-signed-certs\""
Jul 14 14:13:09.404044 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:09.404000    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ff9f03\"/\"openshift-service-ca.crt\""
Jul 14 14:13:09.409635 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:09.409615    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl"]
Jul 14 14:13:09.553241 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:09.553207    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65c92d83-cbd5-46d9-b356-00ae5c27c60d-dshm\") pod \"rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl\" (UID: \"65c92d83-cbd5-46d9-b356-00ae5c27c60d\") " pod="ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl"
Jul 14 14:13:09.553424 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:09.553261    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65c92d83-cbd5-46d9-b356-00ae5c27c60d-tls-certs\") pod \"rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl\" (UID: \"65c92d83-cbd5-46d9-b356-00ae5c27c60d\") " pod="ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl"
Jul 14 14:13:09.553424 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:09.553282    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65c92d83-cbd5-46d9-b356-00ae5c27c60d-home\") pod \"rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl\" (UID: \"65c92d83-cbd5-46d9-b356-00ae5c27c60d\") " pod="ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl"
Jul 14 14:13:09.553424 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:09.553336    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29t5m\" (UniqueName: \"kubernetes.io/projected/65c92d83-cbd5-46d9-b356-00ae5c27c60d-kube-api-access-29t5m\") pod \"rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl\" (UID: \"65c92d83-cbd5-46d9-b356-00ae5c27c60d\") " pod="ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl"
Jul 14 14:13:09.553424 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:09.553355    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65c92d83-cbd5-46d9-b356-00ae5c27c60d-tmp-dir\") pod \"rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl\" (UID: \"65c92d83-cbd5-46d9-b356-00ae5c27c60d\") " pod="ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl"
Jul 14 14:13:09.553424 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:09.553408    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65c92d83-cbd5-46d9-b356-00ae5c27c60d-model-cache\") pod \"rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl\" (UID: \"65c92d83-cbd5-46d9-b356-00ae5c27c60d\") " pod="ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl"
Jul 14 14:13:09.654800 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:09.654700    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65c92d83-cbd5-46d9-b356-00ae5c27c60d-tls-certs\") pod \"rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl\" (UID: \"65c92d83-cbd5-46d9-b356-00ae5c27c60d\") " pod="ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl"
Jul 14 14:13:09.654800 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:09.654739    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65c92d83-cbd5-46d9-b356-00ae5c27c60d-home\") pod \"rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl\" (UID: \"65c92d83-cbd5-46d9-b356-00ae5c27c60d\") " pod="ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl"
Jul 14 14:13:09.654800 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:09.654764    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29t5m\" (UniqueName: \"kubernetes.io/projected/65c92d83-cbd5-46d9-b356-00ae5c27c60d-kube-api-access-29t5m\") pod \"rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl\" (UID: \"65c92d83-cbd5-46d9-b356-00ae5c27c60d\") " pod="ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl"
Jul 14 14:13:09.654800 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:09.654786    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65c92d83-cbd5-46d9-b356-00ae5c27c60d-tmp-dir\") pod \"rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl\" (UID: \"65c92d83-cbd5-46d9-b356-00ae5c27c60d\") " pod="ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl"
Jul 14 14:13:09.655153 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:09.654812    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65c92d83-cbd5-46d9-b356-00ae5c27c60d-model-cache\") pod \"rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl\" (UID: \"65c92d83-cbd5-46d9-b356-00ae5c27c60d\") " pod="ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl"
Jul 14 14:13:09.655153 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:09.654905    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65c92d83-cbd5-46d9-b356-00ae5c27c60d-dshm\") pod \"rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl\" (UID: \"65c92d83-cbd5-46d9-b356-00ae5c27c60d\") " pod="ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl"
Jul 14 14:13:09.655233 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:09.655202    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65c92d83-cbd5-46d9-b356-00ae5c27c60d-home\") pod \"rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl\" (UID: \"65c92d83-cbd5-46d9-b356-00ae5c27c60d\") " pod="ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl"
Jul 14 14:13:09.655233 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:09.655222    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65c92d83-cbd5-46d9-b356-00ae5c27c60d-tmp-dir\") pod \"rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl\" (UID: \"65c92d83-cbd5-46d9-b356-00ae5c27c60d\") " pod="ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl"
Jul 14 14:13:09.655335 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:09.655309    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65c92d83-cbd5-46d9-b356-00ae5c27c60d-model-cache\") pod \"rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl\" (UID: \"65c92d83-cbd5-46d9-b356-00ae5c27c60d\") " pod="ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl"
Jul 14 14:13:09.657171 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:09.657151    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65c92d83-cbd5-46d9-b356-00ae5c27c60d-dshm\") pod \"rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl\" (UID: \"65c92d83-cbd5-46d9-b356-00ae5c27c60d\") " pod="ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl"
Jul 14 14:13:09.657304 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:09.657286    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65c92d83-cbd5-46d9-b356-00ae5c27c60d-tls-certs\") pod \"rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl\" (UID: \"65c92d83-cbd5-46d9-b356-00ae5c27c60d\") " pod="ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl"
Jul 14 14:13:09.664548 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:09.664515    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29t5m\" (UniqueName: \"kubernetes.io/projected/65c92d83-cbd5-46d9-b356-00ae5c27c60d-kube-api-access-29t5m\") pod \"rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl\" (UID: \"65c92d83-cbd5-46d9-b356-00ae5c27c60d\") " pod="ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl"
Jul 14 14:13:09.711179 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:09.711149    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl"
Jul 14 14:13:09.845211 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:09.845182    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl"]
Jul 14 14:13:09.846949 ip-10-0-140-78 kubenswrapper[2569]: W0714 14:13:09.846919    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65c92d83_cbd5_46d9_b356_00ae5c27c60d.slice/crio-a248757579b68b99f03bb859d506bf6d1738137969f020d2d68da96baf5d00b4 WatchSource:0}: Error finding container a248757579b68b99f03bb859d506bf6d1738137969f020d2d68da96baf5d00b4: Status 404 returned error can't find the container with id a248757579b68b99f03bb859d506bf6d1738137969f020d2d68da96baf5d00b4
Jul 14 14:13:10.661537 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:10.661502    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl" event={"ID":"65c92d83-cbd5-46d9-b356-00ae5c27c60d","Type":"ContainerStarted","Data":"6928da44802af95215b4c89b8910cc04c8d11f669eaac3ae5a2682578091be62"}
Jul 14 14:13:10.661915 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:10.661549    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl" event={"ID":"65c92d83-cbd5-46d9-b356-00ae5c27c60d","Type":"ContainerStarted","Data":"a248757579b68b99f03bb859d506bf6d1738137969f020d2d68da96baf5d00b4"}
Jul 14 14:13:10.686224 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:10.686175    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl" podStartSLOduration=1.686159049 podStartE2EDuration="1.686159049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:13:10.683795438 +0000 UTC m=+2756.010115367" watchObservedRunningTime="2026-07-14 14:13:10.686159049 +0000 UTC m=+2756.012479016"
Jul 14 14:13:19.711602 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:19.711568    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl"
Jul 14 14:13:19.712004 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:19.711612    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl"
Jul 14 14:13:19.724270 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:19.724248    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl"
Jul 14 14:13:20.722585 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:20.722559    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl"
Jul 14 14:13:54.185279 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:54.185249    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj"]
Jul 14 14:13:55.101768 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:13:55.101737    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl"]
Jul 14 14:14:09.204865 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:09.204806    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj" podUID="3131c790-b6c7-467a-81dd-e5872757e92e" containerName="main" containerID="cri-o://d4bb63a9bc67130062272592b5af8b32a1d4aa6b3b1d87a3a26070cb4891352d" gracePeriod=45
Jul 14 14:14:09.453782 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:09.453761    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj"
Jul 14 14:14:09.642306 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:09.642277    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ks8x\" (UniqueName: \"kubernetes.io/projected/3131c790-b6c7-467a-81dd-e5872757e92e-kube-api-access-6ks8x\") pod \"3131c790-b6c7-467a-81dd-e5872757e92e\" (UID: \"3131c790-b6c7-467a-81dd-e5872757e92e\") "
Jul 14 14:14:09.642466 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:09.642320    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3131c790-b6c7-467a-81dd-e5872757e92e-model-cache\") pod \"3131c790-b6c7-467a-81dd-e5872757e92e\" (UID: \"3131c790-b6c7-467a-81dd-e5872757e92e\") "
Jul 14 14:14:09.642466 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:09.642360    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3131c790-b6c7-467a-81dd-e5872757e92e-tmp-dir\") pod \"3131c790-b6c7-467a-81dd-e5872757e92e\" (UID: \"3131c790-b6c7-467a-81dd-e5872757e92e\") "
Jul 14 14:14:09.642466 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:09.642392    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3131c790-b6c7-467a-81dd-e5872757e92e-tls-certs\") pod \"3131c790-b6c7-467a-81dd-e5872757e92e\" (UID: \"3131c790-b6c7-467a-81dd-e5872757e92e\") "
Jul 14 14:14:09.642466 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:09.642416    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3131c790-b6c7-467a-81dd-e5872757e92e-dshm\") pod \"3131c790-b6c7-467a-81dd-e5872757e92e\" (UID: \"3131c790-b6c7-467a-81dd-e5872757e92e\") "
Jul 14 14:14:09.642681 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:09.642516    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3131c790-b6c7-467a-81dd-e5872757e92e-home\") pod \"3131c790-b6c7-467a-81dd-e5872757e92e\" (UID: \"3131c790-b6c7-467a-81dd-e5872757e92e\") "
Jul 14 14:14:09.642681 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:09.642640    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3131c790-b6c7-467a-81dd-e5872757e92e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3131c790-b6c7-467a-81dd-e5872757e92e" (UID: "3131c790-b6c7-467a-81dd-e5872757e92e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:14:09.642681 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:09.642653    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3131c790-b6c7-467a-81dd-e5872757e92e-model-cache" (OuterVolumeSpecName: "model-cache") pod "3131c790-b6c7-467a-81dd-e5872757e92e" (UID: "3131c790-b6c7-467a-81dd-e5872757e92e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:14:09.642855 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:09.642832    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3131c790-b6c7-467a-81dd-e5872757e92e-home" (OuterVolumeSpecName: "home") pod "3131c790-b6c7-467a-81dd-e5872757e92e" (UID: "3131c790-b6c7-467a-81dd-e5872757e92e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:14:09.642908 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:09.642838    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3131c790-b6c7-467a-81dd-e5872757e92e-model-cache\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:14:09.642908 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:09.642886    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3131c790-b6c7-467a-81dd-e5872757e92e-tmp-dir\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:14:09.644451 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:09.644419    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3131c790-b6c7-467a-81dd-e5872757e92e-kube-api-access-6ks8x" (OuterVolumeSpecName: "kube-api-access-6ks8x") pod "3131c790-b6c7-467a-81dd-e5872757e92e" (UID: "3131c790-b6c7-467a-81dd-e5872757e92e"). InnerVolumeSpecName "kube-api-access-6ks8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 14:14:09.644571 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:09.644541    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3131c790-b6c7-467a-81dd-e5872757e92e-dshm" (OuterVolumeSpecName: "dshm") pod "3131c790-b6c7-467a-81dd-e5872757e92e" (UID: "3131c790-b6c7-467a-81dd-e5872757e92e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:14:09.644571 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:09.644549    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3131c790-b6c7-467a-81dd-e5872757e92e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3131c790-b6c7-467a-81dd-e5872757e92e" (UID: "3131c790-b6c7-467a-81dd-e5872757e92e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 14:14:09.743783 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:09.743758    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3131c790-b6c7-467a-81dd-e5872757e92e-tls-certs\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:14:09.743783 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:09.743780    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3131c790-b6c7-467a-81dd-e5872757e92e-dshm\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:14:09.743909 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:09.743793    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3131c790-b6c7-467a-81dd-e5872757e92e-home\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:14:09.743909 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:09.743805    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ks8x\" (UniqueName: \"kubernetes.io/projected/3131c790-b6c7-467a-81dd-e5872757e92e-kube-api-access-6ks8x\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:14:09.956466 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:09.956405    2569 generic.go:356] "Generic (PLEG): container finished" podID="3131c790-b6c7-467a-81dd-e5872757e92e" containerID="d4bb63a9bc67130062272592b5af8b32a1d4aa6b3b1d87a3a26070cb4891352d" exitCode=0
Jul 14 14:14:09.956466 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:09.956451    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj"
Jul 14 14:14:09.956658 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:09.956489    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj" event={"ID":"3131c790-b6c7-467a-81dd-e5872757e92e","Type":"ContainerDied","Data":"d4bb63a9bc67130062272592b5af8b32a1d4aa6b3b1d87a3a26070cb4891352d"}
Jul 14 14:14:09.956658 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:09.956536    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj" event={"ID":"3131c790-b6c7-467a-81dd-e5872757e92e","Type":"ContainerDied","Data":"f5c8fddc22fe976065ea79e70ab9f092b60929fc519d1718988c53ffd3a12a03"}
Jul 14 14:14:09.956658 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:09.956557    2569 scope.go:117] "RemoveContainer" containerID="d4bb63a9bc67130062272592b5af8b32a1d4aa6b3b1d87a3a26070cb4891352d"
Jul 14 14:14:09.965927 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:09.965911    2569 scope.go:117] "RemoveContainer" containerID="d4bb63a9bc67130062272592b5af8b32a1d4aa6b3b1d87a3a26070cb4891352d"
Jul 14 14:14:09.966193 ip-10-0-140-78 kubenswrapper[2569]: E0714 14:14:09.966173    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4bb63a9bc67130062272592b5af8b32a1d4aa6b3b1d87a3a26070cb4891352d\": container with ID starting with d4bb63a9bc67130062272592b5af8b32a1d4aa6b3b1d87a3a26070cb4891352d not found: ID does not exist" containerID="d4bb63a9bc67130062272592b5af8b32a1d4aa6b3b1d87a3a26070cb4891352d"
Jul 14 14:14:09.966260 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:09.966201    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4bb63a9bc67130062272592b5af8b32a1d4aa6b3b1d87a3a26070cb4891352d"} err="failed to get container status \"d4bb63a9bc67130062272592b5af8b32a1d4aa6b3b1d87a3a26070cb4891352d\": rpc error: code = NotFound desc = could not find container \"d4bb63a9bc67130062272592b5af8b32a1d4aa6b3b1d87a3a26070cb4891352d\": container with ID starting with d4bb63a9bc67130062272592b5af8b32a1d4aa6b3b1d87a3a26070cb4891352d not found: ID does not exist"
Jul 14 14:14:09.980309 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:09.980290    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj"]
Jul 14 14:14:09.984986 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:09.984967    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-9c0e14/rate-test-model-9c0e14-kserve-5657d875f5-knxrj"]
Jul 14 14:14:10.121446 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:10.121414    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl" podUID="65c92d83-cbd5-46d9-b356-00ae5c27c60d" containerName="main" containerID="cri-o://6928da44802af95215b4c89b8910cc04c8d11f669eaac3ae5a2682578091be62" gracePeriod=45
Jul 14 14:14:10.369057 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:10.369036    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl"
Jul 14 14:14:10.549943 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:10.549916    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65c92d83-cbd5-46d9-b356-00ae5c27c60d-dshm\") pod \"65c92d83-cbd5-46d9-b356-00ae5c27c60d\" (UID: \"65c92d83-cbd5-46d9-b356-00ae5c27c60d\") "
Jul 14 14:14:10.550155 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:10.549961    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29t5m\" (UniqueName: \"kubernetes.io/projected/65c92d83-cbd5-46d9-b356-00ae5c27c60d-kube-api-access-29t5m\") pod \"65c92d83-cbd5-46d9-b356-00ae5c27c60d\" (UID: \"65c92d83-cbd5-46d9-b356-00ae5c27c60d\") "
Jul 14 14:14:10.550155 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:10.549985    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65c92d83-cbd5-46d9-b356-00ae5c27c60d-tls-certs\") pod \"65c92d83-cbd5-46d9-b356-00ae5c27c60d\" (UID: \"65c92d83-cbd5-46d9-b356-00ae5c27c60d\") "
Jul 14 14:14:10.550155 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:10.550031    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65c92d83-cbd5-46d9-b356-00ae5c27c60d-home\") pod \"65c92d83-cbd5-46d9-b356-00ae5c27c60d\" (UID: \"65c92d83-cbd5-46d9-b356-00ae5c27c60d\") "
Jul 14 14:14:10.550155 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:10.550061    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65c92d83-cbd5-46d9-b356-00ae5c27c60d-model-cache\") pod \"65c92d83-cbd5-46d9-b356-00ae5c27c60d\" (UID: \"65c92d83-cbd5-46d9-b356-00ae5c27c60d\") "
Jul 14 14:14:10.550155 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:10.550099    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65c92d83-cbd5-46d9-b356-00ae5c27c60d-tmp-dir\") pod \"65c92d83-cbd5-46d9-b356-00ae5c27c60d\" (UID: \"65c92d83-cbd5-46d9-b356-00ae5c27c60d\") "
Jul 14 14:14:10.550478 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:10.550319    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c92d83-cbd5-46d9-b356-00ae5c27c60d-home" (OuterVolumeSpecName: "home") pod "65c92d83-cbd5-46d9-b356-00ae5c27c60d" (UID: "65c92d83-cbd5-46d9-b356-00ae5c27c60d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:14:10.550478 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:10.550334    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c92d83-cbd5-46d9-b356-00ae5c27c60d-model-cache" (OuterVolumeSpecName: "model-cache") pod "65c92d83-cbd5-46d9-b356-00ae5c27c60d" (UID: "65c92d83-cbd5-46d9-b356-00ae5c27c60d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:14:10.550478 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:10.550427    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c92d83-cbd5-46d9-b356-00ae5c27c60d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "65c92d83-cbd5-46d9-b356-00ae5c27c60d" (UID: "65c92d83-cbd5-46d9-b356-00ae5c27c60d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:14:10.552173 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:10.552150    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c92d83-cbd5-46d9-b356-00ae5c27c60d-dshm" (OuterVolumeSpecName: "dshm") pod "65c92d83-cbd5-46d9-b356-00ae5c27c60d" (UID: "65c92d83-cbd5-46d9-b356-00ae5c27c60d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 14:14:10.552273 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:10.552166    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65c92d83-cbd5-46d9-b356-00ae5c27c60d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "65c92d83-cbd5-46d9-b356-00ae5c27c60d" (UID: "65c92d83-cbd5-46d9-b356-00ae5c27c60d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 14:14:10.552273 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:10.552260    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65c92d83-cbd5-46d9-b356-00ae5c27c60d-kube-api-access-29t5m" (OuterVolumeSpecName: "kube-api-access-29t5m") pod "65c92d83-cbd5-46d9-b356-00ae5c27c60d" (UID: "65c92d83-cbd5-46d9-b356-00ae5c27c60d"). InnerVolumeSpecName "kube-api-access-29t5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 14:14:10.651275 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:10.651250    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65c92d83-cbd5-46d9-b356-00ae5c27c60d-home\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:14:10.651275 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:10.651270    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65c92d83-cbd5-46d9-b356-00ae5c27c60d-model-cache\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:14:10.651413 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:10.651281    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65c92d83-cbd5-46d9-b356-00ae5c27c60d-tmp-dir\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:14:10.651413 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:10.651289    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65c92d83-cbd5-46d9-b356-00ae5c27c60d-dshm\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:14:10.651413 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:10.651297    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29t5m\" (UniqueName: \"kubernetes.io/projected/65c92d83-cbd5-46d9-b356-00ae5c27c60d-kube-api-access-29t5m\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:14:10.651413 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:10.651306    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65c92d83-cbd5-46d9-b356-00ae5c27c60d-tls-certs\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:14:10.964340 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:10.964242    2569 generic.go:356] "Generic (PLEG): container finished" podID="65c92d83-cbd5-46d9-b356-00ae5c27c60d" containerID="6928da44802af95215b4c89b8910cc04c8d11f669eaac3ae5a2682578091be62" exitCode=0
Jul 14 14:14:10.964340 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:10.964284    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl"
Jul 14 14:14:10.964340 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:10.964317    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl" event={"ID":"65c92d83-cbd5-46d9-b356-00ae5c27c60d","Type":"ContainerDied","Data":"6928da44802af95215b4c89b8910cc04c8d11f669eaac3ae5a2682578091be62"}
Jul 14 14:14:10.964619 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:10.964346    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl" event={"ID":"65c92d83-cbd5-46d9-b356-00ae5c27c60d","Type":"ContainerDied","Data":"a248757579b68b99f03bb859d506bf6d1738137969f020d2d68da96baf5d00b4"}
Jul 14 14:14:10.964619 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:10.964369    2569 scope.go:117] "RemoveContainer" containerID="6928da44802af95215b4c89b8910cc04c8d11f669eaac3ae5a2682578091be62"
Jul 14 14:14:10.974078 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:10.974067    2569 scope.go:117] "RemoveContainer" containerID="6928da44802af95215b4c89b8910cc04c8d11f669eaac3ae5a2682578091be62"
Jul 14 14:14:10.974361 ip-10-0-140-78 kubenswrapper[2569]: E0714 14:14:10.974330    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6928da44802af95215b4c89b8910cc04c8d11f669eaac3ae5a2682578091be62\": container with ID starting with 6928da44802af95215b4c89b8910cc04c8d11f669eaac3ae5a2682578091be62 not found: ID does not exist" containerID="6928da44802af95215b4c89b8910cc04c8d11f669eaac3ae5a2682578091be62"
Jul 14 14:14:10.974445 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:10.974369    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6928da44802af95215b4c89b8910cc04c8d11f669eaac3ae5a2682578091be62"} err="failed to get container status \"6928da44802af95215b4c89b8910cc04c8d11f669eaac3ae5a2682578091be62\": rpc error: code = NotFound desc = could not find container \"6928da44802af95215b4c89b8910cc04c8d11f669eaac3ae5a2682578091be62\": container with ID starting with 6928da44802af95215b4c89b8910cc04c8d11f669eaac3ae5a2682578091be62 not found: ID does not exist"
Jul 14 14:14:10.988936 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:10.988911    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl"]
Jul 14 14:14:10.993416 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:10.993395    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-ff9f03/rate-test-model-ff9f03-kserve-5bd485d7d7-wqxgl"]
Jul 14 14:14:11.193056 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:11.193034    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3131c790-b6c7-467a-81dd-e5872757e92e" path="/var/lib/kubelet/pods/3131c790-b6c7-467a-81dd-e5872757e92e/volumes"
Jul 14 14:14:11.193420 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:11.193406    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65c92d83-cbd5-46d9-b356-00ae5c27c60d" path="/var/lib/kubelet/pods/65c92d83-cbd5-46d9-b356-00ae5c27c60d/volumes"
Jul 14 14:14:11.757590 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:11.757560    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7790-139626452215104"]
Jul 14 14:14:11.757967 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:11.757869    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65c92d83-cbd5-46d9-b356-00ae5c27c60d" containerName="main"
Jul 14 14:14:11.757967 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:11.757879    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="65c92d83-cbd5-46d9-b356-00ae5c27c60d" containerName="main"
Jul 14 14:14:11.757967 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:11.757898    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3131c790-b6c7-467a-81dd-e5872757e92e" containerName="main"
Jul 14 14:14:11.757967 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:11.757903    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3131c790-b6c7-467a-81dd-e5872757e92e" containerName="main"
Jul 14 14:14:11.757967 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:11.757958    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3131c790-b6c7-467a-81dd-e5872757e92e" containerName="main"
Jul 14 14:14:11.757967 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:11.757966    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="65c92d83-cbd5-46d9-b356-00ae5c27c60d" containerName="main"
Jul 14 14:14:11.762294 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:11.762278    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7790-139626452215104"
Jul 14 14:14:11.765050 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:11.765030    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-t7lm6\""
Jul 14 14:14:11.766565 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:11.766546    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7790-139626452215104"]
Jul 14 14:14:11.860674 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:11.860644    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtzf8\" (UniqueName: \"kubernetes.io/projected/d6c6c6cc-d46d-46fe-899d-7befab8e59e4-kube-api-access-wtzf8\") pod \"test-curl-7790-139626452215104\" (UID: \"d6c6c6cc-d46d-46fe-899d-7befab8e59e4\") " pod="openshift-ingress/test-curl-7790-139626452215104"
Jul 14 14:14:11.961719 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:11.961688    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtzf8\" (UniqueName: \"kubernetes.io/projected/d6c6c6cc-d46d-46fe-899d-7befab8e59e4-kube-api-access-wtzf8\") pod \"test-curl-7790-139626452215104\" (UID: \"d6c6c6cc-d46d-46fe-899d-7befab8e59e4\") " pod="openshift-ingress/test-curl-7790-139626452215104"
Jul 14 14:14:11.971313 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:11.971292    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtzf8\" (UniqueName: \"kubernetes.io/projected/d6c6c6cc-d46d-46fe-899d-7befab8e59e4-kube-api-access-wtzf8\") pod \"test-curl-7790-139626452215104\" (UID: \"d6c6c6cc-d46d-46fe-899d-7befab8e59e4\") " pod="openshift-ingress/test-curl-7790-139626452215104"
Jul 14 14:14:12.072521 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:12.072470    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7790-139626452215104"
Jul 14 14:14:12.201471 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:12.201445    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7790-139626452215104"]
Jul 14 14:14:12.202689 ip-10-0-140-78 kubenswrapper[2569]: W0714 14:14:12.202663    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6c6c6cc_d46d_46fe_899d_7befab8e59e4.slice/crio-3345574cbe102be67dc4d294081959f413b9353f01adb7bfb53e8aea0dcf7c2a WatchSource:0}: Error finding container 3345574cbe102be67dc4d294081959f413b9353f01adb7bfb53e8aea0dcf7c2a: Status 404 returned error can't find the container with id 3345574cbe102be67dc4d294081959f413b9353f01adb7bfb53e8aea0dcf7c2a
Jul 14 14:14:12.203961 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:12.203943    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 14:14:12.976129 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:12.976095    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7790-139626452215104" event={"ID":"d6c6c6cc-d46d-46fe-899d-7befab8e59e4","Type":"ContainerStarted","Data":"81f6844ebf72ce1bde7a89850fc4bda3762e8338fa822f1968cde366973285ab"}
Jul 14 14:14:12.976431 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:12.976139    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7790-139626452215104" event={"ID":"d6c6c6cc-d46d-46fe-899d-7befab8e59e4","Type":"ContainerStarted","Data":"3345574cbe102be67dc4d294081959f413b9353f01adb7bfb53e8aea0dcf7c2a"}
Jul 14 14:14:12.994283 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:12.994242    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/test-curl-7790-139626452215104" podStartSLOduration=1.309602109 podStartE2EDuration="1.994227833s" totalImagesPullingTime="684.625724ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:14:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:14:12.992162099 +0000 UTC m=+2818.318482029" watchObservedRunningTime="2026-07-14 14:14:12.994227833 +0000 UTC m=+2818.320547762"
Jul 14 14:14:13.101922 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:13.101897    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7790-139626452215104_d6c6c6cc-d46d-46fe-899d-7befab8e59e4/test-curl-7790-139626452215104/0.log"
Jul 14 14:14:13.983388 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:13.983354    2569 generic.go:356] "Generic (PLEG): container finished" podID="d6c6c6cc-d46d-46fe-899d-7befab8e59e4" containerID="81f6844ebf72ce1bde7a89850fc4bda3762e8338fa822f1968cde366973285ab" exitCode=0
Jul 14 14:14:13.983770 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:13.983398    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7790-139626452215104" event={"ID":"d6c6c6cc-d46d-46fe-899d-7befab8e59e4","Type":"ContainerDied","Data":"81f6844ebf72ce1bde7a89850fc4bda3762e8338fa822f1968cde366973285ab"}
Jul 14 14:14:15.128366 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:15.128345    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7790-139626452215104"
Jul 14 14:14:15.165362 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:15.165332    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7790-139626452215104"]
Jul 14 14:14:15.169895 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:15.169873    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7790-139626452215104"]
Jul 14 14:14:15.185168 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:15.185146    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtzf8\" (UniqueName: \"kubernetes.io/projected/d6c6c6cc-d46d-46fe-899d-7befab8e59e4-kube-api-access-wtzf8\") pod \"d6c6c6cc-d46d-46fe-899d-7befab8e59e4\" (UID: \"d6c6c6cc-d46d-46fe-899d-7befab8e59e4\") "
Jul 14 14:14:15.187399 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:15.187377    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c6c6cc-d46d-46fe-899d-7befab8e59e4-kube-api-access-wtzf8" (OuterVolumeSpecName: "kube-api-access-wtzf8") pod "d6c6c6cc-d46d-46fe-899d-7befab8e59e4" (UID: "d6c6c6cc-d46d-46fe-899d-7befab8e59e4"). InnerVolumeSpecName "kube-api-access-wtzf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 14:14:15.193592 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:15.193566    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6c6c6cc-d46d-46fe-899d-7befab8e59e4" path="/var/lib/kubelet/pods/d6c6c6cc-d46d-46fe-899d-7befab8e59e4/volumes"
Jul 14 14:14:15.286690 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:15.286638    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtzf8\" (UniqueName: \"kubernetes.io/projected/d6c6c6cc-d46d-46fe-899d-7befab8e59e4-kube-api-access-wtzf8\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:14:15.994719 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:15.994692    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7790-139626452215104"
Jul 14 14:14:15.994909 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:15.994731    2569 scope.go:117] "RemoveContainer" containerID="81f6844ebf72ce1bde7a89850fc4bda3762e8338fa822f1968cde366973285ab"
Jul 14 14:14:25.895791 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:25.895762    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm"]
Jul 14 14:14:25.896338 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:25.895974    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm" podUID="3ca80e71-3c4f-4462-8e26-85295439fe1b" containerName="maas-api" containerID="cri-o://11055ad07adb8f7ef186ca0e7bd0021d627bd8be55356f70e1d0d261dad2006c" gracePeriod=30
Jul 14 14:14:26.044862 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:26.044831    2569 generic.go:356] "Generic (PLEG): container finished" podID="3ca80e71-3c4f-4462-8e26-85295439fe1b" containerID="11055ad07adb8f7ef186ca0e7bd0021d627bd8be55356f70e1d0d261dad2006c" exitCode=0
Jul 14 14:14:26.045041 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:26.044875    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm" event={"ID":"3ca80e71-3c4f-4462-8e26-85295439fe1b","Type":"ContainerDied","Data":"11055ad07adb8f7ef186ca0e7bd0021d627bd8be55356f70e1d0d261dad2006c"}
Jul 14 14:14:26.143568 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:26.143544    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm"
Jul 14 14:14:26.167142 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:26.167055    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvnvp\" (UniqueName: \"kubernetes.io/projected/3ca80e71-3c4f-4462-8e26-85295439fe1b-kube-api-access-tvnvp\") pod \"3ca80e71-3c4f-4462-8e26-85295439fe1b\" (UID: \"3ca80e71-3c4f-4462-8e26-85295439fe1b\") "
Jul 14 14:14:26.167707 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:26.167633    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ca80e71-3c4f-4462-8e26-85295439fe1b-maas-api-tls\") pod \"3ca80e71-3c4f-4462-8e26-85295439fe1b\" (UID: \"3ca80e71-3c4f-4462-8e26-85295439fe1b\") "
Jul 14 14:14:26.167862 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:26.167840    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ca80e71-3c4f-4462-8e26-85295439fe1b-openshift-service-ca\") pod \"3ca80e71-3c4f-4462-8e26-85295439fe1b\" (UID: \"3ca80e71-3c4f-4462-8e26-85295439fe1b\") "
Jul 14 14:14:26.168510 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:26.168484    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ca80e71-3c4f-4462-8e26-85295439fe1b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3ca80e71-3c4f-4462-8e26-85295439fe1b" (UID: "3ca80e71-3c4f-4462-8e26-85295439fe1b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 14:14:26.170361 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:26.170329    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ca80e71-3c4f-4462-8e26-85295439fe1b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3ca80e71-3c4f-4462-8e26-85295439fe1b" (UID: "3ca80e71-3c4f-4462-8e26-85295439fe1b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 14:14:26.170459 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:26.170421    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ca80e71-3c4f-4462-8e26-85295439fe1b-kube-api-access-tvnvp" (OuterVolumeSpecName: "kube-api-access-tvnvp") pod "3ca80e71-3c4f-4462-8e26-85295439fe1b" (UID: "3ca80e71-3c4f-4462-8e26-85295439fe1b"). InnerVolumeSpecName "kube-api-access-tvnvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 14:14:26.268751 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:26.268721    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvnvp\" (UniqueName: \"kubernetes.io/projected/3ca80e71-3c4f-4462-8e26-85295439fe1b-kube-api-access-tvnvp\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:14:26.268751 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:26.268750    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ca80e71-3c4f-4462-8e26-85295439fe1b-maas-api-tls\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:14:26.268905 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:26.268763    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ca80e71-3c4f-4462-8e26-85295439fe1b-openshift-service-ca\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:14:27.051129 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:27.051093    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm" event={"ID":"3ca80e71-3c4f-4462-8e26-85295439fe1b","Type":"ContainerDied","Data":"c038fec0b3d3d709a423ff72ffb9d966a00da3522f74a0d170be83b8c5faed0d"}
Jul 14 14:14:27.051597 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:27.051154    2569 scope.go:117] "RemoveContainer" containerID="11055ad07adb8f7ef186ca0e7bd0021d627bd8be55356f70e1d0d261dad2006c"
Jul 14 14:14:27.051597 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:27.051171    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm"
Jul 14 14:14:27.075971 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:27.075948    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm"]
Jul 14 14:14:27.081609 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:27.081588    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-9c0e14-67b84768b-kjpjm"]
Jul 14 14:14:27.192337 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:27.192314    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ca80e71-3c4f-4462-8e26-85295439fe1b" path="/var/lib/kubelet/pods/3ca80e71-3c4f-4462-8e26-85295439fe1b/volumes"
Jul 14 14:14:51.960868 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:51.960832    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv"]
Jul 14 14:14:51.961413 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:51.961081    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv" podUID="12480401-c287-4035-a99e-717966dda4e9" containerName="maas-api" containerID="cri-o://14f03bdeb92b69a5aa6a67ac2505c0cc9a44a117c155fce874200dd7128ab388" gracePeriod=30
Jul 14 14:14:52.182233 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:52.182201    2569 generic.go:356] "Generic (PLEG): container finished" podID="12480401-c287-4035-a99e-717966dda4e9" containerID="14f03bdeb92b69a5aa6a67ac2505c0cc9a44a117c155fce874200dd7128ab388" exitCode=0
Jul 14 14:14:52.182368 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:52.182280    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv" event={"ID":"12480401-c287-4035-a99e-717966dda4e9","Type":"ContainerDied","Data":"14f03bdeb92b69a5aa6a67ac2505c0cc9a44a117c155fce874200dd7128ab388"}
Jul 14 14:14:52.226316 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:52.226296    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv"
Jul 14 14:14:52.355482 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:52.355458    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8d9fs\" (UniqueName: \"kubernetes.io/projected/12480401-c287-4035-a99e-717966dda4e9-kube-api-access-8d9fs\") pod \"12480401-c287-4035-a99e-717966dda4e9\" (UID: \"12480401-c287-4035-a99e-717966dda4e9\") "
Jul 14 14:14:52.355587 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:52.355499    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12480401-c287-4035-a99e-717966dda4e9-openshift-service-ca\") pod \"12480401-c287-4035-a99e-717966dda4e9\" (UID: \"12480401-c287-4035-a99e-717966dda4e9\") "
Jul 14 14:14:52.355587 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:52.355571    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12480401-c287-4035-a99e-717966dda4e9-maas-api-tls\") pod \"12480401-c287-4035-a99e-717966dda4e9\" (UID: \"12480401-c287-4035-a99e-717966dda4e9\") "
Jul 14 14:14:52.355796 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:52.355772    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12480401-c287-4035-a99e-717966dda4e9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "12480401-c287-4035-a99e-717966dda4e9" (UID: "12480401-c287-4035-a99e-717966dda4e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 14:14:52.355915 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:52.355876    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12480401-c287-4035-a99e-717966dda4e9-openshift-service-ca\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:14:52.357495 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:52.357469    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12480401-c287-4035-a99e-717966dda4e9-kube-api-access-8d9fs" (OuterVolumeSpecName: "kube-api-access-8d9fs") pod "12480401-c287-4035-a99e-717966dda4e9" (UID: "12480401-c287-4035-a99e-717966dda4e9"). InnerVolumeSpecName "kube-api-access-8d9fs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 14:14:52.357577 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:52.357531    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12480401-c287-4035-a99e-717966dda4e9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "12480401-c287-4035-a99e-717966dda4e9" (UID: "12480401-c287-4035-a99e-717966dda4e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 14:14:52.456550 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:52.456528    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d9fs\" (UniqueName: \"kubernetes.io/projected/12480401-c287-4035-a99e-717966dda4e9-kube-api-access-8d9fs\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:14:52.456550 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:52.456552    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12480401-c287-4035-a99e-717966dda4e9-maas-api-tls\") on node \"ip-10-0-140-78.ec2.internal\" DevicePath \"\""
Jul 14 14:14:53.188474 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:53.188445    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv"
Jul 14 14:14:53.193088 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:53.193062    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv" event={"ID":"12480401-c287-4035-a99e-717966dda4e9","Type":"ContainerDied","Data":"c8c12161e1cfa6a9757bb3f38f06c48d21e6ffd2692f5d306b9932427e71458c"}
Jul 14 14:14:53.193249 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:53.193107    2569 scope.go:117] "RemoveContainer" containerID="14f03bdeb92b69a5aa6a67ac2505c0cc9a44a117c155fce874200dd7128ab388"
Jul 14 14:14:53.219449 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:53.219421    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv"]
Jul 14 14:14:53.227843 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:53.227821    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ff9f03-745488dcf7-z8qgv"]
Jul 14 14:14:55.193452 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:14:55.193417    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12480401-c287-4035-a99e-717966dda4e9" path="/var/lib/kubelet/pods/12480401-c287-4035-a99e-717966dda4e9/volumes"
Jul 14 14:15:08.076680 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:08.076603    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-fb96d7cb4-7m689_529223ad-b813-4f48-862a-25a29e1df450/authorino/0.log"
Jul 14 14:15:13.885683 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:13.885653    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9464f545f-d9n84_9b4b8e64-9a5b-4839-a34c-edf6e5742d03/copy-manifests/0.log"
Jul 14 14:15:13.907042 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:13.907001    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9464f545f-d9n84_9b4b8e64-9a5b-4839-a34c-edf6e5742d03/manager/0.log"
Jul 14 14:15:14.422854 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:14.422831    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7b58d875b8-7x6xl_29b396cc-7e5d-488a-8b17-c2058d40108b/maas-api/0.log"
Jul 14 14:15:17.158838 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:17.158803    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf_c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55/util/0.log"
Jul 14 14:15:17.165650 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:17.165630    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf_c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55/pull/0.log"
Jul 14 14:15:17.172118 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:17.172092    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf_c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55/extract/0.log"
Jul 14 14:15:17.295607 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:17.295588    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v_e0ff5e6d-1a86-4426-92a1-6765774cfe35/util/0.log"
Jul 14 14:15:17.302383 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:17.302362    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v_e0ff5e6d-1a86-4426-92a1-6765774cfe35/pull/0.log"
Jul 14 14:15:17.309082 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:17.309062    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v_e0ff5e6d-1a86-4426-92a1-6765774cfe35/extract/0.log"
Jul 14 14:15:17.423774 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:17.423708    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582_b9488dcf-d5f5-472c-a31b-30ab21f755a3/util/0.log"
Jul 14 14:15:17.429993 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:17.429973    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582_b9488dcf-d5f5-472c-a31b-30ab21f755a3/pull/0.log"
Jul 14 14:15:17.435952 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:17.435920    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582_b9488dcf-d5f5-472c-a31b-30ab21f755a3/extract/0.log"
Jul 14 14:15:17.554995 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:17.554976    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp_688ce8c2-6ba4-450e-bd7c-e5a025b403a5/util/0.log"
Jul 14 14:15:17.562114 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:17.562095    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp_688ce8c2-6ba4-450e-bd7c-e5a025b403a5/pull/0.log"
Jul 14 14:15:17.570923 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:17.570903    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp_688ce8c2-6ba4-450e-bd7c-e5a025b403a5/extract/0.log"
Jul 14 14:15:17.715064 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:17.714995    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-fb96d7cb4-7m689_529223ad-b813-4f48-862a-25a29e1df450/authorino/0.log"
Jul 14 14:15:18.095843 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:18.095817    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-vpptc_2827e420-b281-45e5-86a6-48dd0735031a/kuadrant-console-plugin/0.log"
Jul 14 14:15:19.276143 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:19.276111    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-594bccb575-kj7t8_1f9fa9ff-066a-4e48-a0c2-b7087967960b/kube-auth-proxy/0.log"
Jul 14 14:15:19.773187 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:19.773161    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-d28fh_6cb5d20f-6774-416d-a71b-29eced8216f8/payload-processing/0.log"
Jul 14 14:15:20.393868 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:20.393833    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-wzllw_b9e3d658-7b47-4f4a-85d0-b79891998c5d/main/0.log"
Jul 14 14:15:20.765629 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:20.765601    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-74djq_6fc34038-e1bc-49d8-acab-37badc7896ed/main/0.log"
Jul 14 14:15:20.897884 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:20.897855    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-nsckv_da3ed516-912b-4b7f-84a3-a05c30a5732f/main/0.log"
Jul 14 14:15:25.242745 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:25.242713    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t9pjq/must-gather-2qrtq"]
Jul 14 14:15:25.243131 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:25.243004    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12480401-c287-4035-a99e-717966dda4e9" containerName="maas-api"
Jul 14 14:15:25.243131 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:25.243034    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="12480401-c287-4035-a99e-717966dda4e9" containerName="maas-api"
Jul 14 14:15:25.243131 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:25.243055    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ca80e71-3c4f-4462-8e26-85295439fe1b" containerName="maas-api"
Jul 14 14:15:25.243131 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:25.243064    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ca80e71-3c4f-4462-8e26-85295439fe1b" containerName="maas-api"
Jul 14 14:15:25.243131 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:25.243076    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6c6c6cc-d46d-46fe-899d-7befab8e59e4" containerName="test-curl-7790-139626452215104"
Jul 14 14:15:25.243131 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:25.243086    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6c6c6cc-d46d-46fe-899d-7befab8e59e4" containerName="test-curl-7790-139626452215104"
Jul 14 14:15:25.243325 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:25.243144    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6c6c6cc-d46d-46fe-899d-7befab8e59e4" containerName="test-curl-7790-139626452215104"
Jul 14 14:15:25.243325 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:25.243153    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="12480401-c287-4035-a99e-717966dda4e9" containerName="maas-api"
Jul 14 14:15:25.243325 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:25.243160    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ca80e71-3c4f-4462-8e26-85295439fe1b" containerName="maas-api"
Jul 14 14:15:25.245985 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:25.245967    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t9pjq/must-gather-2qrtq"
Jul 14 14:15:25.248587 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:25.248560    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t9pjq\"/\"kube-root-ca.crt\""
Jul 14 14:15:25.248863 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:25.248843    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t9pjq\"/\"openshift-service-ca.crt\""
Jul 14 14:15:25.248956 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:25.248900    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-t9pjq\"/\"default-dockercfg-gqsqg\""
Jul 14 14:15:25.255593 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:25.255567    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t9pjq/must-gather-2qrtq"]
Jul 14 14:15:25.396070 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:25.396046    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c002366c-9261-44ca-a5ec-aa8d81bc6701-must-gather-output\") pod \"must-gather-2qrtq\" (UID: \"c002366c-9261-44ca-a5ec-aa8d81bc6701\") " pod="openshift-must-gather-t9pjq/must-gather-2qrtq"
Jul 14 14:15:25.396182 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:25.396092    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpxhm\" (UniqueName: \"kubernetes.io/projected/c002366c-9261-44ca-a5ec-aa8d81bc6701-kube-api-access-dpxhm\") pod \"must-gather-2qrtq\" (UID: \"c002366c-9261-44ca-a5ec-aa8d81bc6701\") " pod="openshift-must-gather-t9pjq/must-gather-2qrtq"
Jul 14 14:15:25.497088 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:25.497027    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c002366c-9261-44ca-a5ec-aa8d81bc6701-must-gather-output\") pod \"must-gather-2qrtq\" (UID: \"c002366c-9261-44ca-a5ec-aa8d81bc6701\") " pod="openshift-must-gather-t9pjq/must-gather-2qrtq"
Jul 14 14:15:25.497088 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:25.497061    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpxhm\" (UniqueName: \"kubernetes.io/projected/c002366c-9261-44ca-a5ec-aa8d81bc6701-kube-api-access-dpxhm\") pod \"must-gather-2qrtq\" (UID: \"c002366c-9261-44ca-a5ec-aa8d81bc6701\") " pod="openshift-must-gather-t9pjq/must-gather-2qrtq"
Jul 14 14:15:25.497410 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:25.497393    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c002366c-9261-44ca-a5ec-aa8d81bc6701-must-gather-output\") pod \"must-gather-2qrtq\" (UID: \"c002366c-9261-44ca-a5ec-aa8d81bc6701\") " pod="openshift-must-gather-t9pjq/must-gather-2qrtq"
Jul 14 14:15:25.506065 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:25.506041    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpxhm\" (UniqueName: \"kubernetes.io/projected/c002366c-9261-44ca-a5ec-aa8d81bc6701-kube-api-access-dpxhm\") pod \"must-gather-2qrtq\" (UID: \"c002366c-9261-44ca-a5ec-aa8d81bc6701\") " pod="openshift-must-gather-t9pjq/must-gather-2qrtq"
Jul 14 14:15:25.556339 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:25.556319    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t9pjq/must-gather-2qrtq"
Jul 14 14:15:25.681848 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:25.681822    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t9pjq/must-gather-2qrtq"]
Jul 14 14:15:25.683300 ip-10-0-140-78 kubenswrapper[2569]: W0714 14:15:25.683273    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc002366c_9261_44ca_a5ec_aa8d81bc6701.slice/crio-9f52438707f4181a5057a8a0b5219ceafc1401248b1df063f817d54a213b237b WatchSource:0}: Error finding container 9f52438707f4181a5057a8a0b5219ceafc1401248b1df063f817d54a213b237b: Status 404 returned error can't find the container with id 9f52438707f4181a5057a8a0b5219ceafc1401248b1df063f817d54a213b237b
Jul 14 14:15:26.355728 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:26.355690    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t9pjq/must-gather-2qrtq" event={"ID":"c002366c-9261-44ca-a5ec-aa8d81bc6701","Type":"ContainerStarted","Data":"9f52438707f4181a5057a8a0b5219ceafc1401248b1df063f817d54a213b237b"}
Jul 14 14:15:27.363365 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:27.363330    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t9pjq/must-gather-2qrtq" event={"ID":"c002366c-9261-44ca-a5ec-aa8d81bc6701","Type":"ContainerStarted","Data":"5b1aa5009bfe14e972bd38b00777830181f06603814986b4e1f1b1da6d912355"}
Jul 14 14:15:27.363844 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:27.363374    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t9pjq/must-gather-2qrtq" event={"ID":"c002366c-9261-44ca-a5ec-aa8d81bc6701","Type":"ContainerStarted","Data":"5a6d29a025572c859b0e67af10b96e3f73253bfc3b34537e0f00da3f77b57c12"}
Jul 14 14:15:27.382884 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:27.382839    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-t9pjq/must-gather-2qrtq" podStartSLOduration=1.541157264 podStartE2EDuration="2.382824816s" totalImagesPullingTime="841.667552ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 14:15:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:15:27.382208923 +0000 UTC m=+2892.708528853" watchObservedRunningTime="2026-07-14 14:15:27.382824816 +0000 UTC m=+2892.709144743"
Jul 14 14:15:28.209736 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:28.209706    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-s7ft2_b33bdbd5-63be-4828-a2d6-91d4bead7b80/global-pull-secret-syncer/0.log"
Jul 14 14:15:28.349059 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:28.349029    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kkhvt_62b825ef-ccb4-4f79-bd95-c0cb2753b4f6/konnectivity-agent/0.log"
Jul 14 14:15:28.433816 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:28.433774    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-78.ec2.internal_9dc64344bcf6e13eee2ce8f9a1eefa2b/haproxy/0.log"
Jul 14 14:15:32.539880 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:32.539842    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf_c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55/extract/0.log"
Jul 14 14:15:32.572149 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:32.572119    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf_c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55/util/0.log"
Jul 14 14:15:32.601101 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:32.601068    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sctdf_c1d0134f-2bbc-40a0-9c11-bfc9d0d65a55/pull/0.log"
Jul 14 14:15:32.629525 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:32.629494    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v_e0ff5e6d-1a86-4426-92a1-6765774cfe35/extract/0.log"
Jul 14 14:15:32.656413 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:32.656386    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v_e0ff5e6d-1a86-4426-92a1-6765774cfe35/util/0.log"
Jul 14 14:15:32.679871 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:32.679843    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0kz84v_e0ff5e6d-1a86-4426-92a1-6765774cfe35/pull/0.log"
Jul 14 14:15:32.708902 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:32.708273    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582_b9488dcf-d5f5-472c-a31b-30ab21f755a3/extract/0.log"
Jul 14 14:15:32.733211 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:32.733180    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582_b9488dcf-d5f5-472c-a31b-30ab21f755a3/util/0.log"
Jul 14 14:15:32.757161 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:32.757104    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tw582_b9488dcf-d5f5-472c-a31b-30ab21f755a3/pull/0.log"
Jul 14 14:15:32.784833 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:32.784800    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp_688ce8c2-6ba4-450e-bd7c-e5a025b403a5/extract/0.log"
Jul 14 14:15:32.806455 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:32.806372    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp_688ce8c2-6ba4-450e-bd7c-e5a025b403a5/util/0.log"
Jul 14 14:15:32.828411 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:32.828380    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15wpgp_688ce8c2-6ba4-450e-bd7c-e5a025b403a5/pull/0.log"
Jul 14 14:15:33.320035 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:33.317066    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-fb96d7cb4-7m689_529223ad-b813-4f48-862a-25a29e1df450/authorino/0.log"
Jul 14 14:15:33.405823 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:33.405790    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-vpptc_2827e420-b281-45e5-86a6-48dd0735031a/kuadrant-console-plugin/0.log"
Jul 14 14:15:35.665460 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:35.665426    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-bfqm5_fff04318-1a39-4b84-a5c5-d53129f9e071/kube-state-metrics/0.log"
Jul 14 14:15:35.699734 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:35.699705    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-bfqm5_fff04318-1a39-4b84-a5c5-d53129f9e071/kube-rbac-proxy-main/0.log"
Jul 14 14:15:35.738698 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:35.738649    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-bfqm5_fff04318-1a39-4b84-a5c5-d53129f9e071/kube-rbac-proxy-self/0.log"
Jul 14 14:15:35.845335 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:35.845212    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ltxjs_34510396-e510-4255-aab4-724af10b00a4/node-exporter/0.log"
Jul 14 14:15:35.867278 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:35.867243    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ltxjs_34510396-e510-4255-aab4-724af10b00a4/kube-rbac-proxy/0.log"
Jul 14 14:15:35.891851 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:35.891822    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ltxjs_34510396-e510-4255-aab4-724af10b00a4/init-textfile/0.log"
Jul 14 14:15:36.372004 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:36.371968    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-bshjk_ec47e455-0f3c-4b63-bf5d-4481866117eb/prometheus-operator/0.log"
Jul 14 14:15:36.389836 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:36.389804    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-bshjk_ec47e455-0f3c-4b63-bf5d-4481866117eb/kube-rbac-proxy/0.log"
Jul 14 14:15:37.838716 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:37.838687    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-nqmzd_421a6500-d253-4f95-beb4-db5438d099c9/networking-console-plugin/0.log"
Jul 14 14:15:38.107788 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:38.107671    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t9pjq/perf-node-gather-daemonset-s7snx"]
Jul 14 14:15:38.113679 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:38.113648    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t9pjq/perf-node-gather-daemonset-s7snx"
Jul 14 14:15:38.124604 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:38.124539    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t9pjq/perf-node-gather-daemonset-s7snx"]
Jul 14 14:15:38.235692 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:38.235658    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/49aecc92-3850-4b5c-9c60-d97d0fd6b220-podres\") pod \"perf-node-gather-daemonset-s7snx\" (UID: \"49aecc92-3850-4b5c-9c60-d97d0fd6b220\") " pod="openshift-must-gather-t9pjq/perf-node-gather-daemonset-s7snx"
Jul 14 14:15:38.235982 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:38.235950    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjlfj\" (UniqueName: \"kubernetes.io/projected/49aecc92-3850-4b5c-9c60-d97d0fd6b220-kube-api-access-fjlfj\") pod \"perf-node-gather-daemonset-s7snx\" (UID: \"49aecc92-3850-4b5c-9c60-d97d0fd6b220\") " pod="openshift-must-gather-t9pjq/perf-node-gather-daemonset-s7snx"
Jul 14 14:15:38.236192 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:38.236177    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49aecc92-3850-4b5c-9c60-d97d0fd6b220-sys\") pod \"perf-node-gather-daemonset-s7snx\" (UID: \"49aecc92-3850-4b5c-9c60-d97d0fd6b220\") " pod="openshift-must-gather-t9pjq/perf-node-gather-daemonset-s7snx"
Jul 14 14:15:38.236380 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:38.236367    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49aecc92-3850-4b5c-9c60-d97d0fd6b220-lib-modules\") pod \"perf-node-gather-daemonset-s7snx\" (UID: \"49aecc92-3850-4b5c-9c60-d97d0fd6b220\") " pod="openshift-must-gather-t9pjq/perf-node-gather-daemonset-s7snx"
Jul 14 14:15:38.236507 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:38.236492    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/49aecc92-3850-4b5c-9c60-d97d0fd6b220-proc\") pod \"perf-node-gather-daemonset-s7snx\" (UID: \"49aecc92-3850-4b5c-9c60-d97d0fd6b220\") " pod="openshift-must-gather-t9pjq/perf-node-gather-daemonset-s7snx"
Jul 14 14:15:38.337765 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:38.337727    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49aecc92-3850-4b5c-9c60-d97d0fd6b220-lib-modules\") pod \"perf-node-gather-daemonset-s7snx\" (UID: \"49aecc92-3850-4b5c-9c60-d97d0fd6b220\") " pod="openshift-must-gather-t9pjq/perf-node-gather-daemonset-s7snx"
Jul 14 14:15:38.338044 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:38.338023    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/49aecc92-3850-4b5c-9c60-d97d0fd6b220-proc\") pod \"perf-node-gather-daemonset-s7snx\" (UID: \"49aecc92-3850-4b5c-9c60-d97d0fd6b220\") " pod="openshift-must-gather-t9pjq/perf-node-gather-daemonset-s7snx"
Jul 14 14:15:38.338197 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:38.338032    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49aecc92-3850-4b5c-9c60-d97d0fd6b220-lib-modules\") pod \"perf-node-gather-daemonset-s7snx\" (UID: \"49aecc92-3850-4b5c-9c60-d97d0fd6b220\") " pod="openshift-must-gather-t9pjq/perf-node-gather-daemonset-s7snx"
Jul 14 14:15:38.338487 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:38.338089    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/49aecc92-3850-4b5c-9c60-d97d0fd6b220-proc\") pod \"perf-node-gather-daemonset-s7snx\" (UID: \"49aecc92-3850-4b5c-9c60-d97d0fd6b220\") " pod="openshift-must-gather-t9pjq/perf-node-gather-daemonset-s7snx"
Jul 14 14:15:38.338591 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:38.338322    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/49aecc92-3850-4b5c-9c60-d97d0fd6b220-podres\") pod \"perf-node-gather-daemonset-s7snx\" (UID: \"49aecc92-3850-4b5c-9c60-d97d0fd6b220\") " pod="openshift-must-gather-t9pjq/perf-node-gather-daemonset-s7snx"
Jul 14 14:15:38.338707 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:38.338694    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjlfj\" (UniqueName: \"kubernetes.io/projected/49aecc92-3850-4b5c-9c60-d97d0fd6b220-kube-api-access-fjlfj\") pod \"perf-node-gather-daemonset-s7snx\" (UID: \"49aecc92-3850-4b5c-9c60-d97d0fd6b220\") " pod="openshift-must-gather-t9pjq/perf-node-gather-daemonset-s7snx"
Jul 14 14:15:38.339138 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:38.339117    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49aecc92-3850-4b5c-9c60-d97d0fd6b220-sys\") pod \"perf-node-gather-daemonset-s7snx\" (UID: \"49aecc92-3850-4b5c-9c60-d97d0fd6b220\") " pod="openshift-must-gather-t9pjq/perf-node-gather-daemonset-s7snx"
Jul 14 14:15:38.339354 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:38.339281    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49aecc92-3850-4b5c-9c60-d97d0fd6b220-sys\") pod \"perf-node-gather-daemonset-s7snx\" (UID: \"49aecc92-3850-4b5c-9c60-d97d0fd6b220\") " pod="openshift-must-gather-t9pjq/perf-node-gather-daemonset-s7snx"
Jul 14 14:15:38.339452 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:38.338439    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/49aecc92-3850-4b5c-9c60-d97d0fd6b220-podres\") pod \"perf-node-gather-daemonset-s7snx\" (UID: \"49aecc92-3850-4b5c-9c60-d97d0fd6b220\") " pod="openshift-must-gather-t9pjq/perf-node-gather-daemonset-s7snx"
Jul 14 14:15:38.351605 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:38.351572    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjlfj\" (UniqueName: \"kubernetes.io/projected/49aecc92-3850-4b5c-9c60-d97d0fd6b220-kube-api-access-fjlfj\") pod \"perf-node-gather-daemonset-s7snx\" (UID: \"49aecc92-3850-4b5c-9c60-d97d0fd6b220\") " pod="openshift-must-gather-t9pjq/perf-node-gather-daemonset-s7snx"
Jul 14 14:15:38.426447 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:38.426368    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t9pjq/perf-node-gather-daemonset-s7snx"
Jul 14 14:15:38.583275 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:38.583202    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t9pjq/perf-node-gather-daemonset-s7snx"]
Jul 14 14:15:38.587153 ip-10-0-140-78 kubenswrapper[2569]: W0714 14:15:38.587113    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod49aecc92_3850_4b5c_9c60_d97d0fd6b220.slice/crio-cc8279e4a5f9f110d89a7b7a9f39c1ee87e563d55b08ebf3748c6f01bc360ec0 WatchSource:0}: Error finding container cc8279e4a5f9f110d89a7b7a9f39c1ee87e563d55b08ebf3748c6f01bc360ec0: Status 404 returned error can't find the container with id cc8279e4a5f9f110d89a7b7a9f39c1ee87e563d55b08ebf3748c6f01bc360ec0
Jul 14 14:15:39.036781 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:39.036708    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6dfbcbd67f-4d6h8_8b2ecbf2-91f6-4669-b754-335e27b35c7c/console/0.log"
Jul 14 14:15:39.465186 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:39.465092    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t9pjq/perf-node-gather-daemonset-s7snx" event={"ID":"49aecc92-3850-4b5c-9c60-d97d0fd6b220","Type":"ContainerStarted","Data":"e037f2ab16163807c276b3f23bdf9ed45198e37c0be1b206089de5435f2a2396"}
Jul 14 14:15:39.465186 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:39.465170    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t9pjq/perf-node-gather-daemonset-s7snx" event={"ID":"49aecc92-3850-4b5c-9c60-d97d0fd6b220","Type":"ContainerStarted","Data":"cc8279e4a5f9f110d89a7b7a9f39c1ee87e563d55b08ebf3748c6f01bc360ec0"}
Jul 14 14:15:39.465857 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:39.465834    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-t9pjq/perf-node-gather-daemonset-s7snx"
Jul 14 14:15:39.487861 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:39.487812    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-t9pjq/perf-node-gather-daemonset-s7snx" podStartSLOduration=1.487797015 podStartE2EDuration="1.487797015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 14:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 14:15:39.487259253 +0000 UTC m=+2904.813579182" watchObservedRunningTime="2026-07-14 14:15:39.487797015 +0000 UTC m=+2904.814116943"
Jul 14 14:15:40.802662 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:40.802636    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lkjjk_b89a211f-60b9-4eab-84cb-7aad1dcd94b1/dns/0.log"
Jul 14 14:15:40.823099 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:40.823073    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lkjjk_b89a211f-60b9-4eab-84cb-7aad1dcd94b1/kube-rbac-proxy/0.log"
Jul 14 14:15:40.924507 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:40.924479    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mjgwk_03018596-d0b1-43da-af02-ede52df67ceb/dns-node-resolver/0.log"
Jul 14 14:15:41.499712 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:41.499664    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-59f76b4df9-2glpk_ca6401ee-66b4-449f-920d-74ef689c7429/registry/0.log"
Jul 14 14:15:41.523032 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:41.522989    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7l994_bf204a1f-63ca-40ac-825b-e8684c67e36f/node-ca/0.log"
Jul 14 14:15:42.694274 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:42.694239    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-594bccb575-kj7t8_1f9fa9ff-066a-4e48-a0c2-b7087967960b/kube-auth-proxy/0.log"
Jul 14 14:15:42.933444 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:42.933407    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-d28fh_6cb5d20f-6774-416d-a71b-29eced8216f8/payload-processing/0.log"
Jul 14 14:15:43.766904 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:43.766876    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vvmd5_5cdbc484-7145-45f0-9060-a4a818174ff5/serve-healthcheck-canary/0.log"
Jul 14 14:15:44.455679 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:44.455597    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xz2lp_c650b1a4-6066-4d12-b4cb-0217c8f4ebfe/kube-rbac-proxy/0.log"
Jul 14 14:15:44.482775 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:44.482744    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xz2lp_c650b1a4-6066-4d12-b4cb-0217c8f4ebfe/exporter/0.log"
Jul 14 14:15:44.505484 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:44.505419    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xz2lp_c650b1a4-6066-4d12-b4cb-0217c8f4ebfe/extractor/0.log"
Jul 14 14:15:46.486201 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:46.486172    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-t9pjq/perf-node-gather-daemonset-s7snx"
Jul 14 14:15:47.182517 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:47.182476    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9464f545f-d9n84_9b4b8e64-9a5b-4839-a34c-edf6e5742d03/manager/0.log"
Jul 14 14:15:47.200113 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:47.200079    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9464f545f-d9n84_9b4b8e64-9a5b-4839-a34c-edf6e5742d03/copy-manifests/0.log"
Jul 14 14:15:47.763677 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:47.763641    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-594bccb575-kj7t8_1f9fa9ff-066a-4e48-a0c2-b7087967960b/kube-auth-proxy/0.log"
Jul 14 14:15:47.891320 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:47.891295    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-d28fh_6cb5d20f-6774-416d-a71b-29eced8216f8/payload-processing/0.log"
Jul 14 14:15:48.654410 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:48.654381    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-fc4b75864-z5cd5_5e95c6d4-7c7a-4414-bed9-e57c5e5d8e68/manager/0.log"
Jul 14 14:15:49.504979 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:49.504951    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-594bccb575-kj7t8_1f9fa9ff-066a-4e48-a0c2-b7087967960b/kube-auth-proxy/0.log"
Jul 14 14:15:49.625877 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:49.625845    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-d28fh_6cb5d20f-6774-416d-a71b-29eced8216f8/payload-processing/0.log"
Jul 14 14:15:55.293170 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:55.293146    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dhtvj_88a55d43-cfd1-433c-944b-8de4720da8af/kube-multus-additional-cni-plugins/0.log"
Jul 14 14:15:55.316945 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:55.316923    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dhtvj_88a55d43-cfd1-433c-944b-8de4720da8af/egress-router-binary-copy/0.log"
Jul 14 14:15:55.340084 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:55.340060    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dhtvj_88a55d43-cfd1-433c-944b-8de4720da8af/cni-plugins/0.log"
Jul 14 14:15:55.365211 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:55.365184    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dhtvj_88a55d43-cfd1-433c-944b-8de4720da8af/bond-cni-plugin/0.log"
Jul 14 14:15:55.387575 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:55.387555    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dhtvj_88a55d43-cfd1-433c-944b-8de4720da8af/routeoverride-cni/0.log"
Jul 14 14:15:55.412800 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:55.412781    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dhtvj_88a55d43-cfd1-433c-944b-8de4720da8af/whereabouts-cni-bincopy/0.log"
Jul 14 14:15:55.448930 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:55.448912    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dhtvj_88a55d43-cfd1-433c-944b-8de4720da8af/whereabouts-cni/0.log"
Jul 14 14:15:55.901472 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:55.901393    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-mqppd_8c3ae302-40b4-42eb-9a67-5fd1cfcbd33b/kube-multus/0.log"
Jul 14 14:15:56.028469 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:56.028423    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-w4rvb_fd9c8cb5-09e3-4116-b407-01ebe6c06999/network-metrics-daemon/0.log"
Jul 14 14:15:56.047321 ip-10-0-140-78 kubenswrapper[2569]: I0714 14:15:56.047299    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-w4rvb_fd9c8cb5-09e3-4116-b407-01ebe6c06999/kube-rbac-proxy/0.log"