Jul 29 06:20:37.769898 ip-10-0-129-223 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 29 06:20:37.769912 ip-10-0-129-223 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 29 06:20:37.769922 ip-10-0-129-223 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 29 06:20:37.770270 ip-10-0-129-223 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 29 06:20:47.257679 ip-10-0-129-223 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Jul 29 06:20:47.257693 ip-10-0-129-223 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot c45d0a779f974f018f528428debf266c --
Jul 29 06:22:35.404582 ip-10-0-129-223 systemd[1]: Starting Kubernetes Kubelet...
Jul 29 06:22:35.748110 ip-10-0-129-223 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 06:22:35.748110 ip-10-0-129-223 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 29 06:22:35.748110 ip-10-0-129-223 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 06:22:35.748110 ip-10-0-129-223 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 29 06:22:35.748110 ip-10-0-129-223 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 06:22:35.750419 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.750293    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 29 06:22:35.753628 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753613    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 06:22:35.753628 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753627    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 06:22:35.753628 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753631    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 06:22:35.753722 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753634    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 06:22:35.753722 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753637    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 06:22:35.753722 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753640    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 06:22:35.753722 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753643    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 06:22:35.753722 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753645    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 06:22:35.753722 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753648    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 06:22:35.753722 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753650    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 06:22:35.753722 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753653    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 06:22:35.753722 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753655    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 06:22:35.753722 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753658    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 06:22:35.753722 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753660    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 06:22:35.753722 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753664    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 06:22:35.753722 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753668    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 06:22:35.753722 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753671    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 06:22:35.753722 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753674    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 06:22:35.753722 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753677    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 06:22:35.753722 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753684    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 06:22:35.753722 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753687    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 06:22:35.753722 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753690    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 06:22:35.754170 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753692    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 06:22:35.754170 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753695    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 06:22:35.754170 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753697    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 06:22:35.754170 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753700    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 06:22:35.754170 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753702    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 06:22:35.754170 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753705    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 06:22:35.754170 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753707    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 06:22:35.754170 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753710    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 06:22:35.754170 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753712    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 06:22:35.754170 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753715    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 06:22:35.754170 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753717    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 06:22:35.754170 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753720    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 06:22:35.754170 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753722    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 06:22:35.754170 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753725    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 06:22:35.754170 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753728    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 06:22:35.754170 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753731    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 06:22:35.754170 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753734    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 06:22:35.754170 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753736    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 06:22:35.754170 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753739    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 06:22:35.754170 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753741    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 06:22:35.754714 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753743    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 06:22:35.754714 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753746    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 06:22:35.754714 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753748    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 06:22:35.754714 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753751    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 06:22:35.754714 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753754    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 06:22:35.754714 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753757    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 06:22:35.754714 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753759    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 06:22:35.754714 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753762    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 06:22:35.754714 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753764    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 06:22:35.754714 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753766    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 06:22:35.754714 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753769    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 06:22:35.754714 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753771    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 06:22:35.754714 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753773    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 06:22:35.754714 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753776    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 06:22:35.754714 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753779    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 06:22:35.754714 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753781    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 06:22:35.754714 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753784    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 06:22:35.754714 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753786    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 06:22:35.754714 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753789    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 06:22:35.755167 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753791    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 06:22:35.755167 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753793    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 06:22:35.755167 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753796    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 06:22:35.755167 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753798    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 06:22:35.755167 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753801    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 06:22:35.755167 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753803    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 06:22:35.755167 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753806    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 06:22:35.755167 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753809    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 06:22:35.755167 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753813    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 06:22:35.755167 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753816    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 06:22:35.755167 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753820    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 06:22:35.755167 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753824    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 06:22:35.755167 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753828    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 06:22:35.755167 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753830    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 06:22:35.755167 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753833    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 06:22:35.755167 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753836    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 06:22:35.755167 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753839    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 06:22:35.755167 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753842    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 06:22:35.755167 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753845    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 06:22:35.755641 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753848    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 06:22:35.755641 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753850    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 06:22:35.755641 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753853    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 06:22:35.755641 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753856    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 06:22:35.755641 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753858    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 06:22:35.755641 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753861    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 06:22:35.755641 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753863    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 06:22:35.755641 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753866    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 06:22:35.755641 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753869    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 06:22:35.755641 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753872    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 06:22:35.755641 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753874    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 06:22:35.755641 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753877    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 06:22:35.755641 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753879    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 06:22:35.755641 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753881    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 06:22:35.755641 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753884    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 06:22:35.755641 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753886    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 06:22:35.755641 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753889    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 06:22:35.755641 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753891    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 06:22:35.755641 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.753893    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 06:22:35.755641 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755323    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 06:22:35.756120 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755330    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 06:22:35.756120 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755333    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 06:22:35.756120 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755336    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 06:22:35.756120 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755339    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 06:22:35.756120 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755343    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 06:22:35.756120 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755345    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 06:22:35.756120 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755348    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 06:22:35.756120 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755351    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 06:22:35.756120 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755353    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 06:22:35.756120 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755356    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 06:22:35.756120 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755359    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 06:22:35.756120 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755362    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 06:22:35.756120 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755364    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 06:22:35.756120 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755367    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 06:22:35.756120 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755369    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 06:22:35.756120 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755372    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 06:22:35.756120 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755374    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 06:22:35.756120 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755377    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 06:22:35.756120 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755379    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 06:22:35.756120 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755382    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 06:22:35.756629 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755386    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 06:22:35.756629 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755389    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 06:22:35.756629 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755392    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 06:22:35.756629 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755395    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 06:22:35.756629 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755397    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 06:22:35.756629 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755401    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 06:22:35.756629 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755403    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 06:22:35.756629 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755406    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 06:22:35.756629 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755408    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 06:22:35.756629 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755411    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 06:22:35.756629 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755413    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 06:22:35.756629 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755416    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 06:22:35.756629 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755419    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 06:22:35.756629 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755421    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 06:22:35.756629 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755425    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 06:22:35.756629 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755427    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 06:22:35.756629 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755430    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 06:22:35.756629 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755433    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 06:22:35.756629 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755435    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 06:22:35.756629 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755438    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 06:22:35.757154 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755440    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 06:22:35.757154 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755443    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 06:22:35.757154 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755445    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 06:22:35.757154 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755448    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 06:22:35.757154 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755450    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 06:22:35.757154 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755453    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 06:22:35.757154 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755455    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 06:22:35.757154 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755457    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 06:22:35.757154 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755460    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 06:22:35.757154 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755462    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 06:22:35.757154 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755465    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 06:22:35.757154 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755467    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 06:22:35.757154 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755469    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 06:22:35.757154 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755473    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 06:22:35.757154 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755475    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 06:22:35.757154 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755478    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 06:22:35.757154 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755480    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 06:22:35.757154 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755484    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 06:22:35.757154 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755487    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 06:22:35.757154 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755490    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 06:22:35.757673 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755493    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 06:22:35.757673 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755495    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 06:22:35.757673 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755498    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 06:22:35.757673 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755500    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 06:22:35.757673 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755503    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 06:22:35.757673 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755505    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 06:22:35.757673 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755509    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 06:22:35.757673 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755512    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 06:22:35.757673 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755515    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 06:22:35.757673 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755518    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 06:22:35.757673 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755524    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 06:22:35.757673 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755527    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 06:22:35.757673 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755529    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 06:22:35.757673 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755532    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 06:22:35.757673 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755535    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 06:22:35.757673 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755537    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 06:22:35.757673 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755540    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 06:22:35.757673 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755542    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 06:22:35.757673 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755545    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 06:22:35.757673 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755547    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 06:22:35.758162 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755550    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 06:22:35.758162 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755552    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 06:22:35.758162 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755555    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 06:22:35.758162 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755557    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 06:22:35.758162 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755559    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 06:22:35.758162 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755562    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 06:22:35.758162 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755565    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 06:22:35.758162 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755567    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 06:22:35.758162 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755570    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 06:22:35.758162 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755572    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 06:22:35.758162 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755575    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 06:22:35.758162 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755577    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 06:22:35.758162 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755579    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 06:22:35.758162 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755582    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 06:22:35.758162 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755584    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 06:22:35.758162 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755587    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 06:22:35.758162 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755589    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 06:22:35.758162 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755591    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 06:22:35.758162 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755957    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 06:22:35.758638 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755962    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 06:22:35.758638 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755966    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 06:22:35.758638 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755969    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 06:22:35.758638 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755974    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 06:22:35.758638 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755976    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 06:22:35.758638 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755979    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 06:22:35.758638 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755982    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 06:22:35.758638 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755985    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 06:22:35.758638 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755987    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 06:22:35.758638 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755990    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 06:22:35.758638 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755992    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 06:22:35.758638 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755994    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 06:22:35.758638 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755997    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 06:22:35.758638 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.755999    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 06:22:35.758638 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756002    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 06:22:35.758638 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756004    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 06:22:35.758638 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756006    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 06:22:35.758638 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756009    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 06:22:35.758638 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756012    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 06:22:35.759200 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756014    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 06:22:35.759200 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756017    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 06:22:35.759200 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756020    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 06:22:35.759200 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756022    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 06:22:35.759200 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756025    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 06:22:35.759200 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756027    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 06:22:35.759200 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756029    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 06:22:35.759200 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756032    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 06:22:35.759200 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756034    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 06:22:35.759200 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756037    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 06:22:35.759200 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756039    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 06:22:35.759200 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756044    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 06:22:35.759200 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756047    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 06:22:35.759200 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756049    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 06:22:35.759200 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756052    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 06:22:35.759200 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756054    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 06:22:35.759200 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756057    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 06:22:35.759200 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756060    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 06:22:35.759200 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756063    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 06:22:35.759675 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756065    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 06:22:35.759675 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756068    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 06:22:35.759675 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756070    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 06:22:35.759675 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756073    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 06:22:35.759675 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756075    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 06:22:35.759675 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756078    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 06:22:35.759675 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756080    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 06:22:35.759675 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756083    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 06:22:35.759675 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756086    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 06:22:35.759675 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756088    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 06:22:35.759675 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756091    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 06:22:35.759675 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756093    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 06:22:35.759675 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756095    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 06:22:35.759675 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756098    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 06:22:35.759675 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756100    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 06:22:35.759675 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756103    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 06:22:35.759675 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756105    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 06:22:35.759675 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756108    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 06:22:35.759675 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756110    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 06:22:35.759675 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756112    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 06:22:35.760161 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756115    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 06:22:35.760161 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756117    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 06:22:35.760161 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756119    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 06:22:35.760161 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756122    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 06:22:35.760161 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756125    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 06:22:35.760161 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756129    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 06:22:35.760161 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756131    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 06:22:35.760161 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756134    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 06:22:35.760161 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756136    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 06:22:35.760161 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756139    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 06:22:35.760161 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756141    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 06:22:35.760161 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756144    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 06:22:35.760161 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756146    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 06:22:35.760161 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756148    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 06:22:35.760161 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756151    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 06:22:35.760161 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756153    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 06:22:35.760161 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756156    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 06:22:35.760161 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756159    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 06:22:35.760161 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756162    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 06:22:35.760161 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756164    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 06:22:35.760674 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756167    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 06:22:35.760674 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756169    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 06:22:35.760674 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756172    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 06:22:35.760674 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756175    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 06:22:35.760674 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756177    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 06:22:35.760674 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756180    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 06:22:35.760674 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756182    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 06:22:35.760674 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756184    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 06:22:35.760674 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756187    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 06:22:35.760674 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756189    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 06:22:35.760674 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756192    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 06:22:35.760674 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756194    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 06:22:35.760674 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756196    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 06:22:35.760674 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756199    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 06:22:35.760674 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756201    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 06:22:35.760674 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756204    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 06:22:35.760674 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756206    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 06:22:35.760674 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756209    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 06:22:35.760674 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756211    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 06:22:35.760674 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756213    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 06:22:35.761161 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756290    2563 flags.go:64] FLAG: --address="0.0.0.0"
Jul 29 06:22:35.761161 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756314    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 29 06:22:35.761161 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756324    2563 flags.go:64] FLAG: --anonymous-auth="true"
Jul 29 06:22:35.761161 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756330    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 29 06:22:35.761161 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756336    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 29 06:22:35.761161 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756341    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 29 06:22:35.761161 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756353    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 29 06:22:35.761161 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756358    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 29 06:22:35.761161 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756361    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 29 06:22:35.761161 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756364    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 29 06:22:35.761161 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756367    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 29 06:22:35.761161 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756370    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 29 06:22:35.761161 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756373    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 29 06:22:35.761161 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756376    2563 flags.go:64] FLAG: --cgroup-root=""
Jul 29 06:22:35.761161 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756379    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 29 06:22:35.761161 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756381    2563 flags.go:64] FLAG: --client-ca-file=""
Jul 29 06:22:35.761161 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756384    2563 flags.go:64] FLAG: --cloud-provider="external"
Jul 29 06:22:35.761161 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756387    2563 flags.go:64] FLAG: --cluster-dns="[]"
Jul 29 06:22:35.761161 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756392    2563 flags.go:64] FLAG: --cluster-domain=""
Jul 29 06:22:35.761161 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756394    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 29 06:22:35.761161 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756397    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 29 06:22:35.761161 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756400    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 29 06:22:35.761161 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756403    2563 flags.go:64] FLAG: --container-log-max-files="5"
Jul 29 06:22:35.761161 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756407    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 29 06:22:35.761754 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756410    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 29 06:22:35.761754 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756413    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 29 06:22:35.761754 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756416    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 29 06:22:35.761754 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756419    2563 flags.go:64] FLAG: --contention-profiling="false"
Jul 29 06:22:35.761754 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756422    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 29 06:22:35.761754 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756424    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 29 06:22:35.761754 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756427    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 29 06:22:35.761754 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756430    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 29 06:22:35.761754 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756434    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 29 06:22:35.761754 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756440    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 29 06:22:35.761754 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756443    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 29 06:22:35.761754 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756446    2563 flags.go:64] FLAG: --enable-load-reader="false"
Jul 29 06:22:35.761754 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756449    2563 flags.go:64] FLAG: --enable-server="true"
Jul 29 06:22:35.761754 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756452    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 29 06:22:35.761754 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756456    2563 flags.go:64] FLAG: --event-burst="100"
Jul 29 06:22:35.761754 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756460    2563 flags.go:64] FLAG: --event-qps="50"
Jul 29 06:22:35.761754 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756462    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 29 06:22:35.761754 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756465    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 29 06:22:35.761754 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756468    2563 flags.go:64] FLAG: --eviction-hard=""
Jul 29 06:22:35.761754 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756472    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 29 06:22:35.761754 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756475    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 29 06:22:35.761754 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756477    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 29 06:22:35.761754 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756480    2563 flags.go:64] FLAG: --eviction-soft=""
Jul 29 06:22:35.761754 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756483    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 29 06:22:35.761754 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756485    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 29 06:22:35.762433 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756488    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 29 06:22:35.762433 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756491    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 29 06:22:35.762433 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756494    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 29 06:22:35.762433 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756496    2563 flags.go:64] FLAG: --fail-swap-on="true"
Jul 29 06:22:35.762433 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756499    2563 flags.go:64] FLAG: --feature-gates=""
Jul 29 06:22:35.762433 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756502    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 29 06:22:35.762433 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756505    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 29 06:22:35.762433 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756508    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 29 06:22:35.762433 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756511    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 29 06:22:35.762433 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756514    2563 flags.go:64] FLAG: --healthz-port="10248"
Jul 29 06:22:35.762433 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756517    2563 flags.go:64] FLAG: --help="false"
Jul 29 06:22:35.762433 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756520    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-129-223.ec2.internal"
Jul 29 06:22:35.762433 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756522    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 29 06:22:35.762433 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756525    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 29 06:22:35.762433 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756528    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 29 06:22:35.762433 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756531    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 29 06:22:35.762433 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756535    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 29 06:22:35.762433 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756538    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 29 06:22:35.762433 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756541    2563 flags.go:64] FLAG: --image-service-endpoint=""
Jul 29 06:22:35.762433 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756544    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 29 06:22:35.762433 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756547    2563 flags.go:64] FLAG: --kube-api-burst="100"
Jul 29 06:22:35.762433 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756551    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 29 06:22:35.762433 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756554    2563 flags.go:64] FLAG: --kube-api-qps="50"
Jul 29 06:22:35.762433 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756556    2563 flags.go:64] FLAG: --kube-reserved=""
Jul 29 06:22:35.763042 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756559    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 29 06:22:35.763042 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756562    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 29 06:22:35.763042 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756565    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 29 06:22:35.763042 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756568    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 29 06:22:35.763042 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756571    2563 flags.go:64] FLAG: --lock-file=""
Jul 29 06:22:35.763042 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756573    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 29 06:22:35.763042 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756576    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 29 06:22:35.763042 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756578    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 29 06:22:35.763042 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756584    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 29 06:22:35.763042 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756586    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 29 06:22:35.763042 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756589    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 29 06:22:35.763042 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756592    2563 flags.go:64] FLAG: --logging-format="text"
Jul 29 06:22:35.763042 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756594    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 29 06:22:35.763042 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756598    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 29 06:22:35.763042 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756601    2563 flags.go:64] FLAG: --manifest-url=""
Jul 29 06:22:35.763042 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756603    2563 flags.go:64] FLAG: --manifest-url-header=""
Jul 29 06:22:35.763042 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756607    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 29 06:22:35.763042 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756611    2563 flags.go:64] FLAG: --max-open-files="1000000"
Jul 29 06:22:35.763042 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756614    2563 flags.go:64] FLAG: --max-pods="110"
Jul 29 06:22:35.763042 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756617    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 29 06:22:35.763042 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756620    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 29 06:22:35.763042 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756623    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 29 06:22:35.763042 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756626    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 29 06:22:35.763042 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756629    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 29 06:22:35.763042 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756633    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 29 06:22:35.763664 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756636    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 29 06:22:35.763664 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756643    2563 flags.go:64] FLAG: --node-status-max-images="50"
Jul 29 06:22:35.763664 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756646    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 29 06:22:35.763664 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756649    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 29 06:22:35.763664 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756652    2563 flags.go:64] FLAG: --pod-cidr=""
Jul 29 06:22:35.763664 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756655    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 29 06:22:35.763664 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756660    2563 flags.go:64] FLAG: --pod-manifest-path=""
Jul 29 06:22:35.763664 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756663    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 29 06:22:35.763664 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756666    2563 flags.go:64] FLAG: --pods-per-core="0"
Jul 29 06:22:35.763664 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756668    2563 flags.go:64] FLAG: --port="10250"
Jul 29 06:22:35.763664 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756671    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 29 06:22:35.763664 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756674    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02e92988410560108"
Jul 29 06:22:35.763664 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756677    2563 flags.go:64] FLAG: --qos-reserved=""
Jul 29 06:22:35.763664 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756680    2563 flags.go:64] FLAG: --read-only-port="10255"
Jul 29 06:22:35.763664 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756683    2563 flags.go:64] FLAG: --register-node="true"
Jul 29 06:22:35.763664 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756685    2563 flags.go:64] FLAG: --register-with-taints=""
Jul 29 06:22:35.763664 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756689    2563 flags.go:64] FLAG: --registry-burst="10"
Jul 29 06:22:35.763664 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756692    2563 flags.go:64] FLAG: --registry-qps="5"
Jul 29 06:22:35.763664 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756694    2563 flags.go:64] FLAG: --reserved-cpus=""
Jul 29 06:22:35.763664 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756697    2563 flags.go:64] FLAG: --reserved-memory=""
Jul 29 06:22:35.763664 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756700    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 29 06:22:35.763664 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756703    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 29 06:22:35.763664 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756706    2563 flags.go:64] FLAG: --rotate-certificates="false"
Jul 29 06:22:35.763664 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756709    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 29 06:22:35.763664 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756712    2563 flags.go:64] FLAG: --runonce="false"
Jul 29 06:22:35.764267 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756715    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 29 06:22:35.764267 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756718    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 29 06:22:35.764267 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756721    2563 flags.go:64] FLAG: --seccomp-default="false"
Jul 29 06:22:35.764267 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756723    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 29 06:22:35.764267 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756726    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 29 06:22:35.764267 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756729    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 29 06:22:35.764267 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756732    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 29 06:22:35.764267 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756736    2563 flags.go:64] FLAG: --storage-driver-password="root"
Jul 29 06:22:35.764267 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756738    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 29 06:22:35.764267 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756741    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 29 06:22:35.764267 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756744    2563 flags.go:64] FLAG: --storage-driver-user="root"
Jul 29 06:22:35.764267 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756746    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 29 06:22:35.764267 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756749    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 29 06:22:35.764267 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756753    2563 flags.go:64] FLAG: --system-cgroups=""
Jul 29 06:22:35.764267 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756756    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 29 06:22:35.764267 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756761    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 29 06:22:35.764267 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756763    2563 flags.go:64] FLAG: --tls-cert-file=""
Jul 29 06:22:35.764267 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756766    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 29 06:22:35.764267 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756770    2563 flags.go:64] FLAG: --tls-min-version=""
Jul 29 06:22:35.764267 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756773    2563 flags.go:64] FLAG: --tls-private-key-file=""
Jul 29 06:22:35.764267 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756776    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 29 06:22:35.764267 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756778    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 29 06:22:35.764267 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756781    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 29 06:22:35.764267 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756784    2563 flags.go:64] FLAG: --v="2"
Jul 29 06:22:35.764267 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756787    2563 flags.go:64] FLAG: --version="false"
Jul 29 06:22:35.764891 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756791    2563 flags.go:64] FLAG: --vmodule=""
Jul 29 06:22:35.764891 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756796    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 29 06:22:35.764891 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.756799    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 29 06:22:35.764891 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756913    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 06:22:35.764891 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756918    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 06:22:35.764891 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756921    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 06:22:35.764891 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756924    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 06:22:35.764891 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756927    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 06:22:35.764891 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756929    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 06:22:35.764891 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756932    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 06:22:35.764891 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756934    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 06:22:35.764891 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756936    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 06:22:35.764891 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756939    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 06:22:35.764891 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756942    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 06:22:35.764891 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756945    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 06:22:35.764891 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756948    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 06:22:35.764891 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756950    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 06:22:35.764891 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756953    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 06:22:35.764891 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756955    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 06:22:35.764891 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756958    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 06:22:35.765394 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756960    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 06:22:35.765394 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756963    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 06:22:35.765394 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756966    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 06:22:35.765394 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756968    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 06:22:35.765394 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756971    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 06:22:35.765394 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756973    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 06:22:35.765394 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756976    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 06:22:35.765394 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756980    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 06:22:35.765394 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756983    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 06:22:35.765394 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756986    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 06:22:35.765394 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756989    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 06:22:35.765394 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756991    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 06:22:35.765394 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756993    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 06:22:35.765394 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.756997    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 06:22:35.765394 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757000    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 06:22:35.765394 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757002    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 06:22:35.765394 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757005    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 06:22:35.765394 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757007    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 06:22:35.765394 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757010    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 06:22:35.765394 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757012    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 06:22:35.765883 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757014    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 06:22:35.765883 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757017    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 06:22:35.765883 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757019    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 06:22:35.765883 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757022    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 06:22:35.765883 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757025    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 06:22:35.765883 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757029    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 06:22:35.765883 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757033    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 06:22:35.765883 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757036    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 06:22:35.765883 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757039    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 06:22:35.765883 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757041    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 06:22:35.765883 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757044    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 06:22:35.765883 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757046    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 06:22:35.765883 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757049    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 06:22:35.765883 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757051    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 06:22:35.765883 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757054    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 06:22:35.765883 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757057    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 06:22:35.765883 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757059    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 06:22:35.765883 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757062    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 06:22:35.765883 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757064    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 06:22:35.765883 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757066    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 06:22:35.766427 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757069    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 06:22:35.766427 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757071    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 06:22:35.766427 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757082    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 06:22:35.766427 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757084    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 06:22:35.766427 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757087    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 06:22:35.766427 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757091    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 06:22:35.766427 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757093    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 06:22:35.766427 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757095    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 06:22:35.766427 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757098    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 06:22:35.766427 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757100    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 06:22:35.766427 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757103    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 06:22:35.766427 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757107    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 06:22:35.766427 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757109    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 06:22:35.766427 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757112    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 06:22:35.766427 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757129    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 06:22:35.766427 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757133    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 06:22:35.766427 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757136    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 06:22:35.766427 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757139    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 06:22:35.766427 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757143    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 06:22:35.766427 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757145    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 06:22:35.766903 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757147    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 06:22:35.766903 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757150    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 06:22:35.766903 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757152    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 06:22:35.766903 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757155    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 06:22:35.766903 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757157    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 06:22:35.766903 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757160    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 06:22:35.766903 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757163    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 06:22:35.766903 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757166    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 06:22:35.766903 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757169    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 06:22:35.766903 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757171    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 06:22:35.766903 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757174    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 06:22:35.766903 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757176    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 06:22:35.766903 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757179    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 06:22:35.766903 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757181    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 06:22:35.766903 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757183    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 06:22:35.766903 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757186    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 06:22:35.766903 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757188    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 06:22:35.766903 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757192    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 06:22:35.766903 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757195    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 06:22:35.766903 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757197    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 06:22:35.767407 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757199    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 06:22:35.767407 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.757202    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 06:22:35.767407 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.757206    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 06:22:35.767407 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.764261    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 29 06:22:35.767407 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.764278    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 29 06:22:35.767407 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.765052    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 29 06:22:35.767407 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.765066    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 29 06:22:35.767407 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765134    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 06:22:35.767407 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765140    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 06:22:35.767407 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765143    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 06:22:35.767407 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765146    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 06:22:35.767407 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765150    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 06:22:35.767407 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765152    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 06:22:35.767407 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765155    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 06:22:35.767750 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765158    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 06:22:35.767750 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765161    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 06:22:35.767750 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765164    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 06:22:35.767750 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765167    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 06:22:35.767750 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765171    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 06:22:35.767750 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765176    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 06:22:35.767750 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765179    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 06:22:35.767750 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765181    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 06:22:35.767750 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765184    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 06:22:35.767750 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765187    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 06:22:35.767750 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765190    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 06:22:35.767750 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765192    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 06:22:35.767750 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765195    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 06:22:35.767750 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765198    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 06:22:35.767750 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765201    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 06:22:35.767750 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765203    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 06:22:35.767750 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765206    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 06:22:35.767750 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765208    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 06:22:35.767750 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765211    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 06:22:35.767750 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765213    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 06:22:35.768278 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765216    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 06:22:35.768278 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765219    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 06:22:35.768278 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765228    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 06:22:35.768278 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765231    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 06:22:35.768278 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765233    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 06:22:35.768278 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765236    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 06:22:35.768278 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765239    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 06:22:35.768278 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765241    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 06:22:35.768278 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765244    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 06:22:35.768278 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765246    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 06:22:35.768278 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765248    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 06:22:35.768278 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765251    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 06:22:35.768278 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765254    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 06:22:35.768278 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765256    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 06:22:35.768278 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765258    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 06:22:35.768278 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765261    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 06:22:35.768278 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765264    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 06:22:35.768278 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765266    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 06:22:35.768278 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765268    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 06:22:35.768278 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765271    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 06:22:35.768894 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765273    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 06:22:35.768894 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765276    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 06:22:35.768894 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765278    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 06:22:35.768894 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765280    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 06:22:35.768894 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765283    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 06:22:35.768894 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765286    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 06:22:35.768894 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765288    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 06:22:35.768894 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765292    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 06:22:35.768894 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765296    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 06:22:35.768894 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765298    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 06:22:35.768894 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765313    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 06:22:35.768894 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765317    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 06:22:35.768894 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765319    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 06:22:35.768894 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765322    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 06:22:35.768894 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765326    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 06:22:35.768894 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765334    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 06:22:35.768894 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765337    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 06:22:35.768894 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765339    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 06:22:35.768894 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765342    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 06:22:35.768894 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765344    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 06:22:35.769511 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765347    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 06:22:35.769511 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765351    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 06:22:35.769511 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765354    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 06:22:35.769511 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765357    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 06:22:35.769511 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765360    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 06:22:35.769511 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765362    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 06:22:35.769511 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765366    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 06:22:35.769511 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765368    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 06:22:35.769511 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765371    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 06:22:35.769511 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765374    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 06:22:35.769511 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765376    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 06:22:35.769511 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765379    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 06:22:35.769511 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765382    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 06:22:35.769511 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765384    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 06:22:35.769511 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765387    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 06:22:35.769511 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765390    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 06:22:35.769511 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765392    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 06:22:35.769511 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765394    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 06:22:35.769511 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765397    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 06:22:35.770028 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765400    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 06:22:35.770028 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765402    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 06:22:35.770028 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765405    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 06:22:35.770028 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765408    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 06:22:35.770028 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765411    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 06:22:35.770028 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765413    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 06:22:35.770028 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765416    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 06:22:35.770028 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765419    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 06:22:35.770028 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765422    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 06:22:35.770028 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765429    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 06:22:35.770028 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765432    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 06:22:35.770028 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765435    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 06:22:35.770028 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765437    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 06:22:35.770028 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.765442    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 06:22:35.770028 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765596    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 06:22:35.770444 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765606    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 06:22:35.770444 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765609    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 06:22:35.770444 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765612    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 06:22:35.770444 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765615    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 06:22:35.770444 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765618    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 06:22:35.770444 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765621    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 06:22:35.770444 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765624    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 06:22:35.770444 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765626    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 06:22:35.770444 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765629    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 06:22:35.770444 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765632    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 06:22:35.770444 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765634    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 06:22:35.770444 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765637    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 06:22:35.770444 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765639    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 06:22:35.770444 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765642    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 06:22:35.770444 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765644    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 06:22:35.770444 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765646    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 06:22:35.770444 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765649    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 06:22:35.770444 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765651    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 06:22:35.770444 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765653    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 06:22:35.770444 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765656    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 06:22:35.771254 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765658    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 06:22:35.771254 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765661    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 06:22:35.771254 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765663    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 06:22:35.771254 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765665    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 06:22:35.771254 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765668    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 06:22:35.771254 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765671    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 06:22:35.771254 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765674    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 06:22:35.771254 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765685    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 06:22:35.771254 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765688    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 06:22:35.771254 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765691    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 06:22:35.771254 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765693    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 06:22:35.771254 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765696    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 06:22:35.771254 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765700    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 06:22:35.771254 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765703    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 06:22:35.771254 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765705    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 06:22:35.771254 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765708    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 06:22:35.771254 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765711    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 06:22:35.771254 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765713    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 06:22:35.771254 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765716    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 06:22:35.772033 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765719    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 06:22:35.772033 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765721    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 06:22:35.772033 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765724    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 06:22:35.772033 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765727    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 06:22:35.772033 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765729    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 06:22:35.772033 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765731    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 06:22:35.772033 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765734    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 06:22:35.772033 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765736    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 06:22:35.772033 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765739    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 06:22:35.772033 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765741    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 06:22:35.772033 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765744    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 06:22:35.772033 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765746    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 06:22:35.772033 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765748    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 06:22:35.772033 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765751    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 06:22:35.772033 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765753    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 06:22:35.772033 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765756    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 06:22:35.772033 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765758    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 06:22:35.772033 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765761    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 06:22:35.772033 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765764    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 06:22:35.772033 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765766    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 06:22:35.772685 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765768    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 06:22:35.772685 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765777    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 06:22:35.772685 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765779    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 06:22:35.772685 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765782    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 06:22:35.772685 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765785    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 06:22:35.772685 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765787    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 06:22:35.772685 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765790    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 06:22:35.772685 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765792    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 06:22:35.772685 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765794    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 06:22:35.772685 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765797    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 06:22:35.772685 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765799    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 06:22:35.772685 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765801    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 06:22:35.772685 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765804    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 06:22:35.772685 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765806    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 06:22:35.772685 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765808    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 06:22:35.772685 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765812    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 06:22:35.772685 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765815    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 06:22:35.772685 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765818    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 06:22:35.772685 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765821    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 06:22:35.773442 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765824    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 06:22:35.773442 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765826    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 06:22:35.773442 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765829    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 06:22:35.773442 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765831    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 06:22:35.773442 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765834    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 06:22:35.773442 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765836    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 06:22:35.773442 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765838    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 06:22:35.773442 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765840    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 06:22:35.773442 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765843    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 06:22:35.773442 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765845    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 06:22:35.773442 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765847    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 06:22:35.773442 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765850    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 06:22:35.773442 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765852    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 06:22:35.773442 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765855    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 06:22:35.773442 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765857    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 06:22:35.773442 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765865    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 06:22:35.773442 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765867    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 06:22:35.773442 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765870    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 06:22:35.773442 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765872    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 06:22:35.773442 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:35.765874    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 06:22:35.773957 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.765879    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 06:22:35.773957 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.766000    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 29 06:22:35.773957 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.767892    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 29 06:22:35.773957 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.769172    2563 server.go:1019] "Starting client certificate rotation"
Jul 29 06:22:35.773957 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.769344    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 06:22:35.773957 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.769398    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 06:22:35.773957 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.773036    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 06:22:35.774352 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.774334    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 06:22:35.785342 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.785321    2563 log.go:25] "Validated CRI v1 runtime API"
Jul 29 06:22:35.787249 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.787235    2563 log.go:25] "Validated CRI v1 image API"
Jul 29 06:22:35.790148 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.790119    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 29 06:22:35.792622 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.792603    2563 fs.go:135] Filesystem UUIDs: map[4c21f7ba-66d9-4e2d-8241-12877e7d87ad:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c70db154-53a6-4766-9df8-b41c99a90ecf:/dev/nvme0n1p4]
Jul 29 06:22:35.792707 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.792621    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 29 06:22:35.797568 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.797459    2563 manager.go:216] Machine: {Timestamp:2026-07-29 06:22:35.796212151 +0000 UTC m=+0.288392036 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103031 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2eeab5e4b38963ff5cecd32a7d353e SystemUUID:ec2eeab5-e4b3-8963-ff5c-ecd32a7d353e BootID:c45d0a77-9f97-4f01-8f52-8428debf266c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1d:74:65:44:b5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1d:74:65:44:b5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:e6:de:8d:b8:0e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 29 06:22:35.797568 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.797559    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 29 06:22:35.797731 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.797658    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 29 06:22:35.800244 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.800214    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 29 06:22:35.800441 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.800246    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-223.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 29 06:22:35.801033 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.801021    2563 topology_manager.go:138] "Creating topology manager with none policy"
Jul 29 06:22:35.801097 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.801038    2563 container_manager_linux.go:309] "Creating device plugin manager"
Jul 29 06:22:35.801097 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.801055    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 06:22:35.801770 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.801758    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 06:22:35.802132 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.802113    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 06:22:35.802515 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.802502    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 29 06:22:35.804008 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.803993    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 06:22:35.804133 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.804122    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 29 06:22:35.804204 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.804194    2563 kubelet.go:486] "Attempting to sync node with API server"
Jul 29 06:22:35.804252 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.804217    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 29 06:22:35.804252 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.804237    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 29 06:22:35.804252 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.804249    2563 kubelet.go:398] "Adding apiserver pod source"
Jul 29 06:22:35.804405 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.804270    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 29 06:22:35.804830 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.804809    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 06:22:35.804830 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.804832    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 06:22:35.806949 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.806925    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 29 06:22:35.808838 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.808825    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 29 06:22:35.808881 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.808855    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 29 06:22:35.810360 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.810348    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 29 06:22:35.810417 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.810364    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 29 06:22:35.810417 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.810371    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 29 06:22:35.810417 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.810376    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 29 06:22:35.810417 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.810384    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 29 06:22:35.810417 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.810389    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 29 06:22:35.810417 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.810400    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 29 06:22:35.810417 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.810409    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 29 06:22:35.810417 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.810418    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 29 06:22:35.810632 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.810423    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 29 06:22:35.810632 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.810432    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 29 06:22:35.810632 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.810437    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 29 06:22:35.811211 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.811200    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 29 06:22:35.811243 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.811213    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 29 06:22:35.815372 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.815337    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-223.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 29 06:22:35.815452 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:35.815397    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 29 06:22:35.815452 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:35.815432    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-223.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 29 06:22:35.815618 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.815606    2563 server.go:1268] "Started kubelet"
Jul 29 06:22:35.815741 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.815703    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 29 06:22:35.815791 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.815772    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 29 06:22:35.816282 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.816257    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 29 06:22:35.816503 ip-10-0-129-223 systemd[1]: Started Kubernetes Kubelet.
Jul 29 06:22:35.817992 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.817978    2563 server.go:310] "Adding debug handlers to kubelet server"
Jul 29 06:22:35.820033 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.820014    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 29 06:22:35.820356 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:35.820332    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 29 06:22:35.823471 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:35.822739    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-223.ec2.internal.18c6ae7d60b439ed  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-223.ec2.internal,UID:ip-10-0-129-223.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-223.ec2.internal,},FirstTimestamp:2026-07-29 06:22:35.815549421 +0000 UTC m=+0.307729310,LastTimestamp:2026-07-29 06:22:35.815549421 +0000 UTC m=+0.307729310,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-223.ec2.internal,}"
Jul 29 06:22:35.825649 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.825636    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 29 06:22:35.825693 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.825645    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 29 06:22:35.826033 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:35.825785    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-223.ec2.internal\" not found"
Jul 29 06:22:35.826280 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.826263    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 29 06:22:35.826280 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.826263    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 29 06:22:35.826425 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.826290    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 29 06:22:35.826486 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.826469    2563 reconstruct.go:97] "Volume reconstruction finished"
Jul 29 06:22:35.826486 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.826478    2563 reconciler.go:29] "Reconciler: start to sync state"
Jul 29 06:22:35.826563 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.826550    2563 factory.go:55] Registering systemd factory
Jul 29 06:22:35.826604 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.826564    2563 factory.go:223] Registration of the systemd container factory successfully
Jul 29 06:22:35.826787 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.826761    2563 factory.go:153] Registering CRI-O factory
Jul 29 06:22:35.826787 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.826781    2563 factory.go:223] Registration of the crio container factory successfully
Jul 29 06:22:35.826925 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.826831    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 29 06:22:35.826925 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.826855    2563 factory.go:103] Registering Raw factory
Jul 29 06:22:35.826925 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.826877    2563 manager.go:1196] Started watching for new ooms in manager
Jul 29 06:22:35.827404 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.827383    2563 manager.go:349] Starting recovery of all containers
Jul 29 06:22:35.835776 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:35.835749    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-223.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 29 06:22:35.835866 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:35.835842    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 29 06:22:35.839090 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.839073    2563 manager.go:354] Recovery completed
Jul 29 06:22:35.843114 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.843102    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 06:22:35.844965 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.844948    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-223.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 06:22:35.845024 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.844979    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-223.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 06:22:35.845024 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.844990    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-223.ec2.internal" event="NodeHasSufficientPID"
Jul 29 06:22:35.845476 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.845459    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 29 06:22:35.845476 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.845475    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 29 06:22:35.845547 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.845486    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 06:22:35.845547 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.845492    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 06:22:35.846181 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.846166    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-223.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 06:22:35.846238 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.846196    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-223.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 06:22:35.846238 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.846213    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-223.ec2.internal" event="NodeHasSufficientPID"
Jul 29 06:22:35.847089 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.847070    2563 policy_none.go:49] "None policy: Start"
Jul 29 06:22:35.847164 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.847095    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 29 06:22:35.847164 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.847109    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 29 06:22:35.848380 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.848367    2563 policy_none.go:47] "Start"
Jul 29 06:22:35.856232 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:35.856078    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-223.ec2.internal.18c6ae7d627512c2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-223.ec2.internal,UID:ip-10-0-129-223.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-223.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-223.ec2.internal,},FirstTimestamp:2026-07-29 06:22:35.844965058 +0000 UTC m=+0.337144943,LastTimestamp:2026-07-29 06:22:35.844965058 +0000 UTC m=+0.337144943,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-223.ec2.internal,}"
Jul 29 06:22:35.863412 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:35.863353    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-223.ec2.internal.18c6ae7d62755d81  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-223.ec2.internal,UID:ip-10-0-129-223.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-223.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-223.ec2.internal,},FirstTimestamp:2026-07-29 06:22:35.844984193 +0000 UTC m=+0.337164077,LastTimestamp:2026-07-29 06:22:35.844984193 +0000 UTC m=+0.337164077,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-223.ec2.internal,}"
Jul 29 06:22:35.873509 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:35.873447    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-223.ec2.internal.18c6ae7d62758382  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-223.ec2.internal,UID:ip-10-0-129-223.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-223.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-223.ec2.internal,},FirstTimestamp:2026-07-29 06:22:35.844993922 +0000 UTC m=+0.337173807,LastTimestamp:2026-07-29 06:22:35.844993922 +0000 UTC m=+0.337173807,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-223.ec2.internal,}"
Jul 29 06:22:35.883280 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:35.883223    2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-223.ec2.internal.18c6ae7d627512c2\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-223.ec2.internal.18c6ae7d627512c2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-223.ec2.internal,UID:ip-10-0-129-223.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-223.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-223.ec2.internal,},FirstTimestamp:2026-07-29 06:22:35.844965058 +0000 UTC m=+0.337144943,LastTimestamp:2026-07-29 06:22:35.846180875 +0000 UTC m=+0.338360767,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-223.ec2.internal,}"
Jul 29 06:22:35.887548 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.887532    2563 manager.go:337] "Starting Device Plugin manager"
Jul 29 06:22:35.887606 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:35.887581    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 29 06:22:35.887606 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.887593    2563 server.go:87] "Starting device plugin registration server"
Jul 29 06:22:35.887828 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.887817    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 29 06:22:35.887876 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.887831    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 29 06:22:35.888182 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.887986    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 29 06:22:35.888182 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.888082    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 29 06:22:35.888182 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.888090    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 29 06:22:35.888546 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:35.888527    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 29 06:22:35.888621 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:35.888571    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-223.ec2.internal\" not found"
Jul 29 06:22:35.889531 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:35.889472    2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-223.ec2.internal.18c6ae7d62755d81\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-223.ec2.internal.18c6ae7d62755d81  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-223.ec2.internal,UID:ip-10-0-129-223.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-223.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-223.ec2.internal,},FirstTimestamp:2026-07-29 06:22:35.844984193 +0000 UTC m=+0.337164077,LastTimestamp:2026-07-29 06:22:35.846203736 +0000 UTC m=+0.338383622,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-223.ec2.internal,}"
Jul 29 06:22:35.898059 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.898043    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-flzkj"
Jul 29 06:22:35.898677 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:35.898625    2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-223.ec2.internal.18c6ae7d62758382\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-223.ec2.internal.18c6ae7d62758382  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-223.ec2.internal,UID:ip-10-0-129-223.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-223.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-223.ec2.internal,},FirstTimestamp:2026-07-29 06:22:35.844993922 +0000 UTC m=+0.337173807,LastTimestamp:2026-07-29 06:22:35.846218759 +0000 UTC m=+0.338398650,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-223.ec2.internal,}"
Jul 29 06:22:35.905866 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.905851    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-flzkj"
Jul 29 06:22:35.974991 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.974959    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 29 06:22:35.977099 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.976237    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 29 06:22:35.977099 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.976270    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 29 06:22:35.977099 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.976291    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 29 06:22:35.977099 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:35.976342    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 29 06:22:35.978921 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.978902    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 06:22:35.988125 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.988111    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 06:22:35.989045 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.989031    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-223.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 06:22:35.989114 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.989058    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-223.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 06:22:35.989114 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.989071    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-223.ec2.internal" event="NodeHasSufficientPID"
Jul 29 06:22:35.989114 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.989104    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-223.ec2.internal"
Jul 29 06:22:35.996562 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:35.996549    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-223.ec2.internal"
Jul 29 06:22:35.996623 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:35.996568    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-223.ec2.internal\": node \"ip-10-0-129-223.ec2.internal\" not found"
Jul 29 06:22:36.045819 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:36.045766    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-223.ec2.internal\" not found"
Jul 29 06:22:36.076822 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:36.076790    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-223.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-223.ec2.internal"]
Jul 29 06:22:36.076885 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:36.076871    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 06:22:36.078261 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:36.078248    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-223.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 06:22:36.078328 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:36.078275    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-223.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 06:22:36.078328 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:36.078284    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-223.ec2.internal" event="NodeHasSufficientPID"
Jul 29 06:22:36.080521 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:36.080510    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 06:22:36.080648 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:36.080627    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-223.ec2.internal"
Jul 29 06:22:36.080682 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:36.080667    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 06:22:36.081132 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:36.081119    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-223.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 06:22:36.081132 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:36.081129    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-223.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 06:22:36.081227 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:36.081147    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-223.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 06:22:36.081227 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:36.081152    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-223.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 06:22:36.081227 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:36.081162    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-223.ec2.internal" event="NodeHasSufficientPID"
Jul 29 06:22:36.081227 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:36.081166    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-223.ec2.internal" event="NodeHasSufficientPID"
Jul 29 06:22:36.083928 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:36.083913    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-223.ec2.internal"
Jul 29 06:22:36.083969 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:36.083940    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 06:22:36.084515 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:36.084495    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-223.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 06:22:36.084602 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:36.084524    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-223.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 06:22:36.084602 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:36.084535    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-223.ec2.internal" event="NodeHasSufficientPID"
Jul 29 06:22:36.110826 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:36.110802    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-223.ec2.internal\" not found" node="ip-10-0-129-223.ec2.internal"
Jul 29 06:22:36.115103 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:36.115084    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-223.ec2.internal\" not found" node="ip-10-0-129-223.ec2.internal"
Jul 29 06:22:36.128291 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:36.128274    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bc0f4deacee416934506716d269ff92f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-223.ec2.internal\" (UID: \"bc0f4deacee416934506716d269ff92f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-223.ec2.internal"
Jul 29 06:22:36.128379 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:36.128297    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc0f4deacee416934506716d269ff92f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-223.ec2.internal\" (UID: \"bc0f4deacee416934506716d269ff92f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-223.ec2.internal"
Jul 29 06:22:36.128416 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:36.128381    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2ff94682e942fadc8425e32a6af57aed-config\") pod \"kube-apiserver-proxy-ip-10-0-129-223.ec2.internal\" (UID: \"2ff94682e942fadc8425e32a6af57aed\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-223.ec2.internal"
Jul 29 06:22:36.146499 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:36.146479    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-223.ec2.internal\" not found"
Jul 29 06:22:36.228628 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:36.228590    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2ff94682e942fadc8425e32a6af57aed-config\") pod \"kube-apiserver-proxy-ip-10-0-129-223.ec2.internal\" (UID: \"2ff94682e942fadc8425e32a6af57aed\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-223.ec2.internal"
Jul 29 06:22:36.228628 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:36.228634    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bc0f4deacee416934506716d269ff92f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-223.ec2.internal\" (UID: \"bc0f4deacee416934506716d269ff92f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-223.ec2.internal"
Jul 29 06:22:36.228780 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:36.228655    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc0f4deacee416934506716d269ff92f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-223.ec2.internal\" (UID: \"bc0f4deacee416934506716d269ff92f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-223.ec2.internal"
Jul 29 06:22:36.228780 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:36.228694    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc0f4deacee416934506716d269ff92f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-223.ec2.internal\" (UID: \"bc0f4deacee416934506716d269ff92f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-223.ec2.internal"
Jul 29 06:22:36.228780 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:36.228702    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2ff94682e942fadc8425e32a6af57aed-config\") pod \"kube-apiserver-proxy-ip-10-0-129-223.ec2.internal\" (UID: \"2ff94682e942fadc8425e32a6af57aed\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-223.ec2.internal"
Jul 29 06:22:36.228780 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:36.228708    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bc0f4deacee416934506716d269ff92f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-223.ec2.internal\" (UID: \"bc0f4deacee416934506716d269ff92f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-223.ec2.internal"
Jul 29 06:22:36.246719 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:36.246696    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-223.ec2.internal\" not found"
Jul 29 06:22:36.347589 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:36.347522    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-223.ec2.internal\" not found"
Jul 29 06:22:36.413741 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:36.413713    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-223.ec2.internal"
Jul 29 06:22:36.417346 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:36.417325    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-223.ec2.internal"
Jul 29 06:22:36.447668 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:36.447638    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-223.ec2.internal\" not found"
Jul 29 06:22:36.548182 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:36.548139    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-223.ec2.internal\" not found"
Jul 29 06:22:36.648697 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:36.648622    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-223.ec2.internal\" not found"
Jul 29 06:22:38.163842 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:38.163801    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-223.ec2.internal\" not found"
Jul 29 06:22:38.183984 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:38.183958    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 29 06:22:38.184116 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:38.184101    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 06:22:38.263948 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:38.263921    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-223.ec2.internal\" not found"
Jul 29 06:22:38.324067 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:38.323987    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 06:17:37 +0000 UTC" deadline="2028-02-10 22:09:02.811284435 +0000 UTC"
Jul 29 06:22:38.324067 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:38.324028    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13479h46m24.487261045s"
Jul 29 06:22:38.338148 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:38.338124    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 06:22:38.340955 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:38.340935    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-223.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 06:22:38.341068 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:38.340968    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-223.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 06:22:38.341068 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:38.340983    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-223.ec2.internal" event="NodeHasSufficientPID"
Jul 29 06:22:38.362252 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:38.362234    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 06:22:38.412114 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:38.412092    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 06:22:38.433326 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:38.433286    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc0f4deacee416934506716d269ff92f.slice/crio-9e07c456b1c96fc04c5446215d01170d595890737685e26992e41664b00401a9 WatchSource:0}: Error finding container 9e07c456b1c96fc04c5446215d01170d595890737685e26992e41664b00401a9: Status 404 returned error can't find the container with id 9e07c456b1c96fc04c5446215d01170d595890737685e26992e41664b00401a9
Jul 29 06:22:38.433561 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:38.433542    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff94682e942fadc8425e32a6af57aed.slice/crio-20c79b2bc4e0a53c9e7d167d1d4a32741a09293f935980b0beb1a92b7f2bd042 WatchSource:0}: Error finding container 20c79b2bc4e0a53c9e7d167d1d4a32741a09293f935980b0beb1a92b7f2bd042: Status 404 returned error can't find the container with id 20c79b2bc4e0a53c9e7d167d1d4a32741a09293f935980b0beb1a92b7f2bd042
Jul 29 06:22:38.438526 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:38.438512    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 06:22:38.441262 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:38.441245    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-223.ec2.internal"
Jul 29 06:22:38.450955 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:38.450931    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 06:22:38.452116 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:38.452095    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-223.ec2.internal"
Jul 29 06:22:38.459541 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:38.459526    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 06:22:38.756386 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:38.756315    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 06:22:39.220009 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.219675    2563 apiserver.go:52] "Watching apiserver"
Jul 29 06:22:39.225125 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.225097    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 29 06:22:39.225491 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.225470    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-wtdbb","openshift-multus/network-metrics-daemon-vcg74","kube-system/konnectivity-agent-wfqxw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqpxr","openshift-cluster-node-tuning-operator/tuned-7b9qz","openshift-network-diagnostics/network-check-target-njd8x","openshift-network-operator/iptables-alerter-69xfb","openshift-ovn-kubernetes/ovnkube-node-ng5xk","kube-system/global-pull-secret-syncer-dzdr7","kube-system/kube-apiserver-proxy-ip-10-0-129-223.ec2.internal","openshift-dns/node-resolver-rkphn","openshift-image-registry/node-ca-qrnmf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-223.ec2.internal","openshift-multus/multus-additional-cni-plugins-cr75t"]
Jul 29 06:22:39.228677 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.228029    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-69xfb"
Jul 29 06:22:39.230178 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.230157    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rkphn"
Jul 29 06:22:39.230512 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.230488    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 29 06:22:39.230729 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.230709    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 29 06:22:39.230729 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.230718    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 29 06:22:39.230888 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.230816    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-m2jhf\""
Jul 29 06:22:39.232050 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.232030    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 29 06:22:39.232399 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.232383    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 29 06:22:39.232586 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.232481    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4wlg7\""
Jul 29 06:22:39.232738 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.232724    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wfqxw"
Jul 29 06:22:39.235146 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.235129    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqpxr"
Jul 29 06:22:39.237746 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.235948    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 29 06:22:39.237746 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.236222    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 29 06:22:39.237746 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.236537    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-25nsr\""
Jul 29 06:22:39.237746 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.237498    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 29 06:22:39.238000 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.237808    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 29 06:22:39.238538 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.238518    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 29 06:22:39.238844 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.238829    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.239013 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.238992    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9zjjc\""
Jul 29 06:22:39.241116 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.241099    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 29 06:22:39.241323 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.241305    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kx6w7\""
Jul 29 06:22:39.241557 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.241534    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 29 06:22:39.241756 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.241737    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 29 06:22:39.242310 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.242284    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njd8x"
Jul 29 06:22:39.242397 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:39.242331    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njd8x" podUID="d1be4cd1-a95a-4507-8159-0c80944c37c9"
Jul 29 06:22:39.243494 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.243475    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzdr7"
Jul 29 06:22:39.243581 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:39.243534    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzdr7" podUID="0667dd8b-11f7-4b28-8f63-2f1bbbf98c56"
Jul 29 06:22:39.245660 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.245642    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cr75t"
Jul 29 06:22:39.247870 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.247830    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 29 06:22:39.247962 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.247904    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wvrr2\""
Jul 29 06:22:39.247962 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.247836    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 29 06:22:39.247962 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.247837    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 29 06:22:39.248092 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.247915    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 29 06:22:39.248092 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.247910    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 29 06:22:39.248888 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.248868    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:22:39.248990 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:39.248940    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcg74" podUID="2ceff9c5-890d-4d2b-85ee-09bcd9e02833"
Jul 29 06:22:39.251044 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.251026    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.251264 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.251240    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-etc-kubernetes\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.251334 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.251274    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-etc-tuned\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.251334 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.251298    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4a2a21ed-615a-4783-9a2a-b57804724f79-hosts-file\") pod \"node-resolver-rkphn\" (UID: \"4a2a21ed-615a-4783-9a2a-b57804724f79\") " pod="openshift-dns/node-resolver-rkphn"
Jul 29 06:22:39.251334 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.251322    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4wz2\" (UniqueName: \"kubernetes.io/projected/4a2a21ed-615a-4783-9a2a-b57804724f79-kube-api-access-n4wz2\") pod \"node-resolver-rkphn\" (UID: \"4a2a21ed-615a-4783-9a2a-b57804724f79\") " pod="openshift-dns/node-resolver-rkphn"
Jul 29 06:22:39.251449 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.251345    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0667dd8b-11f7-4b28-8f63-2f1bbbf98c56-dbus\") pod \"global-pull-secret-syncer-dzdr7\" (UID: \"0667dd8b-11f7-4b28-8f63-2f1bbbf98c56\") " pod="kube-system/global-pull-secret-syncer-dzdr7"
Jul 29 06:22:39.251449 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.251371    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a4c4d323-1adf-4831-8bd8-3ac96b72bc08-socket-dir\") pod \"aws-ebs-csi-driver-node-jqpxr\" (UID: \"a4c4d323-1adf-4831-8bd8-3ac96b72bc08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqpxr"
Jul 29 06:22:39.251449 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.251393    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a4c4d323-1adf-4831-8bd8-3ac96b72bc08-device-dir\") pod \"aws-ebs-csi-driver-node-jqpxr\" (UID: \"a4c4d323-1adf-4831-8bd8-3ac96b72bc08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqpxr"
Jul 29 06:22:39.251449 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.251417    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfp6c\" (UniqueName: \"kubernetes.io/projected/d1be4cd1-a95a-4507-8159-0c80944c37c9-kube-api-access-qfp6c\") pod \"network-check-target-njd8x\" (UID: \"d1be4cd1-a95a-4507-8159-0c80944c37c9\") " pod="openshift-network-diagnostics/network-check-target-njd8x"
Jul 29 06:22:39.251449 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.251440    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-var-lib-kubelet\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.251603 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.251460    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwnvv\" (UniqueName: \"kubernetes.io/projected/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-kube-api-access-qwnvv\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.251603 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.251484    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b9f2b7f0-1e62-43fd-99b0-78be60aa9881-iptables-alerter-script\") pod \"iptables-alerter-69xfb\" (UID: \"b9f2b7f0-1e62-43fd-99b0-78be60aa9881\") " pod="openshift-network-operator/iptables-alerter-69xfb"
Jul 29 06:22:39.251603 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.251508    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a4c4d323-1adf-4831-8bd8-3ac96b72bc08-etc-selinux\") pod \"aws-ebs-csi-driver-node-jqpxr\" (UID: \"a4c4d323-1adf-4831-8bd8-3ac96b72bc08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqpxr"
Jul 29 06:22:39.251603 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.251534    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-run\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.251603 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.251563    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cclcp\" (UniqueName: \"kubernetes.io/projected/b9f2b7f0-1e62-43fd-99b0-78be60aa9881-kube-api-access-cclcp\") pod \"iptables-alerter-69xfb\" (UID: \"b9f2b7f0-1e62-43fd-99b0-78be60aa9881\") " pod="openshift-network-operator/iptables-alerter-69xfb"
Jul 29 06:22:39.251603 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.251581    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a2a21ed-615a-4783-9a2a-b57804724f79-tmp-dir\") pod \"node-resolver-rkphn\" (UID: \"4a2a21ed-615a-4783-9a2a-b57804724f79\") " pod="openshift-dns/node-resolver-rkphn"
Jul 29 06:22:39.251843 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.251613    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a4c4d323-1adf-4831-8bd8-3ac96b72bc08-registration-dir\") pod \"aws-ebs-csi-driver-node-jqpxr\" (UID: \"a4c4d323-1adf-4831-8bd8-3ac96b72bc08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqpxr"
Jul 29 06:22:39.251843 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.251669    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a4c4d323-1adf-4831-8bd8-3ac96b72bc08-sys-fs\") pod \"aws-ebs-csi-driver-node-jqpxr\" (UID: \"a4c4d323-1adf-4831-8bd8-3ac96b72bc08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqpxr"
Jul 29 06:22:39.251843 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.251695    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-etc-sysctl-d\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.251843 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.251709    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-host\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.251843 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.251765    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-etc-sysconfig\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.251843 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.251793    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-etc-systemd\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.251843 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.251829    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-tmp\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.252171 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.251890    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hngv\" (UniqueName: \"kubernetes.io/projected/a4c4d323-1adf-4831-8bd8-3ac96b72bc08-kube-api-access-6hngv\") pod \"aws-ebs-csi-driver-node-jqpxr\" (UID: \"a4c4d323-1adf-4831-8bd8-3ac96b72bc08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqpxr"
Jul 29 06:22:39.252171 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.251913    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-sys\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.252171 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.251930    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-lib-modules\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.252171 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.251973    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e191231b-69c1-4a7e-baf3-d46d529fe7d6-agent-certs\") pod \"konnectivity-agent-wfqxw\" (UID: \"e191231b-69c1-4a7e-baf3-d46d529fe7d6\") " pod="kube-system/konnectivity-agent-wfqxw"
Jul 29 06:22:39.252171 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.252012    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e191231b-69c1-4a7e-baf3-d46d529fe7d6-konnectivity-ca\") pod \"konnectivity-agent-wfqxw\" (UID: \"e191231b-69c1-4a7e-baf3-d46d529fe7d6\") " pod="kube-system/konnectivity-agent-wfqxw"
Jul 29 06:22:39.252171 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.252039    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a4c4d323-1adf-4831-8bd8-3ac96b72bc08-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jqpxr\" (UID: \"a4c4d323-1adf-4831-8bd8-3ac96b72bc08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqpxr"
Jul 29 06:22:39.252171 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.252058    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-etc-modprobe-d\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.252171 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.252073    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0667dd8b-11f7-4b28-8f63-2f1bbbf98c56-kubelet-config\") pod \"global-pull-secret-syncer-dzdr7\" (UID: \"0667dd8b-11f7-4b28-8f63-2f1bbbf98c56\") " pod="kube-system/global-pull-secret-syncer-dzdr7"
Jul 29 06:22:39.252171 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.252109    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0667dd8b-11f7-4b28-8f63-2f1bbbf98c56-original-pull-secret\") pod \"global-pull-secret-syncer-dzdr7\" (UID: \"0667dd8b-11f7-4b28-8f63-2f1bbbf98c56\") " pod="kube-system/global-pull-secret-syncer-dzdr7"
Jul 29 06:22:39.252171 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.252130    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-etc-sysctl-conf\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.252171 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.252144    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9f2b7f0-1e62-43fd-99b0-78be60aa9881-host-slash\") pod \"iptables-alerter-69xfb\" (UID: \"b9f2b7f0-1e62-43fd-99b0-78be60aa9881\") " pod="openshift-network-operator/iptables-alerter-69xfb"
Jul 29 06:22:39.252609 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.252581    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 06:22:39.252917 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.252759    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 29 06:22:39.252917 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.252790    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 29 06:22:39.253361 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.253339    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 29 06:22:39.253447 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.253411    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 29 06:22:39.253618 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.253462    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 29 06:22:39.253618 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.253345    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.253618 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.253413    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8cwk2\""
Jul 29 06:22:39.253807 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.253686    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 29 06:22:39.255170 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.255156    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 29 06:22:39.255439 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.255412    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4hsg4\""
Jul 29 06:22:39.255600 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.255584    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qrnmf"
Jul 29 06:22:39.257433 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.257410    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 29 06:22:39.257635 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.257622    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 29 06:22:39.258224 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.258207    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ltgbm\""
Jul 29 06:22:39.258297 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.258281    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 29 06:22:39.275772 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.275754    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-q849v"
Jul 29 06:22:39.282550 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.282523    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-q849v"
Jul 29 06:22:39.342039 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.342014    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 29 06:22:39.353080 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353058    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-var-lib-kubelet\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.353195 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353090    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a2a21ed-615a-4783-9a2a-b57804724f79-tmp-dir\") pod \"node-resolver-rkphn\" (UID: \"4a2a21ed-615a-4783-9a2a-b57804724f79\") " pod="openshift-dns/node-resolver-rkphn"
Jul 29 06:22:39.353195 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353119    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1673ef43-e0e0-4677-b830-7e8243686336-os-release\") pod \"multus-additional-cni-plugins-cr75t\" (UID: \"1673ef43-e0e0-4677-b830-7e8243686336\") " pod="openshift-multus/multus-additional-cni-plugins-cr75t"
Jul 29 06:22:39.353195 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353146    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-host-run-netns\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.353195 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353169    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-etc-kubernetes\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.353195 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353182    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-var-lib-kubelet\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.353409 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353214    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fhpt\" (UniqueName: \"kubernetes.io/projected/85622ea0-6a2b-47d9-a05e-1265b9d86baa-kube-api-access-4fhpt\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.353409 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353252    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-run\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.353409 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353279    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cclcp\" (UniqueName: \"kubernetes.io/projected/b9f2b7f0-1e62-43fd-99b0-78be60aa9881-kube-api-access-cclcp\") pod \"iptables-alerter-69xfb\" (UID: \"b9f2b7f0-1e62-43fd-99b0-78be60aa9881\") " pod="openshift-network-operator/iptables-alerter-69xfb"
Jul 29 06:22:39.353409 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353306    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-metrics-certs\") pod \"network-metrics-daemon-vcg74\" (UID: \"2ceff9c5-890d-4d2b-85ee-09bcd9e02833\") " pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:22:39.353409 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353314    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-run\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.353409 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353329    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-system-cni-dir\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.353409 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353356    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2079ff1a-5b9e-439d-bd6c-02c73c344340-serviceca\") pod \"node-ca-qrnmf\" (UID: \"2079ff1a-5b9e-439d-bd6c-02c73c344340\") " pod="openshift-image-registry/node-ca-qrnmf"
Jul 29 06:22:39.353409 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353379    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n78j\" (UniqueName: \"kubernetes.io/projected/2079ff1a-5b9e-439d-bd6c-02c73c344340-kube-api-access-7n78j\") pod \"node-ca-qrnmf\" (UID: \"2079ff1a-5b9e-439d-bd6c-02c73c344340\") " pod="openshift-image-registry/node-ca-qrnmf"
Jul 29 06:22:39.353724 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353410    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a4c4d323-1adf-4831-8bd8-3ac96b72bc08-registration-dir\") pod \"aws-ebs-csi-driver-node-jqpxr\" (UID: \"a4c4d323-1adf-4831-8bd8-3ac96b72bc08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqpxr"
Jul 29 06:22:39.353724 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353433    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a4c4d323-1adf-4831-8bd8-3ac96b72bc08-sys-fs\") pod \"aws-ebs-csi-driver-node-jqpxr\" (UID: \"a4c4d323-1adf-4831-8bd8-3ac96b72bc08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqpxr"
Jul 29 06:22:39.353724 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353413    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a2a21ed-615a-4783-9a2a-b57804724f79-tmp-dir\") pod \"node-resolver-rkphn\" (UID: \"4a2a21ed-615a-4783-9a2a-b57804724f79\") " pod="openshift-dns/node-resolver-rkphn"
Jul 29 06:22:39.353724 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353479    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-etc-sysctl-d\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.353724 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353481    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a4c4d323-1adf-4831-8bd8-3ac96b72bc08-registration-dir\") pod \"aws-ebs-csi-driver-node-jqpxr\" (UID: \"a4c4d323-1adf-4831-8bd8-3ac96b72bc08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqpxr"
Jul 29 06:22:39.353724 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353512    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-host\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.353724 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353536    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a4c4d323-1adf-4831-8bd8-3ac96b72bc08-sys-fs\") pod \"aws-ebs-csi-driver-node-jqpxr\" (UID: \"a4c4d323-1adf-4831-8bd8-3ac96b72bc08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqpxr"
Jul 29 06:22:39.353724 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353537    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-run-ovn\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.353724 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353553    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-host\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.353724 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353569    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6200726d-f1bf-43ba-bf59-6abb0d138092-ovnkube-script-lib\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.353724 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353587    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-cnibin\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.353724 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353608    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2079ff1a-5b9e-439d-bd6c-02c73c344340-host\") pod \"node-ca-qrnmf\" (UID: \"2079ff1a-5b9e-439d-bd6c-02c73c344340\") " pod="openshift-image-registry/node-ca-qrnmf"
Jul 29 06:22:39.353724 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353622    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpngt\" (UniqueName: \"kubernetes.io/projected/6200726d-f1bf-43ba-bf59-6abb0d138092-kube-api-access-lpngt\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.353724 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353643    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-etc-sysctl-d\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.353724 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353656    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-hostroot\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.353724 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353677    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-multus-conf-dir\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.353724 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353703    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e191231b-69c1-4a7e-baf3-d46d529fe7d6-konnectivity-ca\") pod \"konnectivity-agent-wfqxw\" (UID: \"e191231b-69c1-4a7e-baf3-d46d529fe7d6\") " pod="kube-system/konnectivity-agent-wfqxw"
Jul 29 06:22:39.354548 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353719    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-etc-openvswitch\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.354548 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353759    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.354548 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353784    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-os-release\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.354548 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353920    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-multus-socket-dir-parent\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.354548 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353958    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a4c4d323-1adf-4831-8bd8-3ac96b72bc08-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jqpxr\" (UID: \"a4c4d323-1adf-4831-8bd8-3ac96b72bc08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqpxr"
Jul 29 06:22:39.354548 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.353986    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-run-systemd\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.354548 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354012    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0667dd8b-11f7-4b28-8f63-2f1bbbf98c56-kubelet-config\") pod \"global-pull-secret-syncer-dzdr7\" (UID: \"0667dd8b-11f7-4b28-8f63-2f1bbbf98c56\") " pod="kube-system/global-pull-secret-syncer-dzdr7"
Jul 29 06:22:39.354548 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354036    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-etc-sysctl-conf\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.354548 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354100    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a4c4d323-1adf-4831-8bd8-3ac96b72bc08-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jqpxr\" (UID: \"a4c4d323-1adf-4831-8bd8-3ac96b72bc08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqpxr"
Jul 29 06:22:39.354548 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354117    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6200726d-f1bf-43ba-bf59-6abb0d138092-ovn-node-metrics-cert\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.354548 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354103    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0667dd8b-11f7-4b28-8f63-2f1bbbf98c56-kubelet-config\") pod \"global-pull-secret-syncer-dzdr7\" (UID: \"0667dd8b-11f7-4b28-8f63-2f1bbbf98c56\") " pod="kube-system/global-pull-secret-syncer-dzdr7"
Jul 29 06:22:39.354548 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354169    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-etc-sysctl-conf\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.354548 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354182    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4wz2\" (UniqueName: \"kubernetes.io/projected/4a2a21ed-615a-4783-9a2a-b57804724f79-kube-api-access-n4wz2\") pod \"node-resolver-rkphn\" (UID: \"4a2a21ed-615a-4783-9a2a-b57804724f79\") " pod="openshift-dns/node-resolver-rkphn"
Jul 29 06:22:39.354548 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354219    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gvbx\" (UniqueName: \"kubernetes.io/projected/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-kube-api-access-9gvbx\") pod \"network-metrics-daemon-vcg74\" (UID: \"2ceff9c5-890d-4d2b-85ee-09bcd9e02833\") " pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:22:39.354548 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354265    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e191231b-69c1-4a7e-baf3-d46d529fe7d6-konnectivity-ca\") pod \"konnectivity-agent-wfqxw\" (UID: \"e191231b-69c1-4a7e-baf3-d46d529fe7d6\") " pod="kube-system/konnectivity-agent-wfqxw"
Jul 29 06:22:39.354548 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354330    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-systemd-units\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.354548 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354358    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-host-cni-bin\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.355250 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354383    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-host-var-lib-kubelet\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.355250 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354412    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a4c4d323-1adf-4831-8bd8-3ac96b72bc08-socket-dir\") pod \"aws-ebs-csi-driver-node-jqpxr\" (UID: \"a4c4d323-1adf-4831-8bd8-3ac96b72bc08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqpxr"
Jul 29 06:22:39.355250 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354436    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a4c4d323-1adf-4831-8bd8-3ac96b72bc08-device-dir\") pod \"aws-ebs-csi-driver-node-jqpxr\" (UID: \"a4c4d323-1adf-4831-8bd8-3ac96b72bc08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqpxr"
Jul 29 06:22:39.355250 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354461    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwnvv\" (UniqueName: \"kubernetes.io/projected/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-kube-api-access-qwnvv\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.355250 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354484    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b9f2b7f0-1e62-43fd-99b0-78be60aa9881-iptables-alerter-script\") pod \"iptables-alerter-69xfb\" (UID: \"b9f2b7f0-1e62-43fd-99b0-78be60aa9881\") " pod="openshift-network-operator/iptables-alerter-69xfb"
Jul 29 06:22:39.355250 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354526    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-node-log\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.355250 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354530    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a4c4d323-1adf-4831-8bd8-3ac96b72bc08-device-dir\") pod \"aws-ebs-csi-driver-node-jqpxr\" (UID: \"a4c4d323-1adf-4831-8bd8-3ac96b72bc08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqpxr"
Jul 29 06:22:39.355250 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354548    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6200726d-f1bf-43ba-bf59-6abb0d138092-ovnkube-config\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.355250 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354561    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a4c4d323-1adf-4831-8bd8-3ac96b72bc08-socket-dir\") pod \"aws-ebs-csi-driver-node-jqpxr\" (UID: \"a4c4d323-1adf-4831-8bd8-3ac96b72bc08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqpxr"
Jul 29 06:22:39.355250 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354574    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a4c4d323-1adf-4831-8bd8-3ac96b72bc08-etc-selinux\") pod \"aws-ebs-csi-driver-node-jqpxr\" (UID: \"a4c4d323-1adf-4831-8bd8-3ac96b72bc08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqpxr"
Jul 29 06:22:39.355250 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354599    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1673ef43-e0e0-4677-b830-7e8243686336-system-cni-dir\") pod \"multus-additional-cni-plugins-cr75t\" (UID: \"1673ef43-e0e0-4677-b830-7e8243686336\") " pod="openshift-multus/multus-additional-cni-plugins-cr75t"
Jul 29 06:22:39.355250 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354638    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-multus-cni-dir\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.355250 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354662    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-host-run-k8s-cni-cncf-io\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.355250 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354685    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a4c4d323-1adf-4831-8bd8-3ac96b72bc08-etc-selinux\") pod \"aws-ebs-csi-driver-node-jqpxr\" (UID: \"a4c4d323-1adf-4831-8bd8-3ac96b72bc08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqpxr"
Jul 29 06:22:39.355250 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354687    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-etc-sysconfig\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.355250 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354723    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-etc-systemd\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.355250 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354736    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-etc-sysconfig\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.355917 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354739    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-tmp\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.355917 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354770    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-etc-systemd\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.355917 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354790    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1673ef43-e0e0-4677-b830-7e8243686336-cnibin\") pod \"multus-additional-cni-plugins-cr75t\" (UID: \"1673ef43-e0e0-4677-b830-7e8243686336\") " pod="openshift-multus/multus-additional-cni-plugins-cr75t"
Jul 29 06:22:39.355917 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354818    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1673ef43-e0e0-4677-b830-7e8243686336-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cr75t\" (UID: \"1673ef43-e0e0-4677-b830-7e8243686336\") " pod="openshift-multus/multus-additional-cni-plugins-cr75t"
Jul 29 06:22:39.355917 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354844    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhtzl\" (UniqueName: \"kubernetes.io/projected/1673ef43-e0e0-4677-b830-7e8243686336-kube-api-access-vhtzl\") pod \"multus-additional-cni-plugins-cr75t\" (UID: \"1673ef43-e0e0-4677-b830-7e8243686336\") " pod="openshift-multus/multus-additional-cni-plugins-cr75t"
Jul 29 06:22:39.355917 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354884    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-host-run-netns\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.355917 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354923    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hngv\" (UniqueName: \"kubernetes.io/projected/a4c4d323-1adf-4831-8bd8-3ac96b72bc08-kube-api-access-6hngv\") pod \"aws-ebs-csi-driver-node-jqpxr\" (UID: \"a4c4d323-1adf-4831-8bd8-3ac96b72bc08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqpxr"
Jul 29 06:22:39.355917 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354948    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-sys\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.355917 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354965    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b9f2b7f0-1e62-43fd-99b0-78be60aa9881-iptables-alerter-script\") pod \"iptables-alerter-69xfb\" (UID: \"b9f2b7f0-1e62-43fd-99b0-78be60aa9881\") " pod="openshift-network-operator/iptables-alerter-69xfb"
Jul 29 06:22:39.355917 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.354979    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 29 06:22:39.355917 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.355003    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-sys\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.355917 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.355059    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-lib-modules\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.355917 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.355079    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e191231b-69c1-4a7e-baf3-d46d529fe7d6-agent-certs\") pod \"konnectivity-agent-wfqxw\" (UID: \"e191231b-69c1-4a7e-baf3-d46d529fe7d6\") " pod="kube-system/konnectivity-agent-wfqxw"
Jul 29 06:22:39.355917 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.355104    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1673ef43-e0e0-4677-b830-7e8243686336-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cr75t\" (UID: \"1673ef43-e0e0-4677-b830-7e8243686336\") " pod="openshift-multus/multus-additional-cni-plugins-cr75t"
Jul 29 06:22:39.355917 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.355132    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-host-slash\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.355917 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.355153    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-var-lib-openvswitch\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.355917 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.355181    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-log-socket\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.356894 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.355184    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-lib-modules\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.356894 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.355211    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-etc-modprobe-d\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.356894 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.355232    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-host-kubelet\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.356894 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.355265    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-host-run-multus-certs\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.356894 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.355293    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0667dd8b-11f7-4b28-8f63-2f1bbbf98c56-original-pull-secret\") pod \"global-pull-secret-syncer-dzdr7\" (UID: \"0667dd8b-11f7-4b28-8f63-2f1bbbf98c56\") " pod="kube-system/global-pull-secret-syncer-dzdr7"
Jul 29 06:22:39.356894 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.355292    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-etc-modprobe-d\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.356894 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.355322    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9f2b7f0-1e62-43fd-99b0-78be60aa9881-host-slash\") pod \"iptables-alerter-69xfb\" (UID: \"b9f2b7f0-1e62-43fd-99b0-78be60aa9881\") " pod="openshift-network-operator/iptables-alerter-69xfb"
Jul 29 06:22:39.356894 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.355355    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9f2b7f0-1e62-43fd-99b0-78be60aa9881-host-slash\") pod \"iptables-alerter-69xfb\" (UID: \"b9f2b7f0-1e62-43fd-99b0-78be60aa9881\") " pod="openshift-network-operator/iptables-alerter-69xfb"
Jul 29 06:22:39.356894 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.355371    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1673ef43-e0e0-4677-b830-7e8243686336-cni-binary-copy\") pod \"multus-additional-cni-plugins-cr75t\" (UID: \"1673ef43-e0e0-4677-b830-7e8243686336\") " pod="openshift-multus/multus-additional-cni-plugins-cr75t"
Jul 29 06:22:39.356894 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:39.355391    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 06:22:39.356894 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.355409    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6200726d-f1bf-43ba-bf59-6abb0d138092-env-overrides\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.356894 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:39.355467    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0667dd8b-11f7-4b28-8f63-2f1bbbf98c56-original-pull-secret podName:0667dd8b-11f7-4b28-8f63-2f1bbbf98c56 nodeName:}" failed. No retries permitted until 2026-07-29 06:22:39.85543574 +0000 UTC m=+2.933067518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0667dd8b-11f7-4b28-8f63-2f1bbbf98c56-original-pull-secret") pod "global-pull-secret-syncer-dzdr7" (UID: "0667dd8b-11f7-4b28-8f63-2f1bbbf98c56") : object "kube-system"/"original-pull-secret" not registered
Jul 29 06:22:39.356894 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.355482    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85622ea0-6a2b-47d9-a05e-1265b9d86baa-cni-binary-copy\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.356894 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.355504    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-host-var-lib-cni-bin\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.356894 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.355536    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-host-var-lib-cni-multus\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.356894 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.355573    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-etc-kubernetes\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.356894 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.355593    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-etc-tuned\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.357731 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.355608    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4a2a21ed-615a-4783-9a2a-b57804724f79-hosts-file\") pod \"node-resolver-rkphn\" (UID: \"4a2a21ed-615a-4783-9a2a-b57804724f79\") " pod="openshift-dns/node-resolver-rkphn"
Jul 29 06:22:39.357731 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.355633    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1673ef43-e0e0-4677-b830-7e8243686336-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cr75t\" (UID: \"1673ef43-e0e0-4677-b830-7e8243686336\") " pod="openshift-multus/multus-additional-cni-plugins-cr75t"
Jul 29 06:22:39.357731 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.355663    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-run-openvswitch\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.357731 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.355667    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-etc-kubernetes\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.357731 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.355697    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4a2a21ed-615a-4783-9a2a-b57804724f79-hosts-file\") pod \"node-resolver-rkphn\" (UID: \"4a2a21ed-615a-4783-9a2a-b57804724f79\") " pod="openshift-dns/node-resolver-rkphn"
Jul 29 06:22:39.357731 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.355734    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-host-run-ovn-kubernetes\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.357731 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.355762    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-host-cni-netd\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.357731 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.355778    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/85622ea0-6a2b-47d9-a05e-1265b9d86baa-multus-daemon-config\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.357731 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.355818    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0667dd8b-11f7-4b28-8f63-2f1bbbf98c56-dbus\") pod \"global-pull-secret-syncer-dzdr7\" (UID: \"0667dd8b-11f7-4b28-8f63-2f1bbbf98c56\") " pod="kube-system/global-pull-secret-syncer-dzdr7"
Jul 29 06:22:39.357731 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.355837    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfp6c\" (UniqueName: \"kubernetes.io/projected/d1be4cd1-a95a-4507-8159-0c80944c37c9-kube-api-access-qfp6c\") pod \"network-check-target-njd8x\" (UID: \"d1be4cd1-a95a-4507-8159-0c80944c37c9\") " pod="openshift-network-diagnostics/network-check-target-njd8x"
Jul 29 06:22:39.357731 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.355978    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0667dd8b-11f7-4b28-8f63-2f1bbbf98c56-dbus\") pod \"global-pull-secret-syncer-dzdr7\" (UID: \"0667dd8b-11f7-4b28-8f63-2f1bbbf98c56\") " pod="kube-system/global-pull-secret-syncer-dzdr7"
Jul 29 06:22:39.358212 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.358129    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-tmp\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.358350 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.358330    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-etc-tuned\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.358594 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.358580    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e191231b-69c1-4a7e-baf3-d46d529fe7d6-agent-certs\") pod \"konnectivity-agent-wfqxw\" (UID: \"e191231b-69c1-4a7e-baf3-d46d529fe7d6\") " pod="kube-system/konnectivity-agent-wfqxw"
Jul 29 06:22:39.360730 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.360709    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cclcp\" (UniqueName: \"kubernetes.io/projected/b9f2b7f0-1e62-43fd-99b0-78be60aa9881-kube-api-access-cclcp\") pod \"iptables-alerter-69xfb\" (UID: \"b9f2b7f0-1e62-43fd-99b0-78be60aa9881\") " pod="openshift-network-operator/iptables-alerter-69xfb"
Jul 29 06:22:39.363340 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:39.363322    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 06:22:39.363438 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:39.363354    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 06:22:39.363438 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:39.363368    2563 projected.go:196] Error preparing data for projected volume kube-api-access-qfp6c for pod openshift-network-diagnostics/network-check-target-njd8x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 06:22:39.363438 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:39.363437    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1be4cd1-a95a-4507-8159-0c80944c37c9-kube-api-access-qfp6c podName:d1be4cd1-a95a-4507-8159-0c80944c37c9 nodeName:}" failed. No retries permitted until 2026-07-29 06:22:39.863424509 +0000 UTC m=+2.941056254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qfp6c" (UniqueName: "kubernetes.io/projected/d1be4cd1-a95a-4507-8159-0c80944c37c9-kube-api-access-qfp6c") pod "network-check-target-njd8x" (UID: "d1be4cd1-a95a-4507-8159-0c80944c37c9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 06:22:39.365765 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.365743    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hngv\" (UniqueName: \"kubernetes.io/projected/a4c4d323-1adf-4831-8bd8-3ac96b72bc08-kube-api-access-6hngv\") pod \"aws-ebs-csi-driver-node-jqpxr\" (UID: \"a4c4d323-1adf-4831-8bd8-3ac96b72bc08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqpxr"
Jul 29 06:22:39.366290 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.366271    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4wz2\" (UniqueName: \"kubernetes.io/projected/4a2a21ed-615a-4783-9a2a-b57804724f79-kube-api-access-n4wz2\") pod \"node-resolver-rkphn\" (UID: \"4a2a21ed-615a-4783-9a2a-b57804724f79\") " pod="openshift-dns/node-resolver-rkphn"
Jul 29 06:22:39.366372 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.366322    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwnvv\" (UniqueName: \"kubernetes.io/projected/2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa-kube-api-access-qwnvv\") pod \"tuned-7b9qz\" (UID: \"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa\") " pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.394937 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.394888    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-223.ec2.internal" event={"ID":"bc0f4deacee416934506716d269ff92f","Type":"ContainerStarted","Data":"9e07c456b1c96fc04c5446215d01170d595890737685e26992e41664b00401a9"}
Jul 29 06:22:39.396036 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.396008    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-223.ec2.internal" event={"ID":"2ff94682e942fadc8425e32a6af57aed","Type":"ContainerStarted","Data":"20c79b2bc4e0a53c9e7d167d1d4a32741a09293f935980b0beb1a92b7f2bd042"}
Jul 29 06:22:39.456289 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.456249    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-host-kubelet\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.456447 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.456300    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-host-run-multus-certs\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.456447 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.456324    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-host-kubelet\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.456447 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.456342    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1673ef43-e0e0-4677-b830-7e8243686336-cni-binary-copy\") pod \"multus-additional-cni-plugins-cr75t\" (UID: \"1673ef43-e0e0-4677-b830-7e8243686336\") " pod="openshift-multus/multus-additional-cni-plugins-cr75t"
Jul 29 06:22:39.456447 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.456389    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6200726d-f1bf-43ba-bf59-6abb0d138092-env-overrides\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.456447 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.456412    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85622ea0-6a2b-47d9-a05e-1265b9d86baa-cni-binary-copy\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.456447 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.456428    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-host-var-lib-cni-bin\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.456447 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.456447    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-host-var-lib-cni-multus\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.456701 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.456474    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1673ef43-e0e0-4677-b830-7e8243686336-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cr75t\" (UID: \"1673ef43-e0e0-4677-b830-7e8243686336\") " pod="openshift-multus/multus-additional-cni-plugins-cr75t"
Jul 29 06:22:39.456701 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.456481    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-host-var-lib-cni-bin\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.456701 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.456499    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-run-openvswitch\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.456701 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.456542    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-run-openvswitch\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.456701 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.456413    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-host-run-multus-certs\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.456701 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.456579    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-host-var-lib-cni-multus\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.456947 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.456924    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-host-run-ovn-kubernetes\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.456947 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.456937    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6200726d-f1bf-43ba-bf59-6abb0d138092-env-overrides\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.457010 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.456966    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-host-cni-netd\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.457010 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.456992    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/85622ea0-6a2b-47d9-a05e-1265b9d86baa-multus-daemon-config\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.457010 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.456999    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1673ef43-e0e0-4677-b830-7e8243686336-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cr75t\" (UID: \"1673ef43-e0e0-4677-b830-7e8243686336\") " pod="openshift-multus/multus-additional-cni-plugins-cr75t"
Jul 29 06:22:39.457090 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457000    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-host-run-ovn-kubernetes\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.457090 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457016    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85622ea0-6a2b-47d9-a05e-1265b9d86baa-cni-binary-copy\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.457090 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457033    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-host-cni-netd\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.457090 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457058    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1673ef43-e0e0-4677-b830-7e8243686336-os-release\") pod \"multus-additional-cni-plugins-cr75t\" (UID: \"1673ef43-e0e0-4677-b830-7e8243686336\") " pod="openshift-multus/multus-additional-cni-plugins-cr75t"
Jul 29 06:22:39.457090 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457083    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-host-run-netns\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.457270 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457105    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-etc-kubernetes\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.457270 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457129    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fhpt\" (UniqueName: \"kubernetes.io/projected/85622ea0-6a2b-47d9-a05e-1265b9d86baa-kube-api-access-4fhpt\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.457270 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457154    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-metrics-certs\") pod \"network-metrics-daemon-vcg74\" (UID: \"2ceff9c5-890d-4d2b-85ee-09bcd9e02833\") " pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:22:39.457270 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457147    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-host-run-netns\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.457270 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457159    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1673ef43-e0e0-4677-b830-7e8243686336-os-release\") pod \"multus-additional-cni-plugins-cr75t\" (UID: \"1673ef43-e0e0-4677-b830-7e8243686336\") " pod="openshift-multus/multus-additional-cni-plugins-cr75t"
Jul 29 06:22:39.457270 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457189    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-system-cni-dir\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.457270 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457196    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-etc-kubernetes\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.457270 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457224    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2079ff1a-5b9e-439d-bd6c-02c73c344340-serviceca\") pod \"node-ca-qrnmf\" (UID: \"2079ff1a-5b9e-439d-bd6c-02c73c344340\") " pod="openshift-image-registry/node-ca-qrnmf"
Jul 29 06:22:39.457270 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457242    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-system-cni-dir\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.457270 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457249    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n78j\" (UniqueName: \"kubernetes.io/projected/2079ff1a-5b9e-439d-bd6c-02c73c344340-kube-api-access-7n78j\") pod \"node-ca-qrnmf\" (UID: \"2079ff1a-5b9e-439d-bd6c-02c73c344340\") " pod="openshift-image-registry/node-ca-qrnmf"
Jul 29 06:22:39.457270 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457274    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-run-ovn\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.457760 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457345    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6200726d-f1bf-43ba-bf59-6abb0d138092-ovnkube-script-lib\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.457760 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:39.457383    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 06:22:39.457760 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457405    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/85622ea0-6a2b-47d9-a05e-1265b9d86baa-multus-daemon-config\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.457760 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457384    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-cnibin\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.457760 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457433    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-run-ovn\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.457760 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:39.457453    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-metrics-certs podName:2ceff9c5-890d-4d2b-85ee-09bcd9e02833 nodeName:}" failed. No retries permitted until 2026-07-29 06:22:39.957435339 +0000 UTC m=+3.035067102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-metrics-certs") pod "network-metrics-daemon-vcg74" (UID: "2ceff9c5-890d-4d2b-85ee-09bcd9e02833") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 06:22:39.457760 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457451    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-cnibin\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.457760 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457487    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2079ff1a-5b9e-439d-bd6c-02c73c344340-host\") pod \"node-ca-qrnmf\" (UID: \"2079ff1a-5b9e-439d-bd6c-02c73c344340\") " pod="openshift-image-registry/node-ca-qrnmf"
Jul 29 06:22:39.457760 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457515    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpngt\" (UniqueName: \"kubernetes.io/projected/6200726d-f1bf-43ba-bf59-6abb0d138092-kube-api-access-lpngt\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.457760 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457540    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-hostroot\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.457760 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457568    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2079ff1a-5b9e-439d-bd6c-02c73c344340-host\") pod \"node-ca-qrnmf\" (UID: \"2079ff1a-5b9e-439d-bd6c-02c73c344340\") " pod="openshift-image-registry/node-ca-qrnmf"
Jul 29 06:22:39.457760 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457585    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-hostroot\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.457760 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457653    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-multus-conf-dir\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.457760 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457686    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-etc-openvswitch\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.457760 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.457760 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457719    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-os-release\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.457760 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457724    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1673ef43-e0e0-4677-b830-7e8243686336-cni-binary-copy\") pod \"multus-additional-cni-plugins-cr75t\" (UID: \"1673ef43-e0e0-4677-b830-7e8243686336\") " pod="openshift-multus/multus-additional-cni-plugins-cr75t"
Jul 29 06:22:39.457760 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457761    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-etc-openvswitch\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.458756 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457781    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-os-release\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.458756 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457781    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-multus-conf-dir\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.458756 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457794    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-multus-socket-dir-parent\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.458756 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457816    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6200726d-f1bf-43ba-bf59-6abb0d138092-ovnkube-script-lib\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.458756 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457822    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-run-systemd\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.458756 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457805    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.458756 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457864    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6200726d-f1bf-43ba-bf59-6abb0d138092-ovn-node-metrics-cert\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.458756 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457891    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gvbx\" (UniqueName: \"kubernetes.io/projected/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-kube-api-access-9gvbx\") pod \"network-metrics-daemon-vcg74\" (UID: \"2ceff9c5-890d-4d2b-85ee-09bcd9e02833\") " pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:22:39.458756 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457904    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-run-systemd\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.458756 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457913    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-systemd-units\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.458756 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457935    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-host-cni-bin\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.458756 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457931    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-multus-socket-dir-parent\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.458756 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457975    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-systemd-units\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.458756 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.457999    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-host-cni-bin\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.458756 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.458098    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-host-var-lib-kubelet\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.458756 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.458146    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-host-var-lib-kubelet\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.458756 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.458160    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-node-log\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.458756 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.458195    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6200726d-f1bf-43ba-bf59-6abb0d138092-ovnkube-config\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.459680 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.458201    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-node-log\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.459680 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.458227    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1673ef43-e0e0-4677-b830-7e8243686336-system-cni-dir\") pod \"multus-additional-cni-plugins-cr75t\" (UID: \"1673ef43-e0e0-4677-b830-7e8243686336\") " pod="openshift-multus/multus-additional-cni-plugins-cr75t"
Jul 29 06:22:39.459680 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.458227    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2079ff1a-5b9e-439d-bd6c-02c73c344340-serviceca\") pod \"node-ca-qrnmf\" (UID: \"2079ff1a-5b9e-439d-bd6c-02c73c344340\") " pod="openshift-image-registry/node-ca-qrnmf"
Jul 29 06:22:39.459680 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.458251    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-multus-cni-dir\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.459680 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.458271    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-host-run-k8s-cni-cncf-io\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.459680 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.458285    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1673ef43-e0e0-4677-b830-7e8243686336-system-cni-dir\") pod \"multus-additional-cni-plugins-cr75t\" (UID: \"1673ef43-e0e0-4677-b830-7e8243686336\") " pod="openshift-multus/multus-additional-cni-plugins-cr75t"
Jul 29 06:22:39.459680 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.458290    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1673ef43-e0e0-4677-b830-7e8243686336-cnibin\") pod \"multus-additional-cni-plugins-cr75t\" (UID: \"1673ef43-e0e0-4677-b830-7e8243686336\") " pod="openshift-multus/multus-additional-cni-plugins-cr75t"
Jul 29 06:22:39.459680 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.458315    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1673ef43-e0e0-4677-b830-7e8243686336-cnibin\") pod \"multus-additional-cni-plugins-cr75t\" (UID: \"1673ef43-e0e0-4677-b830-7e8243686336\") " pod="openshift-multus/multus-additional-cni-plugins-cr75t"
Jul 29 06:22:39.459680 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.458323    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1673ef43-e0e0-4677-b830-7e8243686336-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cr75t\" (UID: \"1673ef43-e0e0-4677-b830-7e8243686336\") " pod="openshift-multus/multus-additional-cni-plugins-cr75t"
Jul 29 06:22:39.459680 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.458347    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-host-run-k8s-cni-cncf-io\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.459680 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.458350    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhtzl\" (UniqueName: \"kubernetes.io/projected/1673ef43-e0e0-4677-b830-7e8243686336-kube-api-access-vhtzl\") pod \"multus-additional-cni-plugins-cr75t\" (UID: \"1673ef43-e0e0-4677-b830-7e8243686336\") " pod="openshift-multus/multus-additional-cni-plugins-cr75t"
Jul 29 06:22:39.459680 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.458364    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85622ea0-6a2b-47d9-a05e-1265b9d86baa-multus-cni-dir\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.459680 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.458389    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-host-run-netns\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.459680 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.458419    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1673ef43-e0e0-4677-b830-7e8243686336-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cr75t\" (UID: \"1673ef43-e0e0-4677-b830-7e8243686336\") " pod="openshift-multus/multus-additional-cni-plugins-cr75t"
Jul 29 06:22:39.459680 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.458443    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-host-slash\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.459680 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.458466    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-var-lib-openvswitch\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.459680 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.458489    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-log-socket\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.460474 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.458566    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-log-socket\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.460474 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.458605    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-host-slash\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.460474 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.458641    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-var-lib-openvswitch\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.460474 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.458650    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1673ef43-e0e0-4677-b830-7e8243686336-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cr75t\" (UID: \"1673ef43-e0e0-4677-b830-7e8243686336\") " pod="openshift-multus/multus-additional-cni-plugins-cr75t"
Jul 29 06:22:39.460474 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.458679    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6200726d-f1bf-43ba-bf59-6abb0d138092-host-run-netns\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.460474 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.458693    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6200726d-f1bf-43ba-bf59-6abb0d138092-ovnkube-config\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.460474 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.458724    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1673ef43-e0e0-4677-b830-7e8243686336-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cr75t\" (UID: \"1673ef43-e0e0-4677-b830-7e8243686336\") " pod="openshift-multus/multus-additional-cni-plugins-cr75t"
Jul 29 06:22:39.460676 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.460648    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6200726d-f1bf-43ba-bf59-6abb0d138092-ovn-node-metrics-cert\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.469005 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.468978    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhtzl\" (UniqueName: \"kubernetes.io/projected/1673ef43-e0e0-4677-b830-7e8243686336-kube-api-access-vhtzl\") pod \"multus-additional-cni-plugins-cr75t\" (UID: \"1673ef43-e0e0-4677-b830-7e8243686336\") " pod="openshift-multus/multus-additional-cni-plugins-cr75t"
Jul 29 06:22:39.469103 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.469055    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n78j\" (UniqueName: \"kubernetes.io/projected/2079ff1a-5b9e-439d-bd6c-02c73c344340-kube-api-access-7n78j\") pod \"node-ca-qrnmf\" (UID: \"2079ff1a-5b9e-439d-bd6c-02c73c344340\") " pod="openshift-image-registry/node-ca-qrnmf"
Jul 29 06:22:39.469486 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.469465    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpngt\" (UniqueName: \"kubernetes.io/projected/6200726d-f1bf-43ba-bf59-6abb0d138092-kube-api-access-lpngt\") pod \"ovnkube-node-ng5xk\" (UID: \"6200726d-f1bf-43ba-bf59-6abb0d138092\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.469570 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.469516    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gvbx\" (UniqueName: \"kubernetes.io/projected/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-kube-api-access-9gvbx\") pod \"network-metrics-daemon-vcg74\" (UID: \"2ceff9c5-890d-4d2b-85ee-09bcd9e02833\") " pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:22:39.469629 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.469589    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fhpt\" (UniqueName: \"kubernetes.io/projected/85622ea0-6a2b-47d9-a05e-1265b9d86baa-kube-api-access-4fhpt\") pod \"multus-wtdbb\" (UID: \"85622ea0-6a2b-47d9-a05e-1265b9d86baa\") " pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.543883 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.543789    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-69xfb"
Jul 29 06:22:39.550649 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.550627    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rkphn"
Jul 29 06:22:39.557166 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.557149    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wfqxw"
Jul 29 06:22:39.562864 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.562827    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqpxr"
Jul 29 06:22:39.571697 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.571678    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7b9qz"
Jul 29 06:22:39.585474 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.585454    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cr75t"
Jul 29 06:22:39.593259 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.593239    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:22:39.601901 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.601880    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wtdbb"
Jul 29 06:22:39.606490 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.606472    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qrnmf"
Jul 29 06:22:39.663488 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.663466    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 06:22:39.862900 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.862816    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0667dd8b-11f7-4b28-8f63-2f1bbbf98c56-original-pull-secret\") pod \"global-pull-secret-syncer-dzdr7\" (UID: \"0667dd8b-11f7-4b28-8f63-2f1bbbf98c56\") " pod="kube-system/global-pull-secret-syncer-dzdr7"
Jul 29 06:22:39.863054 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:39.862972    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 06:22:39.863054 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:39.863046    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0667dd8b-11f7-4b28-8f63-2f1bbbf98c56-original-pull-secret podName:0667dd8b-11f7-4b28-8f63-2f1bbbf98c56 nodeName:}" failed. No retries permitted until 2026-07-29 06:22:40.863027936 +0000 UTC m=+3.940659699 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0667dd8b-11f7-4b28-8f63-2f1bbbf98c56-original-pull-secret") pod "global-pull-secret-syncer-dzdr7" (UID: "0667dd8b-11f7-4b28-8f63-2f1bbbf98c56") : object "kube-system"/"original-pull-secret" not registered
Jul 29 06:22:39.964092 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.964059    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfp6c\" (UniqueName: \"kubernetes.io/projected/d1be4cd1-a95a-4507-8159-0c80944c37c9-kube-api-access-qfp6c\") pod \"network-check-target-njd8x\" (UID: \"d1be4cd1-a95a-4507-8159-0c80944c37c9\") " pod="openshift-network-diagnostics/network-check-target-njd8x"
Jul 29 06:22:39.964240 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:39.964103    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-metrics-certs\") pod \"network-metrics-daemon-vcg74\" (UID: \"2ceff9c5-890d-4d2b-85ee-09bcd9e02833\") " pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:22:39.964240 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:39.964218    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 06:22:39.964331 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:39.964243    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 06:22:39.964331 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:39.964257    2563 projected.go:196] Error preparing data for projected volume kube-api-access-qfp6c for pod openshift-network-diagnostics/network-check-target-njd8x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 06:22:39.964331 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:39.964260    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 06:22:39.964331 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:39.964317    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-metrics-certs podName:2ceff9c5-890d-4d2b-85ee-09bcd9e02833 nodeName:}" failed. No retries permitted until 2026-07-29 06:22:40.964303957 +0000 UTC m=+4.041935718 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-metrics-certs") pod "network-metrics-daemon-vcg74" (UID: "2ceff9c5-890d-4d2b-85ee-09bcd9e02833") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 06:22:39.964497 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:39.964335    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1be4cd1-a95a-4507-8159-0c80944c37c9-kube-api-access-qfp6c podName:d1be4cd1-a95a-4507-8159-0c80944c37c9 nodeName:}" failed. No retries permitted until 2026-07-29 06:22:40.964326971 +0000 UTC m=+4.041958720 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qfp6c" (UniqueName: "kubernetes.io/projected/d1be4cd1-a95a-4507-8159-0c80944c37c9-kube-api-access-qfp6c") pod "network-check-target-njd8x" (UID: "d1be4cd1-a95a-4507-8159-0c80944c37c9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 06:22:40.264880 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:40.264830    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode191231b_69c1_4a7e_baf3_d46d529fe7d6.slice/crio-895eb7cc777674def39bf101d44f5fed31aa0b2da456e3734b983a4feb58867a WatchSource:0}: Error finding container 895eb7cc777674def39bf101d44f5fed31aa0b2da456e3734b983a4feb58867a: Status 404 returned error can't find the container with id 895eb7cc777674def39bf101d44f5fed31aa0b2da456e3734b983a4feb58867a
Jul 29 06:22:40.265528 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:40.265502    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a2a21ed_615a_4783_9a2a_b57804724f79.slice/crio-63212fb96ddf107e31a192dd84fa488914b8c44201e6dee1247dd6cd78f39a28 WatchSource:0}: Error finding container 63212fb96ddf107e31a192dd84fa488914b8c44201e6dee1247dd6cd78f39a28: Status 404 returned error can't find the container with id 63212fb96ddf107e31a192dd84fa488914b8c44201e6dee1247dd6cd78f39a28
Jul 29 06:22:40.266510 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:40.266474    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1673ef43_e0e0_4677_b830_7e8243686336.slice/crio-aabac1edbcfe6bc12e40603f5c5c43581eab1d5af974a7d469d0b567d7259afc WatchSource:0}: Error finding container aabac1edbcfe6bc12e40603f5c5c43581eab1d5af974a7d469d0b567d7259afc: Status 404 returned error can't find the container with id aabac1edbcfe6bc12e40603f5c5c43581eab1d5af974a7d469d0b567d7259afc
Jul 29 06:22:40.267429 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:40.267404    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6200726d_f1bf_43ba_bf59_6abb0d138092.slice/crio-e0baf7dcbe7d8fddf0590ac9fcaafcce3e954da83dae3717fd894b65c85d6b7c WatchSource:0}: Error finding container e0baf7dcbe7d8fddf0590ac9fcaafcce3e954da83dae3717fd894b65c85d6b7c: Status 404 returned error can't find the container with id e0baf7dcbe7d8fddf0590ac9fcaafcce3e954da83dae3717fd894b65c85d6b7c
Jul 29 06:22:40.272658 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:40.272637    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f2b7f0_1e62_43fd_99b0_78be60aa9881.slice/crio-c5644c1dab396f0fadad4452acb78702b14248d9d672110b14afc5ffce9a8b37 WatchSource:0}: Error finding container c5644c1dab396f0fadad4452acb78702b14248d9d672110b14afc5ffce9a8b37: Status 404 returned error can't find the container with id c5644c1dab396f0fadad4452acb78702b14248d9d672110b14afc5ffce9a8b37
Jul 29 06:22:40.273931 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:22:40.273903    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a95b6af_aa8b_4f61_b13b_5cb4aff9f6fa.slice/crio-808fe92845bcb488f5f70cf7ba78299941a4dd872081378c2fded5d58d75f40d WatchSource:0}: Error finding container 808fe92845bcb488f5f70cf7ba78299941a4dd872081378c2fded5d58d75f40d: Status 404 returned error can't find the container with id 808fe92845bcb488f5f70cf7ba78299941a4dd872081378c2fded5d58d75f40d
Jul 29 06:22:40.285798 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:40.285759    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 06:17:39 +0000 UTC" deadline="2028-04-05 00:33:09.883120226 +0000 UTC"
Jul 29 06:22:40.285798 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:40.285797    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14778h10m29.59732597s"
Jul 29 06:22:40.391847 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:40.391710    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzdr7"
Jul 29 06:22:40.391994 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:40.391972    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzdr7" podUID="0667dd8b-11f7-4b28-8f63-2f1bbbf98c56"
Jul 29 06:22:40.400649 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:40.400621    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-223.ec2.internal" event={"ID":"2ff94682e942fadc8425e32a6af57aed","Type":"ContainerStarted","Data":"cee0d23fae088c73db806ac670fe1300b27d221280a1f1aa181c68b1d665f9e1"}
Jul 29 06:22:40.401966 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:40.401925    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqpxr" event={"ID":"a4c4d323-1adf-4831-8bd8-3ac96b72bc08","Type":"ContainerStarted","Data":"50bb237581b90afc0f513f9e8acf65cd136cd83aaa2848ac5598b1b2f856c7a3"}
Jul 29 06:22:40.404208 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:40.404181    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7b9qz" event={"ID":"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa","Type":"ContainerStarted","Data":"808fe92845bcb488f5f70cf7ba78299941a4dd872081378c2fded5d58d75f40d"}
Jul 29 06:22:40.405342 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:40.405321    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-69xfb" event={"ID":"b9f2b7f0-1e62-43fd-99b0-78be60aa9881","Type":"ContainerStarted","Data":"c5644c1dab396f0fadad4452acb78702b14248d9d672110b14afc5ffce9a8b37"}
Jul 29 06:22:40.406307 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:40.406284    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qrnmf" event={"ID":"2079ff1a-5b9e-439d-bd6c-02c73c344340","Type":"ContainerStarted","Data":"941602ba14bce13eb09508c4ce95757ef5856cde74b5061d75e7ab5fd17ea663"}
Jul 29 06:22:40.407363 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:40.407339    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wtdbb" event={"ID":"85622ea0-6a2b-47d9-a05e-1265b9d86baa","Type":"ContainerStarted","Data":"f0537feb1a7bd00a2716ac6444a0a25279052bad35cf72fae45fbe3840e13aca"}
Jul 29 06:22:40.408341 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:40.408315    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rkphn" event={"ID":"4a2a21ed-615a-4783-9a2a-b57804724f79","Type":"ContainerStarted","Data":"63212fb96ddf107e31a192dd84fa488914b8c44201e6dee1247dd6cd78f39a28"}
Jul 29 06:22:40.409403 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:40.409384    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk" event={"ID":"6200726d-f1bf-43ba-bf59-6abb0d138092","Type":"ContainerStarted","Data":"e0baf7dcbe7d8fddf0590ac9fcaafcce3e954da83dae3717fd894b65c85d6b7c"}
Jul 29 06:22:40.410669 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:40.410649    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cr75t" event={"ID":"1673ef43-e0e0-4677-b830-7e8243686336","Type":"ContainerStarted","Data":"aabac1edbcfe6bc12e40603f5c5c43581eab1d5af974a7d469d0b567d7259afc"}
Jul 29 06:22:40.411820 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:40.411799    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wfqxw" event={"ID":"e191231b-69c1-4a7e-baf3-d46d529fe7d6","Type":"ContainerStarted","Data":"895eb7cc777674def39bf101d44f5fed31aa0b2da456e3734b983a4feb58867a"}
Jul 29 06:22:40.871150 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:40.871110    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0667dd8b-11f7-4b28-8f63-2f1bbbf98c56-original-pull-secret\") pod \"global-pull-secret-syncer-dzdr7\" (UID: \"0667dd8b-11f7-4b28-8f63-2f1bbbf98c56\") " pod="kube-system/global-pull-secret-syncer-dzdr7"
Jul 29 06:22:40.871309 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:40.871290    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 06:22:40.871381 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:40.871352    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0667dd8b-11f7-4b28-8f63-2f1bbbf98c56-original-pull-secret podName:0667dd8b-11f7-4b28-8f63-2f1bbbf98c56 nodeName:}" failed. No retries permitted until 2026-07-29 06:22:42.871333699 +0000 UTC m=+5.948965460 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0667dd8b-11f7-4b28-8f63-2f1bbbf98c56-original-pull-secret") pod "global-pull-secret-syncer-dzdr7" (UID: "0667dd8b-11f7-4b28-8f63-2f1bbbf98c56") : object "kube-system"/"original-pull-secret" not registered
Jul 29 06:22:40.977153 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:40.977113    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfp6c\" (UniqueName: \"kubernetes.io/projected/d1be4cd1-a95a-4507-8159-0c80944c37c9-kube-api-access-qfp6c\") pod \"network-check-target-njd8x\" (UID: \"d1be4cd1-a95a-4507-8159-0c80944c37c9\") " pod="openshift-network-diagnostics/network-check-target-njd8x"
Jul 29 06:22:40.977329 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:40.977171    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-metrics-certs\") pod \"network-metrics-daemon-vcg74\" (UID: \"2ceff9c5-890d-4d2b-85ee-09bcd9e02833\") " pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:22:40.977385 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:40.977330    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 06:22:40.977433 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:40.977391    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-metrics-certs podName:2ceff9c5-890d-4d2b-85ee-09bcd9e02833 nodeName:}" failed. No retries permitted until 2026-07-29 06:22:42.977372997 +0000 UTC m=+6.055004746 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-metrics-certs") pod "network-metrics-daemon-vcg74" (UID: "2ceff9c5-890d-4d2b-85ee-09bcd9e02833") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 06:22:40.977715 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:40.977697    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 06:22:40.977799 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:40.977718    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 06:22:40.977799 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:40.977727    2563 projected.go:196] Error preparing data for projected volume kube-api-access-qfp6c for pod openshift-network-diagnostics/network-check-target-njd8x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 06:22:40.977799 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:40.977762    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1be4cd1-a95a-4507-8159-0c80944c37c9-kube-api-access-qfp6c podName:d1be4cd1-a95a-4507-8159-0c80944c37c9 nodeName:}" failed. No retries permitted until 2026-07-29 06:22:42.977751802 +0000 UTC m=+6.055383550 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qfp6c" (UniqueName: "kubernetes.io/projected/d1be4cd1-a95a-4507-8159-0c80944c37c9-kube-api-access-qfp6c") pod "network-check-target-njd8x" (UID: "d1be4cd1-a95a-4507-8159-0c80944c37c9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 06:22:41.286168 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:41.286061    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 06:17:39 +0000 UTC" deadline="2028-05-12 00:36:18.8560272 +0000 UTC"
Jul 29 06:22:41.286168 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:41.286122    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15666h13m37.569909456s"
Jul 29 06:22:41.391996 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:41.391968    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njd8x"
Jul 29 06:22:41.392156 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:41.392090    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njd8x" podUID="d1be4cd1-a95a-4507-8159-0c80944c37c9"
Jul 29 06:22:41.392229 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:41.392144    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:22:41.392308 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:41.392285    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcg74" podUID="2ceff9c5-890d-4d2b-85ee-09bcd9e02833"
Jul 29 06:22:41.422435 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:41.422392    2563 generic.go:356] "Generic (PLEG): container finished" podID="bc0f4deacee416934506716d269ff92f" containerID="b9a87e4c75eea060221abe120bd5310d723c1826a474f03acbefea62a5ea70b7" exitCode=0
Jul 29 06:22:41.423400 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:41.423374    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-223.ec2.internal" event={"ID":"bc0f4deacee416934506716d269ff92f","Type":"ContainerDied","Data":"b9a87e4c75eea060221abe120bd5310d723c1826a474f03acbefea62a5ea70b7"}
Jul 29 06:22:41.435377 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:41.435324    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-223.ec2.internal" podStartSLOduration=3.43530614 podStartE2EDuration="3.43530614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 06:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:22:40.415171765 +0000 UTC m=+3.492803532" watchObservedRunningTime="2026-07-29 06:22:41.43530614 +0000 UTC m=+4.512937907"
Jul 29 06:22:42.391808 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:42.391341    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzdr7"
Jul 29 06:22:42.391808 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:42.391461    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzdr7" podUID="0667dd8b-11f7-4b28-8f63-2f1bbbf98c56"
Jul 29 06:22:42.436978 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:42.436943    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-223.ec2.internal" event={"ID":"bc0f4deacee416934506716d269ff92f","Type":"ContainerStarted","Data":"f8d8cf914f116e8f8fafa91612b461cc9570f2cfdcc930b469e7026f4ca01485"}
Jul 29 06:22:42.449892 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:42.449806    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-223.ec2.internal" podStartSLOduration=4.44978559 podStartE2EDuration="4.44978559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 06:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:22:42.449135621 +0000 UTC m=+5.526767389" watchObservedRunningTime="2026-07-29 06:22:42.44978559 +0000 UTC m=+5.527417360"
Jul 29 06:22:42.892004 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:42.891969    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0667dd8b-11f7-4b28-8f63-2f1bbbf98c56-original-pull-secret\") pod \"global-pull-secret-syncer-dzdr7\" (UID: \"0667dd8b-11f7-4b28-8f63-2f1bbbf98c56\") " pod="kube-system/global-pull-secret-syncer-dzdr7"
Jul 29 06:22:42.892198 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:42.892163    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 06:22:42.892268 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:42.892230    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0667dd8b-11f7-4b28-8f63-2f1bbbf98c56-original-pull-secret podName:0667dd8b-11f7-4b28-8f63-2f1bbbf98c56 nodeName:}" failed. No retries permitted until 2026-07-29 06:22:46.892211486 +0000 UTC m=+9.969843235 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0667dd8b-11f7-4b28-8f63-2f1bbbf98c56-original-pull-secret") pod "global-pull-secret-syncer-dzdr7" (UID: "0667dd8b-11f7-4b28-8f63-2f1bbbf98c56") : object "kube-system"/"original-pull-secret" not registered
Jul 29 06:22:42.993235 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:42.993195    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfp6c\" (UniqueName: \"kubernetes.io/projected/d1be4cd1-a95a-4507-8159-0c80944c37c9-kube-api-access-qfp6c\") pod \"network-check-target-njd8x\" (UID: \"d1be4cd1-a95a-4507-8159-0c80944c37c9\") " pod="openshift-network-diagnostics/network-check-target-njd8x"
Jul 29 06:22:42.993410 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:42.993250    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-metrics-certs\") pod \"network-metrics-daemon-vcg74\" (UID: \"2ceff9c5-890d-4d2b-85ee-09bcd9e02833\") " pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:22:42.993410 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:42.993378    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 06:22:42.993512 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:42.993437    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-metrics-certs podName:2ceff9c5-890d-4d2b-85ee-09bcd9e02833 nodeName:}" failed. No retries permitted until 2026-07-29 06:22:46.993417571 +0000 UTC m=+10.071049325 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-metrics-certs") pod "network-metrics-daemon-vcg74" (UID: "2ceff9c5-890d-4d2b-85ee-09bcd9e02833") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 06:22:42.993873 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:42.993833    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 06:22:42.993873 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:42.993873    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 06:22:42.994012 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:42.993886    2563 projected.go:196] Error preparing data for projected volume kube-api-access-qfp6c for pod openshift-network-diagnostics/network-check-target-njd8x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 06:22:42.994012 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:42.993930    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1be4cd1-a95a-4507-8159-0c80944c37c9-kube-api-access-qfp6c podName:d1be4cd1-a95a-4507-8159-0c80944c37c9 nodeName:}" failed. No retries permitted until 2026-07-29 06:22:46.9939165 +0000 UTC m=+10.071548258 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qfp6c" (UniqueName: "kubernetes.io/projected/d1be4cd1-a95a-4507-8159-0c80944c37c9-kube-api-access-qfp6c") pod "network-check-target-njd8x" (UID: "d1be4cd1-a95a-4507-8159-0c80944c37c9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 06:22:43.395253 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:43.395225    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:22:43.395704 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:43.395344    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcg74" podUID="2ceff9c5-890d-4d2b-85ee-09bcd9e02833"
Jul 29 06:22:43.395762 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:43.395718    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njd8x"
Jul 29 06:22:43.395831 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:43.395784    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njd8x" podUID="d1be4cd1-a95a-4507-8159-0c80944c37c9"
Jul 29 06:22:44.392961 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:44.392932    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzdr7"
Jul 29 06:22:44.393133 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:44.393052    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzdr7" podUID="0667dd8b-11f7-4b28-8f63-2f1bbbf98c56"
Jul 29 06:22:45.392238 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:45.392208    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njd8x"
Jul 29 06:22:45.392659 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:45.392332    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njd8x" podUID="d1be4cd1-a95a-4507-8159-0c80944c37c9"
Jul 29 06:22:45.394448 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:45.394406    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:22:45.394828 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:45.394797    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcg74" podUID="2ceff9c5-890d-4d2b-85ee-09bcd9e02833"
Jul 29 06:22:46.391365 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:46.391327    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzdr7"
Jul 29 06:22:46.391530 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:46.391439    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzdr7" podUID="0667dd8b-11f7-4b28-8f63-2f1bbbf98c56"
Jul 29 06:22:46.925962 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:46.925388    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0667dd8b-11f7-4b28-8f63-2f1bbbf98c56-original-pull-secret\") pod \"global-pull-secret-syncer-dzdr7\" (UID: \"0667dd8b-11f7-4b28-8f63-2f1bbbf98c56\") " pod="kube-system/global-pull-secret-syncer-dzdr7"
Jul 29 06:22:46.925962 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:46.925549    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 06:22:46.925962 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:46.925608    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0667dd8b-11f7-4b28-8f63-2f1bbbf98c56-original-pull-secret podName:0667dd8b-11f7-4b28-8f63-2f1bbbf98c56 nodeName:}" failed. No retries permitted until 2026-07-29 06:22:54.925590904 +0000 UTC m=+18.003222652 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0667dd8b-11f7-4b28-8f63-2f1bbbf98c56-original-pull-secret") pod "global-pull-secret-syncer-dzdr7" (UID: "0667dd8b-11f7-4b28-8f63-2f1bbbf98c56") : object "kube-system"/"original-pull-secret" not registered
Jul 29 06:22:47.025776 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:47.025735    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfp6c\" (UniqueName: \"kubernetes.io/projected/d1be4cd1-a95a-4507-8159-0c80944c37c9-kube-api-access-qfp6c\") pod \"network-check-target-njd8x\" (UID: \"d1be4cd1-a95a-4507-8159-0c80944c37c9\") " pod="openshift-network-diagnostics/network-check-target-njd8x"
Jul 29 06:22:47.025776 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:47.025784    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-metrics-certs\") pod \"network-metrics-daemon-vcg74\" (UID: \"2ceff9c5-890d-4d2b-85ee-09bcd9e02833\") " pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:22:47.026030 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:47.026009    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 06:22:47.026030 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:47.026031    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 06:22:47.026121 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:47.026044    2563 projected.go:196] Error preparing data for projected volume kube-api-access-qfp6c for pod openshift-network-diagnostics/network-check-target-njd8x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 06:22:47.026121 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:47.026104    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1be4cd1-a95a-4507-8159-0c80944c37c9-kube-api-access-qfp6c podName:d1be4cd1-a95a-4507-8159-0c80944c37c9 nodeName:}" failed. No retries permitted until 2026-07-29 06:22:55.026084761 +0000 UTC m=+18.103716526 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qfp6c" (UniqueName: "kubernetes.io/projected/d1be4cd1-a95a-4507-8159-0c80944c37c9-kube-api-access-qfp6c") pod "network-check-target-njd8x" (UID: "d1be4cd1-a95a-4507-8159-0c80944c37c9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 06:22:47.026635 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:47.026607    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 06:22:47.026750 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:47.026676    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-metrics-certs podName:2ceff9c5-890d-4d2b-85ee-09bcd9e02833 nodeName:}" failed. No retries permitted until 2026-07-29 06:22:55.026661024 +0000 UTC m=+18.104292775 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-metrics-certs") pod "network-metrics-daemon-vcg74" (UID: "2ceff9c5-890d-4d2b-85ee-09bcd9e02833") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 06:22:47.393098 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:47.393016    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njd8x"
Jul 29 06:22:47.393249 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:47.393104    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njd8x" podUID="d1be4cd1-a95a-4507-8159-0c80944c37c9"
Jul 29 06:22:47.393411 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:47.393393    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:22:47.393518 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:47.393500    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcg74" podUID="2ceff9c5-890d-4d2b-85ee-09bcd9e02833"
Jul 29 06:22:48.391907 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:48.391356    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzdr7"
Jul 29 06:22:48.391907 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:48.391482    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzdr7" podUID="0667dd8b-11f7-4b28-8f63-2f1bbbf98c56"
Jul 29 06:22:49.394405 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:49.394380    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njd8x"
Jul 29 06:22:49.394866 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:49.394484    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njd8x" podUID="d1be4cd1-a95a-4507-8159-0c80944c37c9"
Jul 29 06:22:49.394866 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:49.394562    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:22:49.394866 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:49.394665    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcg74" podUID="2ceff9c5-890d-4d2b-85ee-09bcd9e02833"
Jul 29 06:22:50.391578 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:50.391537    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzdr7"
Jul 29 06:22:50.391754 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:50.391705    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzdr7" podUID="0667dd8b-11f7-4b28-8f63-2f1bbbf98c56"
Jul 29 06:22:51.395147 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:51.395113    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:22:51.395549 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:51.395252    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcg74" podUID="2ceff9c5-890d-4d2b-85ee-09bcd9e02833"
Jul 29 06:22:51.395549 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:51.395287    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njd8x"
Jul 29 06:22:51.395549 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:51.395358    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njd8x" podUID="d1be4cd1-a95a-4507-8159-0c80944c37c9"
Jul 29 06:22:52.391674 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:52.391639    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzdr7"
Jul 29 06:22:52.391917 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:52.391744    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzdr7" podUID="0667dd8b-11f7-4b28-8f63-2f1bbbf98c56"
Jul 29 06:22:53.391570 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:53.391535    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njd8x"
Jul 29 06:22:53.391983 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:53.391550    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:22:53.391983 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:53.391655    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njd8x" podUID="d1be4cd1-a95a-4507-8159-0c80944c37c9"
Jul 29 06:22:53.391983 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:53.391720    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcg74" podUID="2ceff9c5-890d-4d2b-85ee-09bcd9e02833"
Jul 29 06:22:54.391984 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:54.391951    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzdr7"
Jul 29 06:22:54.392442 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:54.392078    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzdr7" podUID="0667dd8b-11f7-4b28-8f63-2f1bbbf98c56"
Jul 29 06:22:54.992259 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:54.992234    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0667dd8b-11f7-4b28-8f63-2f1bbbf98c56-original-pull-secret\") pod \"global-pull-secret-syncer-dzdr7\" (UID: \"0667dd8b-11f7-4b28-8f63-2f1bbbf98c56\") " pod="kube-system/global-pull-secret-syncer-dzdr7"
Jul 29 06:22:54.992383 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:54.992367    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 06:22:54.992441 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:54.992425    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0667dd8b-11f7-4b28-8f63-2f1bbbf98c56-original-pull-secret podName:0667dd8b-11f7-4b28-8f63-2f1bbbf98c56 nodeName:}" failed. No retries permitted until 2026-07-29 06:23:10.992410677 +0000 UTC m=+34.070042435 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0667dd8b-11f7-4b28-8f63-2f1bbbf98c56-original-pull-secret") pod "global-pull-secret-syncer-dzdr7" (UID: "0667dd8b-11f7-4b28-8f63-2f1bbbf98c56") : object "kube-system"/"original-pull-secret" not registered
Jul 29 06:22:55.092597 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:55.092575    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfp6c\" (UniqueName: \"kubernetes.io/projected/d1be4cd1-a95a-4507-8159-0c80944c37c9-kube-api-access-qfp6c\") pod \"network-check-target-njd8x\" (UID: \"d1be4cd1-a95a-4507-8159-0c80944c37c9\") " pod="openshift-network-diagnostics/network-check-target-njd8x"
Jul 29 06:22:55.092675 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:55.092614    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-metrics-certs\") pod \"network-metrics-daemon-vcg74\" (UID: \"2ceff9c5-890d-4d2b-85ee-09bcd9e02833\") " pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:22:55.092738 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:55.092722    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 06:22:55.092792 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:55.092738    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 06:22:55.092792 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:55.092763    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 06:22:55.092792 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:55.092770    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-metrics-certs podName:2ceff9c5-890d-4d2b-85ee-09bcd9e02833 nodeName:}" failed. No retries permitted until 2026-07-29 06:23:11.092755168 +0000 UTC m=+34.170386912 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-metrics-certs") pod "network-metrics-daemon-vcg74" (UID: "2ceff9c5-890d-4d2b-85ee-09bcd9e02833") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 06:22:55.092792 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:55.092777    2563 projected.go:196] Error preparing data for projected volume kube-api-access-qfp6c for pod openshift-network-diagnostics/network-check-target-njd8x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 06:22:55.092964 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:55.092823    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1be4cd1-a95a-4507-8159-0c80944c37c9-kube-api-access-qfp6c podName:d1be4cd1-a95a-4507-8159-0c80944c37c9 nodeName:}" failed. No retries permitted until 2026-07-29 06:23:11.092807824 +0000 UTC m=+34.170439583 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qfp6c" (UniqueName: "kubernetes.io/projected/d1be4cd1-a95a-4507-8159-0c80944c37c9-kube-api-access-qfp6c") pod "network-check-target-njd8x" (UID: "d1be4cd1-a95a-4507-8159-0c80944c37c9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 06:22:55.391272 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:55.391241    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njd8x"
Jul 29 06:22:55.391432 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:55.391245    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:22:55.391432 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:55.391358    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njd8x" podUID="d1be4cd1-a95a-4507-8159-0c80944c37c9"
Jul 29 06:22:55.391555 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:55.391464    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcg74" podUID="2ceff9c5-890d-4d2b-85ee-09bcd9e02833"
Jul 29 06:22:56.391151 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:56.391111    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzdr7"
Jul 29 06:22:56.391600 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:56.391237    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzdr7" podUID="0667dd8b-11f7-4b28-8f63-2f1bbbf98c56"
Jul 29 06:22:57.392637 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:57.392599    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:22:57.392999 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:57.392732    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcg74" podUID="2ceff9c5-890d-4d2b-85ee-09bcd9e02833"
Jul 29 06:22:57.392999 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:57.392896    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njd8x"
Jul 29 06:22:57.393087 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:57.392997    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njd8x" podUID="d1be4cd1-a95a-4507-8159-0c80944c37c9"
Jul 29 06:22:58.391757 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:58.391712    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzdr7"
Jul 29 06:22:58.391957 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:58.391864    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzdr7" podUID="0667dd8b-11f7-4b28-8f63-2f1bbbf98c56"
Jul 29 06:22:59.395763 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:59.395511    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:22:59.396373 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:59.395574    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njd8x"
Jul 29 06:22:59.396373 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:59.395845    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcg74" podUID="2ceff9c5-890d-4d2b-85ee-09bcd9e02833"
Jul 29 06:22:59.396373 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:22:59.395935    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njd8x" podUID="d1be4cd1-a95a-4507-8159-0c80944c37c9"
Jul 29 06:22:59.470466 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:59.470434    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqpxr" event={"ID":"a4c4d323-1adf-4831-8bd8-3ac96b72bc08","Type":"ContainerStarted","Data":"229b57c26901fdff9601e619afa5b07eb0c02478caae8c9e33ac2d7c5439928c"}
Jul 29 06:22:59.472219 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:59.472186    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7b9qz" event={"ID":"2a95b6af-aa8b-4f61-b13b-5cb4aff9f6fa","Type":"ContainerStarted","Data":"876913fd705bc8d034b23cfdb1c4134daa894acafb8b6dcf55fb9ee4ef097593"}
Jul 29 06:22:59.473814 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:59.473780    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qrnmf" event={"ID":"2079ff1a-5b9e-439d-bd6c-02c73c344340","Type":"ContainerStarted","Data":"171645cea0f7f49275ae42f1ff6eac75ad7390dd3620c7d5dc988148f1cc1b03"}
Jul 29 06:22:59.479045 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:59.479017    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wtdbb" event={"ID":"85622ea0-6a2b-47d9-a05e-1265b9d86baa","Type":"ContainerStarted","Data":"57b19e79829734f8d1308f20a630bdaee96af581c635bd809d30236102d29183"}
Jul 29 06:22:59.480573 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:59.480551    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rkphn" event={"ID":"4a2a21ed-615a-4783-9a2a-b57804724f79","Type":"ContainerStarted","Data":"369e83f8ff71e45014ddaf7923d8b6fcc7f695b30718994d8ba5dc1e5bcdf932"}
Jul 29 06:22:59.482631 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:59.482602    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk" event={"ID":"6200726d-f1bf-43ba-bf59-6abb0d138092","Type":"ContainerStarted","Data":"63ae807fd02827e058ed1963055365babbebc787f05b39309bc26d9d6db8d4fe"}
Jul 29 06:22:59.482715 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:59.482692    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk" event={"ID":"6200726d-f1bf-43ba-bf59-6abb0d138092","Type":"ContainerStarted","Data":"ffeaaafd0b3b700955e8d1f94651e078c5a2ee1b7cf4e5144d184e4a2b944662"}
Jul 29 06:22:59.486540 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:59.486512    2563 generic.go:356] "Generic (PLEG): container finished" podID="1673ef43-e0e0-4677-b830-7e8243686336" containerID="c2866a7eeb6cff8d48492aa957611d50d34657c56ffd2d6184479ce256375307" exitCode=0
Jul 29 06:22:59.486625 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:59.486589    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cr75t" event={"ID":"1673ef43-e0e0-4677-b830-7e8243686336","Type":"ContainerDied","Data":"c2866a7eeb6cff8d48492aa957611d50d34657c56ffd2d6184479ce256375307"}
Jul 29 06:22:59.488095 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:59.487889    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wfqxw" event={"ID":"e191231b-69c1-4a7e-baf3-d46d529fe7d6","Type":"ContainerStarted","Data":"b3e3f1daf5f49f6d28020c4cffcc4e4b6eb9c0900cd83d32280fdd4ce2570ad1"}
Jul 29 06:22:59.507654 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:59.507599    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7b9qz" podStartSLOduration=3.7639529879999998 podStartE2EDuration="22.5075787s" totalImagesPullingTime="18.743625712s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 06:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:22:59.489733613 +0000 UTC m=+22.567365379" watchObservedRunningTime="2026-07-29 06:22:59.5075787 +0000 UTC m=+22.585210471"
Jul 29 06:22:59.519526 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:59.519487    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rkphn" podStartSLOduration=3.766943246 podStartE2EDuration="22.51947268s" totalImagesPullingTime="18.752529434s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 06:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:22:59.519140942 +0000 UTC m=+22.596772709" watchObservedRunningTime="2026-07-29 06:22:59.51947268 +0000 UTC m=+22.597104447"
Jul 29 06:22:59.533585 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:59.533535    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wtdbb" podStartSLOduration=3.558746687 podStartE2EDuration="22.533516764s" totalImagesPullingTime="18.974770077s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 06:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:22:59.533389369 +0000 UTC m=+22.611021135" watchObservedRunningTime="2026-07-29 06:22:59.533516764 +0000 UTC m=+22.611148532"
Jul 29 06:22:59.553373 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:59.553331    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qrnmf" podStartSLOduration=3.813132745 podStartE2EDuration="22.55332161s" totalImagesPullingTime="18.740188865s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 06:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:22:59.546892275 +0000 UTC m=+22.624524041" watchObservedRunningTime="2026-07-29 06:22:59.55332161 +0000 UTC m=+22.630953379"
Jul 29 06:22:59.563250 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:22:59.563215    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wfqxw" podStartSLOduration=7.741568013 podStartE2EDuration="22.563205028s" totalImagesPullingTime="14.821637015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:22:59.563192289 +0000 UTC m=+22.640824058" watchObservedRunningTime="2026-07-29 06:22:59.563205028 +0000 UTC m=+22.640836795"
Jul 29 06:23:00.391623 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:00.391589    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzdr7"
Jul 29 06:23:00.391762 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:00.391740    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzdr7" podUID="0667dd8b-11f7-4b28-8f63-2f1bbbf98c56"
Jul 29 06:23:00.460892 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:00.460842    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 29 06:23:00.491592 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:00.491540    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-69xfb" event={"ID":"b9f2b7f0-1e62-43fd-99b0-78be60aa9881","Type":"ContainerStarted","Data":"64b7c44a769042a9e4b74662316530a479218aff0e14d8e6129ee720c4764d9f"}
Jul 29 06:23:00.495159 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:00.495134    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk" event={"ID":"6200726d-f1bf-43ba-bf59-6abb0d138092","Type":"ContainerStarted","Data":"fc8d0f93f60c7c2c661c2b3edf03eed64716bb893861e189afcd5f027c6ff832"}
Jul 29 06:23:00.495280 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:00.495215    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk" event={"ID":"6200726d-f1bf-43ba-bf59-6abb0d138092","Type":"ContainerStarted","Data":"fa1a7fe9638c340d1024244dbba89956f61516170ece2d5af333059494459865"}
Jul 29 06:23:00.495280 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:00.495269    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk" event={"ID":"6200726d-f1bf-43ba-bf59-6abb0d138092","Type":"ContainerStarted","Data":"f30dfa6d6a79e1c7b201b1e2950a67c66b4390a65e60fb2c8a900a8c9d7b01ce"}
Jul 29 06:23:00.495388 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:00.495318    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk" event={"ID":"6200726d-f1bf-43ba-bf59-6abb0d138092","Type":"ContainerStarted","Data":"d14be7c94137bb2ff48d2f63c712b00537d31a23ce36d6222697e860571cacee"}
Jul 29 06:23:00.497132 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:00.497106    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqpxr" event={"ID":"a4c4d323-1adf-4831-8bd8-3ac96b72bc08","Type":"ContainerStarted","Data":"7abaa9f96a5af79c286433708beb5dbb88d04da28a9a170cefebb3e970af6572"}
Jul 29 06:23:00.505338 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:00.505284    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-69xfb" podStartSLOduration=4.758317057 podStartE2EDuration="23.505267152s" totalImagesPullingTime="18.746950095s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 06:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:23:00.504234763 +0000 UTC m=+23.581866529" watchObservedRunningTime="2026-07-29 06:23:00.505267152 +0000 UTC m=+23.582898920"
Jul 29 06:23:01.315811 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:01.315681    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T06:23:00.460887797Z","UUID":"8a0da25f-4eab-4f06-a8ff-e3441cb4fce9","Handler":null,"Name":"","Endpoint":""}
Jul 29 06:23:01.317765 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:01.317743    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 29 06:23:01.317911 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:01.317786    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 29 06:23:01.391400 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:01.391352    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njd8x"
Jul 29 06:23:01.391640 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:01.391477    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njd8x" podUID="d1be4cd1-a95a-4507-8159-0c80944c37c9"
Jul 29 06:23:01.391640 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:01.391537    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:23:01.391750 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:01.391644    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcg74" podUID="2ceff9c5-890d-4d2b-85ee-09bcd9e02833"
Jul 29 06:23:02.319214 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:02.319182    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wfqxw"
Jul 29 06:23:02.320015 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:02.319982    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wfqxw"
Jul 29 06:23:02.391128 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:02.391096    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzdr7"
Jul 29 06:23:02.391280 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:02.391219    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzdr7" podUID="0667dd8b-11f7-4b28-8f63-2f1bbbf98c56"
Jul 29 06:23:02.503659 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:02.503574    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqpxr" event={"ID":"a4c4d323-1adf-4831-8bd8-3ac96b72bc08","Type":"ContainerStarted","Data":"a06e2e8677cd94be82df09f01e60973051c4e9078aaffaabc1145726be1cf329"}
Jul 29 06:23:02.503866 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:02.503836    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wfqxw"
Jul 29 06:23:02.504394 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:02.504376    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wfqxw"
Jul 29 06:23:02.531601 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:02.531561    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jqpxr" podStartSLOduration=4.4652973750000005 podStartE2EDuration="25.531543225s" totalImagesPullingTime="21.06624585s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 06:22:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:23:02.519429853 +0000 UTC m=+25.597061622" watchObservedRunningTime="2026-07-29 06:23:02.531543225 +0000 UTC m=+25.609174994"
Jul 29 06:23:03.391806 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:03.391773    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:23:03.392253 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:03.391787    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njd8x"
Jul 29 06:23:03.392253 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:03.391940    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcg74" podUID="2ceff9c5-890d-4d2b-85ee-09bcd9e02833"
Jul 29 06:23:03.392253 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:03.392056    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njd8x" podUID="d1be4cd1-a95a-4507-8159-0c80944c37c9"
Jul 29 06:23:04.391609 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:04.391438    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzdr7"
Jul 29 06:23:04.391826 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:04.391698    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzdr7" podUID="0667dd8b-11f7-4b28-8f63-2f1bbbf98c56"
Jul 29 06:23:04.511513 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:04.511473    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk" event={"ID":"6200726d-f1bf-43ba-bf59-6abb0d138092","Type":"ContainerStarted","Data":"c7a2ff30ef6d09f4d7a69ebb684fb7785b36fe1466a699cbe501f1947a7b23cb"}
Jul 29 06:23:04.513178 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:04.513150    2563 generic.go:356] "Generic (PLEG): container finished" podID="1673ef43-e0e0-4677-b830-7e8243686336" containerID="8f1b9c8bb0bb0a0284fc85fd598606df6b3f1777cf99feeba5a7e4319e6a3510" exitCode=0
Jul 29 06:23:04.513303 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:04.513240    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cr75t" event={"ID":"1673ef43-e0e0-4677-b830-7e8243686336","Type":"ContainerDied","Data":"8f1b9c8bb0bb0a0284fc85fd598606df6b3f1777cf99feeba5a7e4319e6a3510"}
Jul 29 06:23:05.394345 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:05.394278    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:23:05.394345 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:05.394319    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njd8x"
Jul 29 06:23:05.394542 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:05.394401    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcg74" podUID="2ceff9c5-890d-4d2b-85ee-09bcd9e02833"
Jul 29 06:23:05.394542 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:05.394513    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njd8x" podUID="d1be4cd1-a95a-4507-8159-0c80944c37c9"
Jul 29 06:23:06.391288 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:06.391119    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzdr7"
Jul 29 06:23:06.391913 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:06.391360    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzdr7" podUID="0667dd8b-11f7-4b28-8f63-2f1bbbf98c56"
Jul 29 06:23:06.520051 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:06.520023    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk" event={"ID":"6200726d-f1bf-43ba-bf59-6abb0d138092","Type":"ContainerStarted","Data":"d726abe3c3140c01e96f108b11ce19b36a513323ab84f32706d3e095a2f60604"}
Jul 29 06:23:06.520295 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:06.520273    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:23:06.522827 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:06.522802    2563 generic.go:356] "Generic (PLEG): container finished" podID="1673ef43-e0e0-4677-b830-7e8243686336" containerID="281d9c687a07f3fa124f2554ba008d67c8d86c1acc8a3a080de1423e5746bdb3" exitCode=0
Jul 29 06:23:06.522957 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:06.522838    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cr75t" event={"ID":"1673ef43-e0e0-4677-b830-7e8243686336","Type":"ContainerDied","Data":"281d9c687a07f3fa124f2554ba008d67c8d86c1acc8a3a080de1423e5746bdb3"}
Jul 29 06:23:06.537585 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:06.537563    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:23:06.546160 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:06.546123    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk" podStartSLOduration=10.750004518 podStartE2EDuration="29.546112073s" totalImagesPullingTime="18.796107555s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 06:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:23:06.545933796 +0000 UTC m=+29.623565565" watchObservedRunningTime="2026-07-29 06:23:06.546112073 +0000 UTC m=+29.623743839"
Jul 29 06:23:07.391983 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:07.391947    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njd8x"
Jul 29 06:23:07.392406 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:07.392051    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njd8x" podUID="d1be4cd1-a95a-4507-8159-0c80944c37c9"
Jul 29 06:23:07.392406 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:07.392104    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:23:07.392406 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:07.392197    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcg74" podUID="2ceff9c5-890d-4d2b-85ee-09bcd9e02833"
Jul 29 06:23:07.525470 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:07.525371    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:23:07.525575 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:07.525508    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:23:07.545668 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:07.545641    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:23:08.250310 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:08.250273    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vcg74"]
Jul 29 06:23:08.250485 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:08.250466    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:23:08.250623 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:08.250597    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcg74" podUID="2ceff9c5-890d-4d2b-85ee-09bcd9e02833"
Jul 29 06:23:08.253336 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:08.253306    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-njd8x"]
Jul 29 06:23:08.253489 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:08.253474    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njd8x"
Jul 29 06:23:08.253592 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:08.253573    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njd8x" podUID="d1be4cd1-a95a-4507-8159-0c80944c37c9"
Jul 29 06:23:08.254556 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:08.254535    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dzdr7"]
Jul 29 06:23:08.254670 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:08.254618    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzdr7"
Jul 29 06:23:08.254727 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:08.254681    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzdr7" podUID="0667dd8b-11f7-4b28-8f63-2f1bbbf98c56"
Jul 29 06:23:08.529993 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:08.529948    2563 generic.go:356] "Generic (PLEG): container finished" podID="1673ef43-e0e0-4677-b830-7e8243686336" containerID="ec8418e945f7694664352a098f56f14ce860734cf232ce123d2743e45c57e0be" exitCode=0
Jul 29 06:23:08.530374 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:08.530015    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cr75t" event={"ID":"1673ef43-e0e0-4677-b830-7e8243686336","Type":"ContainerDied","Data":"ec8418e945f7694664352a098f56f14ce860734cf232ce123d2743e45c57e0be"}
Jul 29 06:23:09.391425 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:09.391201    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:23:09.391616 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:09.391551    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcg74" podUID="2ceff9c5-890d-4d2b-85ee-09bcd9e02833"
Jul 29 06:23:10.391628 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:10.391535    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njd8x"
Jul 29 06:23:10.392054 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:10.391725    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njd8x" podUID="d1be4cd1-a95a-4507-8159-0c80944c37c9"
Jul 29 06:23:10.392054 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:10.391534    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzdr7"
Jul 29 06:23:10.392054 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:10.392002    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzdr7" podUID="0667dd8b-11f7-4b28-8f63-2f1bbbf98c56"
Jul 29 06:23:11.009092 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.009053    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0667dd8b-11f7-4b28-8f63-2f1bbbf98c56-original-pull-secret\") pod \"global-pull-secret-syncer-dzdr7\" (UID: \"0667dd8b-11f7-4b28-8f63-2f1bbbf98c56\") " pod="kube-system/global-pull-secret-syncer-dzdr7"
Jul 29 06:23:11.009260 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:11.009238    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 06:23:11.009337 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:11.009313    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0667dd8b-11f7-4b28-8f63-2f1bbbf98c56-original-pull-secret podName:0667dd8b-11f7-4b28-8f63-2f1bbbf98c56 nodeName:}" failed. No retries permitted until 2026-07-29 06:23:43.009290918 +0000 UTC m=+66.086922666 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0667dd8b-11f7-4b28-8f63-2f1bbbf98c56-original-pull-secret") pod "global-pull-secret-syncer-dzdr7" (UID: "0667dd8b-11f7-4b28-8f63-2f1bbbf98c56") : object "kube-system"/"original-pull-secret" not registered
Jul 29 06:23:11.109594 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.109553    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-metrics-certs\") pod \"network-metrics-daemon-vcg74\" (UID: \"2ceff9c5-890d-4d2b-85ee-09bcd9e02833\") " pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:23:11.109769 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.109655    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfp6c\" (UniqueName: \"kubernetes.io/projected/d1be4cd1-a95a-4507-8159-0c80944c37c9-kube-api-access-qfp6c\") pod \"network-check-target-njd8x\" (UID: \"d1be4cd1-a95a-4507-8159-0c80944c37c9\") " pod="openshift-network-diagnostics/network-check-target-njd8x"
Jul 29 06:23:11.109769 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:11.109706    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 06:23:11.109911 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:11.109771    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 06:23:11.109911 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:11.109786    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-metrics-certs podName:2ceff9c5-890d-4d2b-85ee-09bcd9e02833 nodeName:}" failed. No retries permitted until 2026-07-29 06:23:43.109766234 +0000 UTC m=+66.187398008 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-metrics-certs") pod "network-metrics-daemon-vcg74" (UID: "2ceff9c5-890d-4d2b-85ee-09bcd9e02833") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 06:23:11.109911 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:11.109787    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 06:23:11.109911 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:11.109805    2563 projected.go:196] Error preparing data for projected volume kube-api-access-qfp6c for pod openshift-network-diagnostics/network-check-target-njd8x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 06:23:11.109911 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:11.109874    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1be4cd1-a95a-4507-8159-0c80944c37c9-kube-api-access-qfp6c podName:d1be4cd1-a95a-4507-8159-0c80944c37c9 nodeName:}" failed. No retries permitted until 2026-07-29 06:23:43.109838959 +0000 UTC m=+66.187470706 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qfp6c" (UniqueName: "kubernetes.io/projected/d1be4cd1-a95a-4507-8159-0c80944c37c9-kube-api-access-qfp6c") pod "network-check-target-njd8x" (UID: "d1be4cd1-a95a-4507-8159-0c80944c37c9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 06:23:11.391941 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.391410    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:23:11.391941 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:11.391552    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcg74" podUID="2ceff9c5-890d-4d2b-85ee-09bcd9e02833"
Jul 29 06:23:11.725689 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.725610    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-223.ec2.internal" event="NodeReady"
Jul 29 06:23:11.725819 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.725757    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 29 06:23:11.756017 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.755988    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-mrklz"]
Jul 29 06:23:11.761592 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.761573    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-784bf54c5f-hj6jj"]
Jul 29 06:23:11.761709 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.761690    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-mrklz"
Jul 29 06:23:11.763959 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.763718    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 29 06:23:11.763959 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.763764    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 29 06:23:11.763959 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.763764    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9hq6v\""
Jul 29 06:23:11.765331 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.765311    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:23:11.767231 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.767163    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 29 06:23:11.767467 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.767417    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-mrklz"]
Jul 29 06:23:11.767717 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.767695    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 29 06:23:11.768082 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.768064    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vtwx9\""
Jul 29 06:23:11.768214 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.768184    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 29 06:23:11.770109 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.770087    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4pk46"]
Jul 29 06:23:11.772578 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.772559    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 29 06:23:11.774033 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.774015    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4pk46"
Jul 29 06:23:11.775987 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.775970    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 29 06:23:11.776061 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.776045    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bd28d"]
Jul 29 06:23:11.777354 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.777339    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 29 06:23:11.777878 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.777845    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pmx92\""
Jul 29 06:23:11.777995 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.777925    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 29 06:23:11.779727 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.779641    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bd28d"
Jul 29 06:23:11.785232 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.782138    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 29 06:23:11.785232 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.782193    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-z5482\""
Jul 29 06:23:11.785232 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.782407    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 29 06:23:11.785232 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.782811    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4pk46"]
Jul 29 06:23:11.789681 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.789660    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-784bf54c5f-hj6jj"]
Jul 29 06:23:11.790638 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.790617    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bd28d"]
Jul 29 06:23:11.915607 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.915566    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-bound-sa-token\") pod \"image-registry-784bf54c5f-hj6jj\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") " pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:23:11.915607 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.915605    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9slsx\" (UniqueName: \"kubernetes.io/projected/4f4e0417-82ca-4f78-9043-85dc8af1d71b-kube-api-access-9slsx\") pod \"dns-default-bd28d\" (UID: \"4f4e0417-82ca-4f78-9043-85dc8af1d71b\") " pod="openshift-dns/dns-default-bd28d"
Jul 29 06:23:11.915880 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.915629    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9zxq\" (UniqueName: \"kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-kube-api-access-v9zxq\") pod \"image-registry-784bf54c5f-hj6jj\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") " pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:23:11.915880 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.915695    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/37763295-c6d9-4846-8891-f4c5b78ec095-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-mrklz\" (UID: \"37763295-c6d9-4846-8891-f4c5b78ec095\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-mrklz"
Jul 29 06:23:11.915880 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.915738    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/37763295-c6d9-4846-8891-f4c5b78ec095-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-mrklz\" (UID: \"37763295-c6d9-4846-8891-f4c5b78ec095\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-mrklz"
Jul 29 06:23:11.915880 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.915787    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpngg\" (UniqueName: \"kubernetes.io/projected/2f05bf26-3930-4e41-ad15-78055946eb87-kube-api-access-qpngg\") pod \"ingress-canary-4pk46\" (UID: \"2f05bf26-3930-4e41-ad15-78055946eb87\") " pod="openshift-ingress-canary/ingress-canary-4pk46"
Jul 29 06:23:11.915880 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.915839    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-ca-trust-extracted\") pod \"image-registry-784bf54c5f-hj6jj\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") " pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:23:11.916073 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.915910    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-certificates\") pod \"image-registry-784bf54c5f-hj6jj\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") " pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:23:11.916073 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.915926    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-installation-pull-secrets\") pod \"image-registry-784bf54c5f-hj6jj\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") " pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:23:11.916073 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.916003    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-trusted-ca\") pod \"image-registry-784bf54c5f-hj6jj\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") " pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:23:11.916073 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.916042    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f05bf26-3930-4e41-ad15-78055946eb87-cert\") pod \"ingress-canary-4pk46\" (UID: \"2f05bf26-3930-4e41-ad15-78055946eb87\") " pod="openshift-ingress-canary/ingress-canary-4pk46"
Jul 29 06:23:11.916073 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.916070    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f4e0417-82ca-4f78-9043-85dc8af1d71b-metrics-tls\") pod \"dns-default-bd28d\" (UID: \"4f4e0417-82ca-4f78-9043-85dc8af1d71b\") " pod="openshift-dns/dns-default-bd28d"
Jul 29 06:23:11.916240 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.916093    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4f4e0417-82ca-4f78-9043-85dc8af1d71b-config-volume\") pod \"dns-default-bd28d\" (UID: \"4f4e0417-82ca-4f78-9043-85dc8af1d71b\") " pod="openshift-dns/dns-default-bd28d"
Jul 29 06:23:11.916240 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.916108    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f4e0417-82ca-4f78-9043-85dc8af1d71b-tmp-dir\") pod \"dns-default-bd28d\" (UID: \"4f4e0417-82ca-4f78-9043-85dc8af1d71b\") " pod="openshift-dns/dns-default-bd28d"
Jul 29 06:23:11.916240 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.916136    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-image-registry-private-configuration\") pod \"image-registry-784bf54c5f-hj6jj\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") " pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:23:11.916240 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:11.916156    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-tls\") pod \"image-registry-784bf54c5f-hj6jj\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") " pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:23:12.017105 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.017029    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-certificates\") pod \"image-registry-784bf54c5f-hj6jj\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") " pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:23:12.017105 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.017061    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-installation-pull-secrets\") pod \"image-registry-784bf54c5f-hj6jj\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") " pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:23:12.017105 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.017083    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-trusted-ca\") pod \"image-registry-784bf54c5f-hj6jj\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") " pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:23:12.017105 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.017103    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f05bf26-3930-4e41-ad15-78055946eb87-cert\") pod \"ingress-canary-4pk46\" (UID: \"2f05bf26-3930-4e41-ad15-78055946eb87\") " pod="openshift-ingress-canary/ingress-canary-4pk46"
Jul 29 06:23:12.017437 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.017125    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f4e0417-82ca-4f78-9043-85dc8af1d71b-metrics-tls\") pod \"dns-default-bd28d\" (UID: \"4f4e0417-82ca-4f78-9043-85dc8af1d71b\") " pod="openshift-dns/dns-default-bd28d"
Jul 29 06:23:12.017437 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.017149    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4f4e0417-82ca-4f78-9043-85dc8af1d71b-config-volume\") pod \"dns-default-bd28d\" (UID: \"4f4e0417-82ca-4f78-9043-85dc8af1d71b\") " pod="openshift-dns/dns-default-bd28d"
Jul 29 06:23:12.017437 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.017170    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f4e0417-82ca-4f78-9043-85dc8af1d71b-tmp-dir\") pod \"dns-default-bd28d\" (UID: \"4f4e0417-82ca-4f78-9043-85dc8af1d71b\") " pod="openshift-dns/dns-default-bd28d"
Jul 29 06:23:12.017437 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.017193    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-image-registry-private-configuration\") pod \"image-registry-784bf54c5f-hj6jj\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") " pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:23:12.017437 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.017216    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-tls\") pod \"image-registry-784bf54c5f-hj6jj\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") " pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:23:12.017437 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.017241    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-bound-sa-token\") pod \"image-registry-784bf54c5f-hj6jj\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") " pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:23:12.017437 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.017257    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9slsx\" (UniqueName: \"kubernetes.io/projected/4f4e0417-82ca-4f78-9043-85dc8af1d71b-kube-api-access-9slsx\") pod \"dns-default-bd28d\" (UID: \"4f4e0417-82ca-4f78-9043-85dc8af1d71b\") " pod="openshift-dns/dns-default-bd28d"
Jul 29 06:23:12.017437 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:12.017274    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 06:23:12.017437 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.017283    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9zxq\" (UniqueName: \"kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-kube-api-access-v9zxq\") pod \"image-registry-784bf54c5f-hj6jj\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") " pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:23:12.017437 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.017307    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/37763295-c6d9-4846-8891-f4c5b78ec095-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-mrklz\" (UID: \"37763295-c6d9-4846-8891-f4c5b78ec095\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-mrklz"
Jul 29 06:23:12.017437 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.017329    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/37763295-c6d9-4846-8891-f4c5b78ec095-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-mrklz\" (UID: \"37763295-c6d9-4846-8891-f4c5b78ec095\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-mrklz"
Jul 29 06:23:12.017437 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:12.017353    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f4e0417-82ca-4f78-9043-85dc8af1d71b-metrics-tls podName:4f4e0417-82ca-4f78-9043-85dc8af1d71b nodeName:}" failed. No retries permitted until 2026-07-29 06:23:12.517330053 +0000 UTC m=+35.594961814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f4e0417-82ca-4f78-9043-85dc8af1d71b-metrics-tls") pod "dns-default-bd28d" (UID: "4f4e0417-82ca-4f78-9043-85dc8af1d71b") : secret "dns-default-metrics-tls" not found
Jul 29 06:23:12.017437 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:12.017399    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 06:23:12.017437 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.017411    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpngg\" (UniqueName: \"kubernetes.io/projected/2f05bf26-3930-4e41-ad15-78055946eb87-kube-api-access-qpngg\") pod \"ingress-canary-4pk46\" (UID: \"2f05bf26-3930-4e41-ad15-78055946eb87\") " pod="openshift-ingress-canary/ingress-canary-4pk46"
Jul 29 06:23:12.018122 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.017463    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-ca-trust-extracted\") pod \"image-registry-784bf54c5f-hj6jj\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") " pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:23:12.018122 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.017733    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-certificates\") pod \"image-registry-784bf54c5f-hj6jj\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") " pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:23:12.018122 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.017796    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-ca-trust-extracted\") pod \"image-registry-784bf54c5f-hj6jj\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") " pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:23:12.018122 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:12.017875    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 06:23:12.018122 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:12.017894    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-784bf54c5f-hj6jj: secret "image-registry-tls" not found
Jul 29 06:23:12.018122 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:12.017955    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-tls podName:e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48 nodeName:}" failed. No retries permitted until 2026-07-29 06:23:12.517938099 +0000 UTC m=+35.595569862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-tls") pod "image-registry-784bf54c5f-hj6jj" (UID: "e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48") : secret "image-registry-tls" not found
Jul 29 06:23:12.018122 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.018001    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4f4e0417-82ca-4f78-9043-85dc8af1d71b-config-volume\") pod \"dns-default-bd28d\" (UID: \"4f4e0417-82ca-4f78-9043-85dc8af1d71b\") " pod="openshift-dns/dns-default-bd28d"
Jul 29 06:23:12.018122 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.018041    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-trusted-ca\") pod \"image-registry-784bf54c5f-hj6jj\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") " pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:23:12.018122 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:12.018096    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37763295-c6d9-4846-8891-f4c5b78ec095-networking-console-plugin-cert podName:37763295-c6d9-4846-8891-f4c5b78ec095 nodeName:}" failed. No retries permitted until 2026-07-29 06:23:12.518077513 +0000 UTC m=+35.595709265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/37763295-c6d9-4846-8891-f4c5b78ec095-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-mrklz" (UID: "37763295-c6d9-4846-8891-f4c5b78ec095") : secret "networking-console-plugin-cert" not found
Jul 29 06:23:12.018625 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.018398    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f4e0417-82ca-4f78-9043-85dc8af1d71b-tmp-dir\") pod \"dns-default-bd28d\" (UID: \"4f4e0417-82ca-4f78-9043-85dc8af1d71b\") " pod="openshift-dns/dns-default-bd28d"
Jul 29 06:23:12.018625 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.018549    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/37763295-c6d9-4846-8891-f4c5b78ec095-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-mrklz\" (UID: \"37763295-c6d9-4846-8891-f4c5b78ec095\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-mrklz"
Jul 29 06:23:12.021924 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.021903    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-installation-pull-secrets\") pod \"image-registry-784bf54c5f-hj6jj\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") " pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:23:12.021924 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.021920    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f05bf26-3930-4e41-ad15-78055946eb87-cert\") pod \"ingress-canary-4pk46\" (UID: \"2f05bf26-3930-4e41-ad15-78055946eb87\") " pod="openshift-ingress-canary/ingress-canary-4pk46"
Jul 29 06:23:12.022091 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.021934    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-image-registry-private-configuration\") pod \"image-registry-784bf54c5f-hj6jj\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") " pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:23:12.026259 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.026240    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpngg\" (UniqueName: \"kubernetes.io/projected/2f05bf26-3930-4e41-ad15-78055946eb87-kube-api-access-qpngg\") pod \"ingress-canary-4pk46\" (UID: \"2f05bf26-3930-4e41-ad15-78055946eb87\") " pod="openshift-ingress-canary/ingress-canary-4pk46"
Jul 29 06:23:12.026417 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.026393    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-bound-sa-token\") pod \"image-registry-784bf54c5f-hj6jj\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") " pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:23:12.026499 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.026413    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9slsx\" (UniqueName: \"kubernetes.io/projected/4f4e0417-82ca-4f78-9043-85dc8af1d71b-kube-api-access-9slsx\") pod \"dns-default-bd28d\" (UID: \"4f4e0417-82ca-4f78-9043-85dc8af1d71b\") " pod="openshift-dns/dns-default-bd28d"
Jul 29 06:23:12.027118 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.027100    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9zxq\" (UniqueName: \"kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-kube-api-access-v9zxq\") pod \"image-registry-784bf54c5f-hj6jj\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") " pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:23:12.096010 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.095972    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4pk46"
Jul 29 06:23:12.391664 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.391632    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzdr7"
Jul 29 06:23:12.391836 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.391694    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njd8x"
Jul 29 06:23:12.394137 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.394114    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 06:23:12.394700 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.394385    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4kn4r\""
Jul 29 06:23:12.394700 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.394432    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 06:23:12.394700 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.394429    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 06:23:12.521520 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.521486    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f4e0417-82ca-4f78-9043-85dc8af1d71b-metrics-tls\") pod \"dns-default-bd28d\" (UID: \"4f4e0417-82ca-4f78-9043-85dc8af1d71b\") " pod="openshift-dns/dns-default-bd28d"
Jul 29 06:23:12.521645 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.521539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-tls\") pod \"image-registry-784bf54c5f-hj6jj\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") " pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:23:12.521645 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.521587    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/37763295-c6d9-4846-8891-f4c5b78ec095-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-mrklz\" (UID: \"37763295-c6d9-4846-8891-f4c5b78ec095\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-mrklz"
Jul 29 06:23:12.521804 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:12.521786    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 06:23:12.521895 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:12.521884    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37763295-c6d9-4846-8891-f4c5b78ec095-networking-console-plugin-cert podName:37763295-c6d9-4846-8891-f4c5b78ec095 nodeName:}" failed. No retries permitted until 2026-07-29 06:23:13.52186431 +0000 UTC m=+36.599496071 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/37763295-c6d9-4846-8891-f4c5b78ec095-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-mrklz" (UID: "37763295-c6d9-4846-8891-f4c5b78ec095") : secret "networking-console-plugin-cert" not found
Jul 29 06:23:12.522318 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:12.522297    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 06:23:12.522404 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:12.522350    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f4e0417-82ca-4f78-9043-85dc8af1d71b-metrics-tls podName:4f4e0417-82ca-4f78-9043-85dc8af1d71b nodeName:}" failed. No retries permitted until 2026-07-29 06:23:13.522335388 +0000 UTC m=+36.599967150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f4e0417-82ca-4f78-9043-85dc8af1d71b-metrics-tls") pod "dns-default-bd28d" (UID: "4f4e0417-82ca-4f78-9043-85dc8af1d71b") : secret "dns-default-metrics-tls" not found
Jul 29 06:23:12.522461 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:12.522417    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 06:23:12.522461 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:12.522428    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-784bf54c5f-hj6jj: secret "image-registry-tls" not found
Jul 29 06:23:12.522461 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:12.522459    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-tls podName:e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48 nodeName:}" failed. No retries permitted until 2026-07-29 06:23:13.522449953 +0000 UTC m=+36.600081712 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-tls") pod "image-registry-784bf54c5f-hj6jj" (UID: "e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48") : secret "image-registry-tls" not found
Jul 29 06:23:12.561494 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:12.561379    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4pk46"]
Jul 29 06:23:12.564463 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:23:12.564438    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f05bf26_3930_4e41_ad15_78055946eb87.slice/crio-e05027eba7d95bd4957c09570cefc1763474f18c886ff69ece7c95897e9d427a WatchSource:0}: Error finding container e05027eba7d95bd4957c09570cefc1763474f18c886ff69ece7c95897e9d427a: Status 404 returned error can't find the container with id e05027eba7d95bd4957c09570cefc1763474f18c886ff69ece7c95897e9d427a
Jul 29 06:23:13.391804 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:13.391779    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:23:13.393880 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:13.393832    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 06:23:13.393995 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:13.393907    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-btmpr\""
Jul 29 06:23:13.531533 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:13.531501    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f4e0417-82ca-4f78-9043-85dc8af1d71b-metrics-tls\") pod \"dns-default-bd28d\" (UID: \"4f4e0417-82ca-4f78-9043-85dc8af1d71b\") " pod="openshift-dns/dns-default-bd28d"
Jul 29 06:23:13.531533 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:13.531541    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-tls\") pod \"image-registry-784bf54c5f-hj6jj\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") " pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:23:13.532286 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:13.531592    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/37763295-c6d9-4846-8891-f4c5b78ec095-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-mrklz\" (UID: \"37763295-c6d9-4846-8891-f4c5b78ec095\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-mrklz"
Jul 29 06:23:13.532286 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:13.531687    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 06:23:13.532286 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:13.531687    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 06:23:13.532286 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:13.531690    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 06:23:13.532286 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:13.531744    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37763295-c6d9-4846-8891-f4c5b78ec095-networking-console-plugin-cert podName:37763295-c6d9-4846-8891-f4c5b78ec095 nodeName:}" failed. No retries permitted until 2026-07-29 06:23:15.531730368 +0000 UTC m=+38.609362126 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/37763295-c6d9-4846-8891-f4c5b78ec095-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-mrklz" (UID: "37763295-c6d9-4846-8891-f4c5b78ec095") : secret "networking-console-plugin-cert" not found
Jul 29 06:23:13.532286 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:13.531749    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-784bf54c5f-hj6jj: secret "image-registry-tls" not found
Jul 29 06:23:13.532286 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:13.531757    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f4e0417-82ca-4f78-9043-85dc8af1d71b-metrics-tls podName:4f4e0417-82ca-4f78-9043-85dc8af1d71b nodeName:}" failed. No retries permitted until 2026-07-29 06:23:15.531751882 +0000 UTC m=+38.609383627 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f4e0417-82ca-4f78-9043-85dc8af1d71b-metrics-tls") pod "dns-default-bd28d" (UID: "4f4e0417-82ca-4f78-9043-85dc8af1d71b") : secret "dns-default-metrics-tls" not found
Jul 29 06:23:13.532286 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:13.531789    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-tls podName:e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48 nodeName:}" failed. No retries permitted until 2026-07-29 06:23:15.531772649 +0000 UTC m=+38.609404408 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-tls") pod "image-registry-784bf54c5f-hj6jj" (UID: "e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48") : secret "image-registry-tls" not found
Jul 29 06:23:13.545164 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:13.545136    2563 generic.go:356] "Generic (PLEG): container finished" podID="1673ef43-e0e0-4677-b830-7e8243686336" containerID="f3817f4bbf9d51412a93ff7cdd4804a110a650895ae645e209b4a26cc4bd5ec0" exitCode=0
Jul 29 06:23:13.545282 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:13.545210    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cr75t" event={"ID":"1673ef43-e0e0-4677-b830-7e8243686336","Type":"ContainerDied","Data":"f3817f4bbf9d51412a93ff7cdd4804a110a650895ae645e209b4a26cc4bd5ec0"}
Jul 29 06:23:13.546740 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:13.546713    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4pk46" event={"ID":"2f05bf26-3930-4e41-ad15-78055946eb87","Type":"ContainerStarted","Data":"e05027eba7d95bd4957c09570cefc1763474f18c886ff69ece7c95897e9d427a"}
Jul 29 06:23:14.553924 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:14.553882    2563 generic.go:356] "Generic (PLEG): container finished" podID="1673ef43-e0e0-4677-b830-7e8243686336" containerID="cdc3c8587325e48f9f9e2907b3055c23e714ef5009b74a4ea366afc78f083033" exitCode=0
Jul 29 06:23:14.553924 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:14.553936    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cr75t" event={"ID":"1673ef43-e0e0-4677-b830-7e8243686336","Type":"ContainerDied","Data":"cdc3c8587325e48f9f9e2907b3055c23e714ef5009b74a4ea366afc78f083033"}
Jul 29 06:23:15.550193 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:15.550104    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f4e0417-82ca-4f78-9043-85dc8af1d71b-metrics-tls\") pod \"dns-default-bd28d\" (UID: \"4f4e0417-82ca-4f78-9043-85dc8af1d71b\") " pod="openshift-dns/dns-default-bd28d"
Jul 29 06:23:15.550193 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:15.550144    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-tls\") pod \"image-registry-784bf54c5f-hj6jj\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") " pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:23:15.550193 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:15.550172    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/37763295-c6d9-4846-8891-f4c5b78ec095-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-mrklz\" (UID: \"37763295-c6d9-4846-8891-f4c5b78ec095\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-mrklz"
Jul 29 06:23:15.550405 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:15.550273    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 06:23:15.550405 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:15.550287    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 06:23:15.550405 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:15.550294    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-784bf54c5f-hj6jj: secret "image-registry-tls" not found
Jul 29 06:23:15.550405 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:15.550337    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-tls podName:e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48 nodeName:}" failed. No retries permitted until 2026-07-29 06:23:19.550322955 +0000 UTC m=+42.627954700 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-tls") pod "image-registry-784bf54c5f-hj6jj" (UID: "e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48") : secret "image-registry-tls" not found
Jul 29 06:23:15.550405 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:15.550351    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37763295-c6d9-4846-8891-f4c5b78ec095-networking-console-plugin-cert podName:37763295-c6d9-4846-8891-f4c5b78ec095 nodeName:}" failed. No retries permitted until 2026-07-29 06:23:19.550344869 +0000 UTC m=+42.627976614 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/37763295-c6d9-4846-8891-f4c5b78ec095-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-mrklz" (UID: "37763295-c6d9-4846-8891-f4c5b78ec095") : secret "networking-console-plugin-cert" not found
Jul 29 06:23:15.550405 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:15.550274    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 06:23:15.550405 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:15.550373    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f4e0417-82ca-4f78-9043-85dc8af1d71b-metrics-tls podName:4f4e0417-82ca-4f78-9043-85dc8af1d71b nodeName:}" failed. No retries permitted until 2026-07-29 06:23:19.550368375 +0000 UTC m=+42.628000120 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f4e0417-82ca-4f78-9043-85dc8af1d71b-metrics-tls") pod "dns-default-bd28d" (UID: "4f4e0417-82ca-4f78-9043-85dc8af1d71b") : secret "dns-default-metrics-tls" not found
Jul 29 06:23:15.559184 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:15.559163    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cr75t" event={"ID":"1673ef43-e0e0-4677-b830-7e8243686336","Type":"ContainerStarted","Data":"89c3bcce12ae8ddf62c0cc851e6f5be320b1a90e3f30cfd88492addf24453afe"}
Jul 29 06:23:15.560553 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:15.560534    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4pk46" event={"ID":"2f05bf26-3930-4e41-ad15-78055946eb87","Type":"ContainerStarted","Data":"a8ab5e7ec5bccfcd918cb60b2d98e51ea9c6d376074acffc585646b8a336f07a"}
Jul 29 06:23:15.578621 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:15.578573    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cr75t" podStartSLOduration=9.471514917 podStartE2EDuration="38.578556595s" totalImagesPullingTime="29.107041678s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 06:22:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:23:15.577148158 +0000 UTC m=+38.654779925" watchObservedRunningTime="2026-07-29 06:23:15.578556595 +0000 UTC m=+38.656188355"
Jul 29 06:23:15.590657 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:15.590623    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4pk46" podStartSLOduration=1.952700275 podStartE2EDuration="4.59061124s" totalImagesPullingTime="2.637910965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:23:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:23:15.590079264 +0000 UTC m=+38.667711032" watchObservedRunningTime="2026-07-29 06:23:15.59061124 +0000 UTC m=+38.668243007"
Jul 29 06:23:19.583569 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:19.583397    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f4e0417-82ca-4f78-9043-85dc8af1d71b-metrics-tls\") pod \"dns-default-bd28d\" (UID: \"4f4e0417-82ca-4f78-9043-85dc8af1d71b\") " pod="openshift-dns/dns-default-bd28d"
Jul 29 06:23:19.584052 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:19.583585    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-tls\") pod \"image-registry-784bf54c5f-hj6jj\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") " pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:23:19.584052 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:19.583612    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/37763295-c6d9-4846-8891-f4c5b78ec095-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-mrklz\" (UID: \"37763295-c6d9-4846-8891-f4c5b78ec095\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-mrklz"
Jul 29 06:23:19.584052 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:19.583536    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 06:23:19.584052 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:19.583679    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f4e0417-82ca-4f78-9043-85dc8af1d71b-metrics-tls podName:4f4e0417-82ca-4f78-9043-85dc8af1d71b nodeName:}" failed. No retries permitted until 2026-07-29 06:23:27.583663643 +0000 UTC m=+50.661295388 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f4e0417-82ca-4f78-9043-85dc8af1d71b-metrics-tls") pod "dns-default-bd28d" (UID: "4f4e0417-82ca-4f78-9043-85dc8af1d71b") : secret "dns-default-metrics-tls" not found
Jul 29 06:23:19.584052 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:19.583719    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 06:23:19.584052 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:19.583736    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 06:23:19.584052 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:19.583753    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-784bf54c5f-hj6jj: secret "image-registry-tls" not found
Jul 29 06:23:19.584052 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:19.583756    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37763295-c6d9-4846-8891-f4c5b78ec095-networking-console-plugin-cert podName:37763295-c6d9-4846-8891-f4c5b78ec095 nodeName:}" failed. No retries permitted until 2026-07-29 06:23:27.583745179 +0000 UTC m=+50.661376946 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/37763295-c6d9-4846-8891-f4c5b78ec095-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-mrklz" (UID: "37763295-c6d9-4846-8891-f4c5b78ec095") : secret "networking-console-plugin-cert" not found
Jul 29 06:23:19.584052 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:19.583813    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-tls podName:e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48 nodeName:}" failed. No retries permitted until 2026-07-29 06:23:27.583797483 +0000 UTC m=+50.661429233 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-tls") pod "image-registry-784bf54c5f-hj6jj" (UID: "e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48") : secret "image-registry-tls" not found
Jul 29 06:23:27.640552 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:27.640515    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/37763295-c6d9-4846-8891-f4c5b78ec095-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-mrklz\" (UID: \"37763295-c6d9-4846-8891-f4c5b78ec095\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-mrklz"
Jul 29 06:23:27.641006 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:27.640618    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f4e0417-82ca-4f78-9043-85dc8af1d71b-metrics-tls\") pod \"dns-default-bd28d\" (UID: \"4f4e0417-82ca-4f78-9043-85dc8af1d71b\") " pod="openshift-dns/dns-default-bd28d"
Jul 29 06:23:27.641006 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:27.640651    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-tls\") pod \"image-registry-784bf54c5f-hj6jj\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") " pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:23:27.641006 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:27.640661    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 06:23:27.641006 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:27.640730    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 06:23:27.641006 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:27.640758    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 06:23:27.641006 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:27.640771    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-784bf54c5f-hj6jj: secret "image-registry-tls" not found
Jul 29 06:23:27.641006 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:27.640733    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37763295-c6d9-4846-8891-f4c5b78ec095-networking-console-plugin-cert podName:37763295-c6d9-4846-8891-f4c5b78ec095 nodeName:}" failed. No retries permitted until 2026-07-29 06:23:43.640717293 +0000 UTC m=+66.718349074 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/37763295-c6d9-4846-8891-f4c5b78ec095-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-mrklz" (UID: "37763295-c6d9-4846-8891-f4c5b78ec095") : secret "networking-console-plugin-cert" not found
Jul 29 06:23:27.641006 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:27.640795    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f4e0417-82ca-4f78-9043-85dc8af1d71b-metrics-tls podName:4f4e0417-82ca-4f78-9043-85dc8af1d71b nodeName:}" failed. No retries permitted until 2026-07-29 06:23:43.64078198 +0000 UTC m=+66.718413726 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f4e0417-82ca-4f78-9043-85dc8af1d71b-metrics-tls") pod "dns-default-bd28d" (UID: "4f4e0417-82ca-4f78-9043-85dc8af1d71b") : secret "dns-default-metrics-tls" not found
Jul 29 06:23:27.641006 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:27.640811    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-tls podName:e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48 nodeName:}" failed. No retries permitted until 2026-07-29 06:23:43.640798501 +0000 UTC m=+66.718430249 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-tls") pod "image-registry-784bf54c5f-hj6jj" (UID: "e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48") : secret "image-registry-tls" not found
Jul 29 06:23:39.542870 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:39.542828    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ng5xk"
Jul 29 06:23:43.051788 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:43.051737    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0667dd8b-11f7-4b28-8f63-2f1bbbf98c56-original-pull-secret\") pod \"global-pull-secret-syncer-dzdr7\" (UID: \"0667dd8b-11f7-4b28-8f63-2f1bbbf98c56\") " pod="kube-system/global-pull-secret-syncer-dzdr7"
Jul 29 06:23:43.053832 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:43.053816    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 06:23:43.066084 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:43.066063    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0667dd8b-11f7-4b28-8f63-2f1bbbf98c56-original-pull-secret\") pod \"global-pull-secret-syncer-dzdr7\" (UID: \"0667dd8b-11f7-4b28-8f63-2f1bbbf98c56\") " pod="kube-system/global-pull-secret-syncer-dzdr7"
Jul 29 06:23:43.152808 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:43.152780    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfp6c\" (UniqueName: \"kubernetes.io/projected/d1be4cd1-a95a-4507-8159-0c80944c37c9-kube-api-access-qfp6c\") pod \"network-check-target-njd8x\" (UID: \"d1be4cd1-a95a-4507-8159-0c80944c37c9\") " pod="openshift-network-diagnostics/network-check-target-njd8x"
Jul 29 06:23:43.152956 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:43.152816    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-metrics-certs\") pod \"network-metrics-daemon-vcg74\" (UID: \"2ceff9c5-890d-4d2b-85ee-09bcd9e02833\") " pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:23:43.155308 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:43.155291    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 06:23:43.155461 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:43.155446    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 06:23:43.164039 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:43.164023    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 29 06:23:43.164089 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:43.164078    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-metrics-certs podName:2ceff9c5-890d-4d2b-85ee-09bcd9e02833 nodeName:}" failed. No retries permitted until 2026-07-29 06:24:47.164059566 +0000 UTC m=+130.241691325 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-metrics-certs") pod "network-metrics-daemon-vcg74" (UID: "2ceff9c5-890d-4d2b-85ee-09bcd9e02833") : secret "metrics-daemon-secret" not found
Jul 29 06:23:43.165432 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:43.165417    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 06:23:43.176748 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:43.176729    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfp6c\" (UniqueName: \"kubernetes.io/projected/d1be4cd1-a95a-4507-8159-0c80944c37c9-kube-api-access-qfp6c\") pod \"network-check-target-njd8x\" (UID: \"d1be4cd1-a95a-4507-8159-0c80944c37c9\") " pod="openshift-network-diagnostics/network-check-target-njd8x"
Jul 29 06:23:43.303497 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:43.303420    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzdr7"
Jul 29 06:23:43.311590 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:43.311570    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4kn4r\""
Jul 29 06:23:43.320003 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:43.319984    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njd8x"
Jul 29 06:23:43.438221 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:43.438194    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dzdr7"]
Jul 29 06:23:43.442212 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:23:43.442189    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0667dd8b_11f7_4b28_8f63_2f1bbbf98c56.slice/crio-0d365470a516a9403599f44241ece2941032ef9d279ee917d2829c2e23dda654 WatchSource:0}: Error finding container 0d365470a516a9403599f44241ece2941032ef9d279ee917d2829c2e23dda654: Status 404 returned error can't find the container with id 0d365470a516a9403599f44241ece2941032ef9d279ee917d2829c2e23dda654
Jul 29 06:23:43.448677 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:43.448655    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-njd8x"]
Jul 29 06:23:43.451259 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:23:43.451238    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1be4cd1_a95a_4507_8159_0c80944c37c9.slice/crio-41c8410d18846559af1410a6155f848b92a91062ee581bcce719112de20470ae WatchSource:0}: Error finding container 41c8410d18846559af1410a6155f848b92a91062ee581bcce719112de20470ae: Status 404 returned error can't find the container with id 41c8410d18846559af1410a6155f848b92a91062ee581bcce719112de20470ae
Jul 29 06:23:43.626467 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:43.626436    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-njd8x" event={"ID":"d1be4cd1-a95a-4507-8159-0c80944c37c9","Type":"ContainerStarted","Data":"41c8410d18846559af1410a6155f848b92a91062ee581bcce719112de20470ae"}
Jul 29 06:23:43.627485 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:43.627461    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dzdr7" event={"ID":"0667dd8b-11f7-4b28-8f63-2f1bbbf98c56","Type":"ContainerStarted","Data":"0d365470a516a9403599f44241ece2941032ef9d279ee917d2829c2e23dda654"}
Jul 29 06:23:43.657946 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:43.657924    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f4e0417-82ca-4f78-9043-85dc8af1d71b-metrics-tls\") pod \"dns-default-bd28d\" (UID: \"4f4e0417-82ca-4f78-9043-85dc8af1d71b\") " pod="openshift-dns/dns-default-bd28d"
Jul 29 06:23:43.658062 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:43.657952    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-tls\") pod \"image-registry-784bf54c5f-hj6jj\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") " pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:23:43.658062 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:43.657979    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/37763295-c6d9-4846-8891-f4c5b78ec095-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-mrklz\" (UID: \"37763295-c6d9-4846-8891-f4c5b78ec095\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-mrklz"
Jul 29 06:23:43.658177 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:43.658075    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 06:23:43.658177 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:43.658078    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 06:23:43.658177 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:43.658091    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 06:23:43.658177 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:43.658111    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-784bf54c5f-hj6jj: secret "image-registry-tls" not found
Jul 29 06:23:43.658177 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:43.658124    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37763295-c6d9-4846-8891-f4c5b78ec095-networking-console-plugin-cert podName:37763295-c6d9-4846-8891-f4c5b78ec095 nodeName:}" failed. No retries permitted until 2026-07-29 06:24:15.658109247 +0000 UTC m=+98.735740992 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/37763295-c6d9-4846-8891-f4c5b78ec095-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-mrklz" (UID: "37763295-c6d9-4846-8891-f4c5b78ec095") : secret "networking-console-plugin-cert" not found
Jul 29 06:23:43.658177 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:43.658136    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f4e0417-82ca-4f78-9043-85dc8af1d71b-metrics-tls podName:4f4e0417-82ca-4f78-9043-85dc8af1d71b nodeName:}" failed. No retries permitted until 2026-07-29 06:24:15.658130144 +0000 UTC m=+98.735761889 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f4e0417-82ca-4f78-9043-85dc8af1d71b-metrics-tls") pod "dns-default-bd28d" (UID: "4f4e0417-82ca-4f78-9043-85dc8af1d71b") : secret "dns-default-metrics-tls" not found
Jul 29 06:23:43.658177 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:23:43.658158    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-tls podName:e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48 nodeName:}" failed. No retries permitted until 2026-07-29 06:24:15.658141341 +0000 UTC m=+98.735773090 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-tls") pod "image-registry-784bf54c5f-hj6jj" (UID: "e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48") : secret "image-registry-tls" not found
Jul 29 06:23:48.642646 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:48.642604    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-njd8x" event={"ID":"d1be4cd1-a95a-4507-8159-0c80944c37c9","Type":"ContainerStarted","Data":"aa12da7219a372f0763b98adfe8905ac82b07c683170ee87c393554dd1ea40bc"}
Jul 29 06:23:48.643135 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:48.642810    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-njd8x"
Jul 29 06:23:48.644104 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:48.644083    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dzdr7" event={"ID":"0667dd8b-11f7-4b28-8f63-2f1bbbf98c56","Type":"ContainerStarted","Data":"09fb8793740d4e5f3c397893d49dd4b9e74d2286164d4fc41ef6a38fa50533cd"}
Jul 29 06:23:48.655337 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:48.655291    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-njd8x" podStartSLOduration=67.18938266 podStartE2EDuration="1m11.655278389s" totalImagesPullingTime="4.465895729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:23:48.655232941 +0000 UTC m=+71.732864708" watchObservedRunningTime="2026-07-29 06:23:48.655278389 +0000 UTC m=+71.732910156"
Jul 29 06:23:48.668722 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:23:48.668266    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dzdr7" podStartSLOduration=67.190265442 podStartE2EDuration="1m11.66824843s" totalImagesPullingTime="4.477982988s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 06:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:23:48.667151194 +0000 UTC m=+71.744782962" watchObservedRunningTime="2026-07-29 06:23:48.66824843 +0000 UTC m=+71.745880198"
Jul 29 06:24:15.685746 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:24:15.685633    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f4e0417-82ca-4f78-9043-85dc8af1d71b-metrics-tls\") pod \"dns-default-bd28d\" (UID: \"4f4e0417-82ca-4f78-9043-85dc8af1d71b\") " pod="openshift-dns/dns-default-bd28d"
Jul 29 06:24:15.685746 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:24:15.685673    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-tls\") pod \"image-registry-784bf54c5f-hj6jj\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") " pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:24:15.685746 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:24:15.685700    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/37763295-c6d9-4846-8891-f4c5b78ec095-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-mrklz\" (UID: \"37763295-c6d9-4846-8891-f4c5b78ec095\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-mrklz"
Jul 29 06:24:15.686263 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:24:15.685794    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 06:24:15.686263 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:24:15.685794    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 06:24:15.686263 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:24:15.685799    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 06:24:15.686263 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:24:15.685876    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-784bf54c5f-hj6jj: secret "image-registry-tls" not found
Jul 29 06:24:15.686263 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:24:15.685878    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37763295-c6d9-4846-8891-f4c5b78ec095-networking-console-plugin-cert podName:37763295-c6d9-4846-8891-f4c5b78ec095 nodeName:}" failed. No retries permitted until 2026-07-29 06:25:19.685847049 +0000 UTC m=+162.763478794 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/37763295-c6d9-4846-8891-f4c5b78ec095-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-mrklz" (UID: "37763295-c6d9-4846-8891-f4c5b78ec095") : secret "networking-console-plugin-cert" not found
Jul 29 06:24:15.686263 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:24:15.685918    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f4e0417-82ca-4f78-9043-85dc8af1d71b-metrics-tls podName:4f4e0417-82ca-4f78-9043-85dc8af1d71b nodeName:}" failed. No retries permitted until 2026-07-29 06:25:19.68590316 +0000 UTC m=+162.763534904 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f4e0417-82ca-4f78-9043-85dc8af1d71b-metrics-tls") pod "dns-default-bd28d" (UID: "4f4e0417-82ca-4f78-9043-85dc8af1d71b") : secret "dns-default-metrics-tls" not found
Jul 29 06:24:15.686263 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:24:15.685928    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-tls podName:e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48 nodeName:}" failed. No retries permitted until 2026-07-29 06:25:19.68592212 +0000 UTC m=+162.763553865 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-tls") pod "image-registry-784bf54c5f-hj6jj" (UID: "e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48") : secret "image-registry-tls" not found
Jul 29 06:24:19.648805 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:24:19.648774    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-njd8x"
Jul 29 06:24:47.206566 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:24:47.206526    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-metrics-certs\") pod \"network-metrics-daemon-vcg74\" (UID: \"2ceff9c5-890d-4d2b-85ee-09bcd9e02833\") " pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:24:47.207054 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:24:47.206676    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 29 06:24:47.207054 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:24:47.206745    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-metrics-certs podName:2ceff9c5-890d-4d2b-85ee-09bcd9e02833 nodeName:}" failed. No retries permitted until 2026-07-29 06:26:49.206727271 +0000 UTC m=+252.284359038 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-metrics-certs") pod "network-metrics-daemon-vcg74" (UID: "2ceff9c5-890d-4d2b-85ee-09bcd9e02833") : secret "metrics-daemon-secret" not found
Jul 29 06:25:14.775606 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:25:14.775567    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-74c9f884b7-mrklz" podUID="37763295-c6d9-4846-8891-f4c5b78ec095"
Jul 29 06:25:14.787736 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:25:14.787694    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj" podUID="e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48"
Jul 29 06:25:14.800907 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:25:14.800885    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-bd28d" podUID="4f4e0417-82ca-4f78-9043-85dc8af1d71b"
Jul 29 06:25:14.845722 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:25:14.845703    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:25:14.845882 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:25:14.845723    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bd28d"
Jul 29 06:25:14.845882 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:25:14.845703    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-mrklz"
Jul 29 06:25:16.402296 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:25:16.402234    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-vcg74" podUID="2ceff9c5-890d-4d2b-85ee-09bcd9e02833"
Jul 29 06:25:19.729106 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:25:19.729076    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f4e0417-82ca-4f78-9043-85dc8af1d71b-metrics-tls\") pod \"dns-default-bd28d\" (UID: \"4f4e0417-82ca-4f78-9043-85dc8af1d71b\") " pod="openshift-dns/dns-default-bd28d"
Jul 29 06:25:19.729508 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:25:19.729117    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-tls\") pod \"image-registry-784bf54c5f-hj6jj\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") " pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:25:19.729508 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:25:19.729154    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/37763295-c6d9-4846-8891-f4c5b78ec095-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-mrklz\" (UID: \"37763295-c6d9-4846-8891-f4c5b78ec095\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-mrklz"
Jul 29 06:25:19.729508 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:25:19.729225    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 06:25:19.729508 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:25:19.729268    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 06:25:19.729508 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:25:19.729271    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 06:25:19.729508 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:25:19.729287    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-784bf54c5f-hj6jj: secret "image-registry-tls" not found
Jul 29 06:25:19.729508 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:25:19.729301    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f4e0417-82ca-4f78-9043-85dc8af1d71b-metrics-tls podName:4f4e0417-82ca-4f78-9043-85dc8af1d71b nodeName:}" failed. No retries permitted until 2026-07-29 06:27:21.729284721 +0000 UTC m=+284.806916466 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f4e0417-82ca-4f78-9043-85dc8af1d71b-metrics-tls") pod "dns-default-bd28d" (UID: "4f4e0417-82ca-4f78-9043-85dc8af1d71b") : secret "dns-default-metrics-tls" not found
Jul 29 06:25:19.729508 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:25:19.729324    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37763295-c6d9-4846-8891-f4c5b78ec095-networking-console-plugin-cert podName:37763295-c6d9-4846-8891-f4c5b78ec095 nodeName:}" failed. No retries permitted until 2026-07-29 06:27:21.729307563 +0000 UTC m=+284.806939331 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/37763295-c6d9-4846-8891-f4c5b78ec095-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-mrklz" (UID: "37763295-c6d9-4846-8891-f4c5b78ec095") : secret "networking-console-plugin-cert" not found
Jul 29 06:25:19.729508 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:25:19.729342    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-tls podName:e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48 nodeName:}" failed. No retries permitted until 2026-07-29 06:27:21.729332378 +0000 UTC m=+284.806964128 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-tls") pod "image-registry-784bf54c5f-hj6jj" (UID: "e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48") : secret "image-registry-tls" not found
Jul 29 06:25:28.391415 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:25:28.391326    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:26:38.095383 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.095349    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-87wjh"]
Jul 29 06:26:38.097145 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.097125    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-npz5n"]
Jul 29 06:26:38.097270 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.097244    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-87wjh"
Jul 29 06:26:38.098700 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.098676    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-mr497"]
Jul 29 06:26:38.098700 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.098697    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-npz5n"
Jul 29 06:26:38.099201 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.099163    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 29 06:26:38.099537 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.099521    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 29 06:26:38.099768 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.099753    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-7w454\""
Jul 29 06:26:38.100578 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.100560    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-mr497"
Jul 29 06:26:38.100671 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.100650    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 29 06:26:38.100739 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.100693    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 29 06:26:38.100999 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.100985    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 29 06:26:38.100999 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.100993    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 29 06:26:38.101349 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.101330    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-x7hwg\""
Jul 29 06:26:38.101349 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.101345    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 29 06:26:38.101447 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.101414    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 29 06:26:38.102506 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.102488    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-zjqhp\""
Jul 29 06:26:38.102605 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.102492    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 29 06:26:38.103085 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.103070    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 29 06:26:38.103300 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.103284    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 29 06:26:38.103435 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.103386    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 29 06:26:38.109419 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.109402    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 29 06:26:38.110526 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.110501    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-87wjh"]
Jul 29 06:26:38.116218 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.116198    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-npz5n"]
Jul 29 06:26:38.126867 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.126830    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-mr497"]
Jul 29 06:26:38.186833 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.186806    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-ghwrd"]
Jul 29 06:26:38.188629 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.188610    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-ghwrd"
Jul 29 06:26:38.190821 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.190805    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-cvtfh\""
Jul 29 06:26:38.200060 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.200033    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-ghwrd"]
Jul 29 06:26:38.253074 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.253045    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4-serving-cert\") pod \"insights-operator-5885fdb759-mr497\" (UID: \"cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4\") " pod="openshift-insights/insights-operator-5885fdb759-mr497"
Jul 29 06:26:38.253074 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.253078    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/526ca1b2-bbc3-41fc-bda0-6754c7e98e37-serving-cert\") pod \"service-ca-operator-5d6c75f84f-npz5n\" (UID: \"526ca1b2-bbc3-41fc-bda0-6754c7e98e37\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-npz5n"
Jul 29 06:26:38.253221 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.253094    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btgcq\" (UniqueName: \"kubernetes.io/projected/cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4-kube-api-access-btgcq\") pod \"insights-operator-5885fdb759-mr497\" (UID: \"cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4\") " pod="openshift-insights/insights-operator-5885fdb759-mr497"
Jul 29 06:26:38.253221 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.253126    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1786842b-1ae8-4c6b-9c4c-0d21298a4e15-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-87wjh\" (UID: \"1786842b-1ae8-4c6b-9c4c-0d21298a4e15\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-87wjh"
Jul 29 06:26:38.253221 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.253148    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/526ca1b2-bbc3-41fc-bda0-6754c7e98e37-config\") pod \"service-ca-operator-5d6c75f84f-npz5n\" (UID: \"526ca1b2-bbc3-41fc-bda0-6754c7e98e37\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-npz5n"
Jul 29 06:26:38.253316 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.253226    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1786842b-1ae8-4c6b-9c4c-0d21298a4e15-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-87wjh\" (UID: \"1786842b-1ae8-4c6b-9c4c-0d21298a4e15\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-87wjh"
Jul 29 06:26:38.253316 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.253253    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4-service-ca-bundle\") pod \"insights-operator-5885fdb759-mr497\" (UID: \"cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4\") " pod="openshift-insights/insights-operator-5885fdb759-mr497"
Jul 29 06:26:38.253316 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.253274    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-447z2\" (UniqueName: \"kubernetes.io/projected/1786842b-1ae8-4c6b-9c4c-0d21298a4e15-kube-api-access-447z2\") pod \"cluster-monitoring-operator-548b489cb8-87wjh\" (UID: \"1786842b-1ae8-4c6b-9c4c-0d21298a4e15\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-87wjh"
Jul 29 06:26:38.253410 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.253322    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhgp8\" (UniqueName: \"kubernetes.io/projected/526ca1b2-bbc3-41fc-bda0-6754c7e98e37-kube-api-access-mhgp8\") pod \"service-ca-operator-5d6c75f84f-npz5n\" (UID: \"526ca1b2-bbc3-41fc-bda0-6754c7e98e37\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-npz5n"
Jul 29 06:26:38.253410 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.253346    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4-snapshots\") pod \"insights-operator-5885fdb759-mr497\" (UID: \"cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4\") " pod="openshift-insights/insights-operator-5885fdb759-mr497"
Jul 29 06:26:38.253410 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.253377    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-mr497\" (UID: \"cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4\") " pod="openshift-insights/insights-operator-5885fdb759-mr497"
Jul 29 06:26:38.253410 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.253396    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4-tmp\") pod \"insights-operator-5885fdb759-mr497\" (UID: \"cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4\") " pod="openshift-insights/insights-operator-5885fdb759-mr497"
Jul 29 06:26:38.354565 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.354448    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-447z2\" (UniqueName: \"kubernetes.io/projected/1786842b-1ae8-4c6b-9c4c-0d21298a4e15-kube-api-access-447z2\") pod \"cluster-monitoring-operator-548b489cb8-87wjh\" (UID: \"1786842b-1ae8-4c6b-9c4c-0d21298a4e15\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-87wjh"
Jul 29 06:26:38.354565 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.354523    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhgp8\" (UniqueName: \"kubernetes.io/projected/526ca1b2-bbc3-41fc-bda0-6754c7e98e37-kube-api-access-mhgp8\") pod \"service-ca-operator-5d6c75f84f-npz5n\" (UID: \"526ca1b2-bbc3-41fc-bda0-6754c7e98e37\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-npz5n"
Jul 29 06:26:38.354565 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.354548    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4-snapshots\") pod \"insights-operator-5885fdb759-mr497\" (UID: \"cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4\") " pod="openshift-insights/insights-operator-5885fdb759-mr497"
Jul 29 06:26:38.354834 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.354576    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-mr497\" (UID: \"cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4\") " pod="openshift-insights/insights-operator-5885fdb759-mr497"
Jul 29 06:26:38.354834 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.354602    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4-tmp\") pod \"insights-operator-5885fdb759-mr497\" (UID: \"cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4\") " pod="openshift-insights/insights-operator-5885fdb759-mr497"
Jul 29 06:26:38.354834 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.354659    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4-serving-cert\") pod \"insights-operator-5885fdb759-mr497\" (UID: \"cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4\") " pod="openshift-insights/insights-operator-5885fdb759-mr497"
Jul 29 06:26:38.354834 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.354686    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29qws\" (UniqueName: \"kubernetes.io/projected/a49057f8-cbf7-433b-b2c9-3e45e014d35a-kube-api-access-29qws\") pod \"network-check-source-7d88cb55d8-ghwrd\" (UID: \"a49057f8-cbf7-433b-b2c9-3e45e014d35a\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-ghwrd"
Jul 29 06:26:38.354834 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.354718    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/526ca1b2-bbc3-41fc-bda0-6754c7e98e37-serving-cert\") pod \"service-ca-operator-5d6c75f84f-npz5n\" (UID: \"526ca1b2-bbc3-41fc-bda0-6754c7e98e37\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-npz5n"
Jul 29 06:26:38.354834 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.354742    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btgcq\" (UniqueName: \"kubernetes.io/projected/cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4-kube-api-access-btgcq\") pod \"insights-operator-5885fdb759-mr497\" (UID: \"cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4\") " pod="openshift-insights/insights-operator-5885fdb759-mr497"
Jul 29 06:26:38.354834 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.354771    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1786842b-1ae8-4c6b-9c4c-0d21298a4e15-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-87wjh\" (UID: \"1786842b-1ae8-4c6b-9c4c-0d21298a4e15\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-87wjh"
Jul 29 06:26:38.354834 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.354792    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/526ca1b2-bbc3-41fc-bda0-6754c7e98e37-config\") pod \"service-ca-operator-5d6c75f84f-npz5n\" (UID: \"526ca1b2-bbc3-41fc-bda0-6754c7e98e37\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-npz5n"
Jul 29 06:26:38.355286 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.354875    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1786842b-1ae8-4c6b-9c4c-0d21298a4e15-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-87wjh\" (UID: \"1786842b-1ae8-4c6b-9c4c-0d21298a4e15\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-87wjh"
Jul 29 06:26:38.355286 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.354904    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4-service-ca-bundle\") pod \"insights-operator-5885fdb759-mr497\" (UID: \"cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4\") " pod="openshift-insights/insights-operator-5885fdb759-mr497"
Jul 29 06:26:38.355286 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:38.355056    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 06:26:38.355286 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:38.355133    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1786842b-1ae8-4c6b-9c4c-0d21298a4e15-cluster-monitoring-operator-tls podName:1786842b-1ae8-4c6b-9c4c-0d21298a4e15 nodeName:}" failed. No retries permitted until 2026-07-29 06:26:38.855112684 +0000 UTC m=+241.932744430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1786842b-1ae8-4c6b-9c4c-0d21298a4e15-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-87wjh" (UID: "1786842b-1ae8-4c6b-9c4c-0d21298a4e15") : secret "cluster-monitoring-operator-tls" not found
Jul 29 06:26:38.355472 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.355301    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4-snapshots\") pod \"insights-operator-5885fdb759-mr497\" (UID: \"cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4\") " pod="openshift-insights/insights-operator-5885fdb759-mr497"
Jul 29 06:26:38.355472 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.355441    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4-service-ca-bundle\") pod \"insights-operator-5885fdb759-mr497\" (UID: \"cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4\") " pod="openshift-insights/insights-operator-5885fdb759-mr497"
Jul 29 06:26:38.355686 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.355658    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4-tmp\") pod \"insights-operator-5885fdb759-mr497\" (UID: \"cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4\") " pod="openshift-insights/insights-operator-5885fdb759-mr497"
Jul 29 06:26:38.355784 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.355761    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/526ca1b2-bbc3-41fc-bda0-6754c7e98e37-config\") pod \"service-ca-operator-5d6c75f84f-npz5n\" (UID: \"526ca1b2-bbc3-41fc-bda0-6754c7e98e37\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-npz5n"
Jul 29 06:26:38.355898 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.355874    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-mr497\" (UID: \"cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4\") " pod="openshift-insights/insights-operator-5885fdb759-mr497"
Jul 29 06:26:38.356008 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.355993    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1786842b-1ae8-4c6b-9c4c-0d21298a4e15-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-87wjh\" (UID: \"1786842b-1ae8-4c6b-9c4c-0d21298a4e15\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-87wjh"
Jul 29 06:26:38.357274 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.357248    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/526ca1b2-bbc3-41fc-bda0-6754c7e98e37-serving-cert\") pod \"service-ca-operator-5d6c75f84f-npz5n\" (UID: \"526ca1b2-bbc3-41fc-bda0-6754c7e98e37\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-npz5n"
Jul 29 06:26:38.357356 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.357341    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4-serving-cert\") pod \"insights-operator-5885fdb759-mr497\" (UID: \"cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4\") " pod="openshift-insights/insights-operator-5885fdb759-mr497"
Jul 29 06:26:38.362511 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.362479    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhgp8\" (UniqueName: \"kubernetes.io/projected/526ca1b2-bbc3-41fc-bda0-6754c7e98e37-kube-api-access-mhgp8\") pod \"service-ca-operator-5d6c75f84f-npz5n\" (UID: \"526ca1b2-bbc3-41fc-bda0-6754c7e98e37\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-npz5n"
Jul 29 06:26:38.363729 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.363705    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btgcq\" (UniqueName: \"kubernetes.io/projected/cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4-kube-api-access-btgcq\") pod \"insights-operator-5885fdb759-mr497\" (UID: \"cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4\") " pod="openshift-insights/insights-operator-5885fdb759-mr497"
Jul 29 06:26:38.363824 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.363708    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-447z2\" (UniqueName: \"kubernetes.io/projected/1786842b-1ae8-4c6b-9c4c-0d21298a4e15-kube-api-access-447z2\") pod \"cluster-monitoring-operator-548b489cb8-87wjh\" (UID: \"1786842b-1ae8-4c6b-9c4c-0d21298a4e15\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-87wjh"
Jul 29 06:26:38.415142 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.415106    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-npz5n"
Jul 29 06:26:38.421278 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.421259    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-mr497"
Jul 29 06:26:38.456052 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.456021    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29qws\" (UniqueName: \"kubernetes.io/projected/a49057f8-cbf7-433b-b2c9-3e45e014d35a-kube-api-access-29qws\") pod \"network-check-source-7d88cb55d8-ghwrd\" (UID: \"a49057f8-cbf7-433b-b2c9-3e45e014d35a\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-ghwrd"
Jul 29 06:26:38.463924 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.463875    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29qws\" (UniqueName: \"kubernetes.io/projected/a49057f8-cbf7-433b-b2c9-3e45e014d35a-kube-api-access-29qws\") pod \"network-check-source-7d88cb55d8-ghwrd\" (UID: \"a49057f8-cbf7-433b-b2c9-3e45e014d35a\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-ghwrd"
Jul 29 06:26:38.496923 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.496895    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-ghwrd"
Jul 29 06:26:38.545479 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.545443    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-npz5n"]
Jul 29 06:26:38.551133 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:26:38.551068    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod526ca1b2_bbc3_41fc_bda0_6754c7e98e37.slice/crio-0b4d9748f41cb4a3604e3e04aa6325a7756e8989c2480e655664871649323ede WatchSource:0}: Error finding container 0b4d9748f41cb4a3604e3e04aa6325a7756e8989c2480e655664871649323ede: Status 404 returned error can't find the container with id 0b4d9748f41cb4a3604e3e04aa6325a7756e8989c2480e655664871649323ede
Jul 29 06:26:38.561563 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.561538    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-mr497"]
Jul 29 06:26:38.565613 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:26:38.565591    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf59d0f1_c3e6_4d35_aa15_250b8ca1cfb4.slice/crio-7d9f091e2d613c73d41aae7daaa81896ee0b74d87fd2c099261e354bba08de01 WatchSource:0}: Error finding container 7d9f091e2d613c73d41aae7daaa81896ee0b74d87fd2c099261e354bba08de01: Status 404 returned error can't find the container with id 7d9f091e2d613c73d41aae7daaa81896ee0b74d87fd2c099261e354bba08de01
Jul 29 06:26:38.623029 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.622965    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-ghwrd"]
Jul 29 06:26:38.625871 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:26:38.625835    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda49057f8_cbf7_433b_b2c9_3e45e014d35a.slice/crio-7567aae02439da4540eb9fa58f126ac797209abc7fc3992e35ebe185cafb8904 WatchSource:0}: Error finding container 7567aae02439da4540eb9fa58f126ac797209abc7fc3992e35ebe185cafb8904: Status 404 returned error can't find the container with id 7567aae02439da4540eb9fa58f126ac797209abc7fc3992e35ebe185cafb8904
Jul 29 06:26:38.859949 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:38.859916    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1786842b-1ae8-4c6b-9c4c-0d21298a4e15-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-87wjh\" (UID: \"1786842b-1ae8-4c6b-9c4c-0d21298a4e15\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-87wjh"
Jul 29 06:26:38.860146 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:38.860058    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 06:26:38.860146 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:38.860127    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1786842b-1ae8-4c6b-9c4c-0d21298a4e15-cluster-monitoring-operator-tls podName:1786842b-1ae8-4c6b-9c4c-0d21298a4e15 nodeName:}" failed. No retries permitted until 2026-07-29 06:26:39.860109172 +0000 UTC m=+242.937740930 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1786842b-1ae8-4c6b-9c4c-0d21298a4e15-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-87wjh" (UID: "1786842b-1ae8-4c6b-9c4c-0d21298a4e15") : secret "cluster-monitoring-operator-tls" not found
Jul 29 06:26:39.041724 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:39.041643    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-mr497" event={"ID":"cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4","Type":"ContainerStarted","Data":"7d9f091e2d613c73d41aae7daaa81896ee0b74d87fd2c099261e354bba08de01"}
Jul 29 06:26:39.042734 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:39.042708    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-npz5n" event={"ID":"526ca1b2-bbc3-41fc-bda0-6754c7e98e37","Type":"ContainerStarted","Data":"0b4d9748f41cb4a3604e3e04aa6325a7756e8989c2480e655664871649323ede"}
Jul 29 06:26:39.044070 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:39.044039    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-ghwrd" event={"ID":"a49057f8-cbf7-433b-b2c9-3e45e014d35a","Type":"ContainerStarted","Data":"922fb6c0cd15bd19e6e3c9dea770227b111f65b44a30a2dde60d17cb76319d36"}
Jul 29 06:26:39.044170 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:39.044086    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-ghwrd" event={"ID":"a49057f8-cbf7-433b-b2c9-3e45e014d35a","Type":"ContainerStarted","Data":"7567aae02439da4540eb9fa58f126ac797209abc7fc3992e35ebe185cafb8904"}
Jul 29 06:26:39.057965 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:39.057918    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-ghwrd" podStartSLOduration=1.057901749 podStartE2EDuration="1.057901749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:26:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:26:39.057825913 +0000 UTC m=+242.135457692" watchObservedRunningTime="2026-07-29 06:26:39.057901749 +0000 UTC m=+242.135533518"
Jul 29 06:26:39.870926 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:39.870871    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1786842b-1ae8-4c6b-9c4c-0d21298a4e15-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-87wjh\" (UID: \"1786842b-1ae8-4c6b-9c4c-0d21298a4e15\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-87wjh"
Jul 29 06:26:39.871452 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:39.870977    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 06:26:39.871452 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:39.871055    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1786842b-1ae8-4c6b-9c4c-0d21298a4e15-cluster-monitoring-operator-tls podName:1786842b-1ae8-4c6b-9c4c-0d21298a4e15 nodeName:}" failed. No retries permitted until 2026-07-29 06:26:41.871034993 +0000 UTC m=+244.948666745 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1786842b-1ae8-4c6b-9c4c-0d21298a4e15-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-87wjh" (UID: "1786842b-1ae8-4c6b-9c4c-0d21298a4e15") : secret "cluster-monitoring-operator-tls" not found
Jul 29 06:26:41.888738 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:41.888695    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1786842b-1ae8-4c6b-9c4c-0d21298a4e15-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-87wjh\" (UID: \"1786842b-1ae8-4c6b-9c4c-0d21298a4e15\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-87wjh"
Jul 29 06:26:41.889198 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:41.888886    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 06:26:41.889198 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:41.888970    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1786842b-1ae8-4c6b-9c4c-0d21298a4e15-cluster-monitoring-operator-tls podName:1786842b-1ae8-4c6b-9c4c-0d21298a4e15 nodeName:}" failed. No retries permitted until 2026-07-29 06:26:45.88894749 +0000 UTC m=+248.966579235 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1786842b-1ae8-4c6b-9c4c-0d21298a4e15-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-87wjh" (UID: "1786842b-1ae8-4c6b-9c4c-0d21298a4e15") : secret "cluster-monitoring-operator-tls" not found
Jul 29 06:26:42.054215 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:42.054175    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-mr497" event={"ID":"cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4","Type":"ContainerStarted","Data":"17192d10ea2a0780ba298350a0f77aa35f084e074069678045f9751fa9ea2082"}
Jul 29 06:26:42.056050 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:42.056024    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-npz5n" event={"ID":"526ca1b2-bbc3-41fc-bda0-6754c7e98e37","Type":"ContainerStarted","Data":"f93b9ad8d7af0ed5935635a07313da490ca4a3390d994125f501a4f9e271fc82"}
Jul 29 06:26:42.069941 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:42.069902    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-mr497" podStartSLOduration=1.515835979 podStartE2EDuration="4.069888625s" totalImagesPullingTime="2.554052646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:26:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:26:42.069070198 +0000 UTC m=+245.146701964" watchObservedRunningTime="2026-07-29 06:26:42.069888625 +0000 UTC m=+245.147520406"
Jul 29 06:26:42.085685 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:42.085639    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-npz5n" podStartSLOduration=1.513975635 podStartE2EDuration="4.085620818s" totalImagesPullingTime="2.571645183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:26:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:26:42.084674642 +0000 UTC m=+245.162306420" watchObservedRunningTime="2026-07-29 06:26:42.085620818 +0000 UTC m=+245.163252590"
Jul 29 06:26:42.156522 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:42.156402    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vt5qk"]
Jul 29 06:26:42.159073 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:42.159046    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vt5qk"
Jul 29 06:26:42.161516 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:42.161485    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 29 06:26:42.161638 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:42.161598    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-75z2t\""
Jul 29 06:26:42.163884 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:42.162137    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 29 06:26:42.173293 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:42.173270    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vt5qk"]
Jul 29 06:26:42.291993 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:42.291966    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qxpl\" (UniqueName: \"kubernetes.io/projected/3fec1db3-1218-4a65-b5c1-1e7473869742-kube-api-access-5qxpl\") pod \"insights-runtime-extractor-vt5qk\" (UID: \"3fec1db3-1218-4a65-b5c1-1e7473869742\") " pod="openshift-insights/insights-runtime-extractor-vt5qk"
Jul 29 06:26:42.291993 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:42.291995    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fec1db3-1218-4a65-b5c1-1e7473869742-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vt5qk\" (UID: \"3fec1db3-1218-4a65-b5c1-1e7473869742\") " pod="openshift-insights/insights-runtime-extractor-vt5qk"
Jul 29 06:26:42.292205 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:42.292039    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fec1db3-1218-4a65-b5c1-1e7473869742-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vt5qk\" (UID: \"3fec1db3-1218-4a65-b5c1-1e7473869742\") " pod="openshift-insights/insights-runtime-extractor-vt5qk"
Jul 29 06:26:42.292205 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:42.292112    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3fec1db3-1218-4a65-b5c1-1e7473869742-crio-socket\") pod \"insights-runtime-extractor-vt5qk\" (UID: \"3fec1db3-1218-4a65-b5c1-1e7473869742\") " pod="openshift-insights/insights-runtime-extractor-vt5qk"
Jul 29 06:26:42.292205 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:42.292184    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3fec1db3-1218-4a65-b5c1-1e7473869742-data-volume\") pod \"insights-runtime-extractor-vt5qk\" (UID: \"3fec1db3-1218-4a65-b5c1-1e7473869742\") " pod="openshift-insights/insights-runtime-extractor-vt5qk"
Jul 29 06:26:42.392724 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:42.392694    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qxpl\" (UniqueName: \"kubernetes.io/projected/3fec1db3-1218-4a65-b5c1-1e7473869742-kube-api-access-5qxpl\") pod \"insights-runtime-extractor-vt5qk\" (UID: \"3fec1db3-1218-4a65-b5c1-1e7473869742\") " pod="openshift-insights/insights-runtime-extractor-vt5qk"
Jul 29 06:26:42.392724 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:42.392724    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fec1db3-1218-4a65-b5c1-1e7473869742-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vt5qk\" (UID: \"3fec1db3-1218-4a65-b5c1-1e7473869742\") " pod="openshift-insights/insights-runtime-extractor-vt5qk"
Jul 29 06:26:42.392945 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:42.392758    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fec1db3-1218-4a65-b5c1-1e7473869742-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vt5qk\" (UID: \"3fec1db3-1218-4a65-b5c1-1e7473869742\") " pod="openshift-insights/insights-runtime-extractor-vt5qk"
Jul 29 06:26:42.392945 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:42.392781    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3fec1db3-1218-4a65-b5c1-1e7473869742-crio-socket\") pod \"insights-runtime-extractor-vt5qk\" (UID: \"3fec1db3-1218-4a65-b5c1-1e7473869742\") " pod="openshift-insights/insights-runtime-extractor-vt5qk"
Jul 29 06:26:42.392945 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:42.392836    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3fec1db3-1218-4a65-b5c1-1e7473869742-crio-socket\") pod \"insights-runtime-extractor-vt5qk\" (UID: \"3fec1db3-1218-4a65-b5c1-1e7473869742\") " pod="openshift-insights/insights-runtime-extractor-vt5qk"
Jul 29 06:26:42.392945 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:42.392835    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 06:26:42.392945 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:42.392922    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3fec1db3-1218-4a65-b5c1-1e7473869742-data-volume\") pod \"insights-runtime-extractor-vt5qk\" (UID: \"3fec1db3-1218-4a65-b5c1-1e7473869742\") " pod="openshift-insights/insights-runtime-extractor-vt5qk"
Jul 29 06:26:42.393154 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:42.392973    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3fec1db3-1218-4a65-b5c1-1e7473869742-kube-rbac-proxy-cm podName:3fec1db3-1218-4a65-b5c1-1e7473869742 nodeName:}" failed. No retries permitted until 2026-07-29 06:26:42.892953718 +0000 UTC m=+245.970585463 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3fec1db3-1218-4a65-b5c1-1e7473869742-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vt5qk" (UID: "3fec1db3-1218-4a65-b5c1-1e7473869742") : configmap "kube-rbac-proxy" not found
Jul 29 06:26:42.393154 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:42.392876    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 06:26:42.393154 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:42.393042    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fec1db3-1218-4a65-b5c1-1e7473869742-insights-runtime-extractor-tls podName:3fec1db3-1218-4a65-b5c1-1e7473869742 nodeName:}" failed. No retries permitted until 2026-07-29 06:26:42.893026566 +0000 UTC m=+245.970658311 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3fec1db3-1218-4a65-b5c1-1e7473869742-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vt5qk" (UID: "3fec1db3-1218-4a65-b5c1-1e7473869742") : secret "insights-runtime-extractor-tls" not found
Jul 29 06:26:42.393255 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:42.393238    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3fec1db3-1218-4a65-b5c1-1e7473869742-data-volume\") pod \"insights-runtime-extractor-vt5qk\" (UID: \"3fec1db3-1218-4a65-b5c1-1e7473869742\") " pod="openshift-insights/insights-runtime-extractor-vt5qk"
Jul 29 06:26:42.400796 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:42.400768    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qxpl\" (UniqueName: \"kubernetes.io/projected/3fec1db3-1218-4a65-b5c1-1e7473869742-kube-api-access-5qxpl\") pod \"insights-runtime-extractor-vt5qk\" (UID: \"3fec1db3-1218-4a65-b5c1-1e7473869742\") " pod="openshift-insights/insights-runtime-extractor-vt5qk"
Jul 29 06:26:42.685107 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:42.685076    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-gqxcv"]
Jul 29 06:26:42.686996 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:42.686980    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gqxcv"
Jul 29 06:26:42.688763 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:42.688731    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 29 06:26:42.688868 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:42.688740    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-t96mr\""
Jul 29 06:26:42.688868 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:42.688827    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 29 06:26:42.695182 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:42.695159    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-gqxcv"]
Jul 29 06:26:42.795622 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:42.795585    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8trsr\" (UniqueName: \"kubernetes.io/projected/54586d7d-67b0-4934-bfb4-a9ee309e8624-kube-api-access-8trsr\") pod \"migrator-6865c9967-gqxcv\" (UID: \"54586d7d-67b0-4934-bfb4-a9ee309e8624\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gqxcv"
Jul 29 06:26:42.897049 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:42.897012    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fec1db3-1218-4a65-b5c1-1e7473869742-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vt5qk\" (UID: \"3fec1db3-1218-4a65-b5c1-1e7473869742\") " pod="openshift-insights/insights-runtime-extractor-vt5qk"
Jul 29 06:26:42.897486 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:42.897068    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fec1db3-1218-4a65-b5c1-1e7473869742-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vt5qk\" (UID: \"3fec1db3-1218-4a65-b5c1-1e7473869742\") " pod="openshift-insights/insights-runtime-extractor-vt5qk"
Jul 29 06:26:42.897486 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:42.897122    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8trsr\" (UniqueName: \"kubernetes.io/projected/54586d7d-67b0-4934-bfb4-a9ee309e8624-kube-api-access-8trsr\") pod \"migrator-6865c9967-gqxcv\" (UID: \"54586d7d-67b0-4934-bfb4-a9ee309e8624\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gqxcv"
Jul 29 06:26:42.897486 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:42.897132    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 06:26:42.897486 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:42.897201    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3fec1db3-1218-4a65-b5c1-1e7473869742-kube-rbac-proxy-cm podName:3fec1db3-1218-4a65-b5c1-1e7473869742 nodeName:}" failed. No retries permitted until 2026-07-29 06:26:43.897180683 +0000 UTC m=+246.974812431 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3fec1db3-1218-4a65-b5c1-1e7473869742-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vt5qk" (UID: "3fec1db3-1218-4a65-b5c1-1e7473869742") : configmap "kube-rbac-proxy" not found
Jul 29 06:26:42.897486 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:42.897245    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 06:26:42.897486 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:42.897308    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fec1db3-1218-4a65-b5c1-1e7473869742-insights-runtime-extractor-tls podName:3fec1db3-1218-4a65-b5c1-1e7473869742 nodeName:}" failed. No retries permitted until 2026-07-29 06:26:43.897290553 +0000 UTC m=+246.974922301 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3fec1db3-1218-4a65-b5c1-1e7473869742-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vt5qk" (UID: "3fec1db3-1218-4a65-b5c1-1e7473869742") : secret "insights-runtime-extractor-tls" not found
Jul 29 06:26:42.903943 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:42.903919    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8trsr\" (UniqueName: \"kubernetes.io/projected/54586d7d-67b0-4934-bfb4-a9ee309e8624-kube-api-access-8trsr\") pod \"migrator-6865c9967-gqxcv\" (UID: \"54586d7d-67b0-4934-bfb4-a9ee309e8624\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gqxcv"
Jul 29 06:26:42.996639 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:42.996562    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gqxcv"
Jul 29 06:26:43.116520 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:43.116431    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-gqxcv"]
Jul 29 06:26:43.119076 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:26:43.119049    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54586d7d_67b0_4934_bfb4_a9ee309e8624.slice/crio-b9a759deb8173d6e01b3b8ab016b8ac5643176207082af20ea0b535489db045e WatchSource:0}: Error finding container b9a759deb8173d6e01b3b8ab016b8ac5643176207082af20ea0b535489db045e: Status 404 returned error can't find the container with id b9a759deb8173d6e01b3b8ab016b8ac5643176207082af20ea0b535489db045e
Jul 29 06:26:43.905234 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:43.905199    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fec1db3-1218-4a65-b5c1-1e7473869742-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vt5qk\" (UID: \"3fec1db3-1218-4a65-b5c1-1e7473869742\") " pod="openshift-insights/insights-runtime-extractor-vt5qk"
Jul 29 06:26:43.905796 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:43.905262    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fec1db3-1218-4a65-b5c1-1e7473869742-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vt5qk\" (UID: \"3fec1db3-1218-4a65-b5c1-1e7473869742\") " pod="openshift-insights/insights-runtime-extractor-vt5qk"
Jul 29 06:26:43.905796 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:43.905342    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 06:26:43.905796 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:43.905392    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 06:26:43.905796 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:43.905518    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3fec1db3-1218-4a65-b5c1-1e7473869742-kube-rbac-proxy-cm podName:3fec1db3-1218-4a65-b5c1-1e7473869742 nodeName:}" failed. No retries permitted until 2026-07-29 06:26:45.905450226 +0000 UTC m=+248.983081975 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3fec1db3-1218-4a65-b5c1-1e7473869742-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vt5qk" (UID: "3fec1db3-1218-4a65-b5c1-1e7473869742") : configmap "kube-rbac-proxy" not found
Jul 29 06:26:43.905796 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:43.905541    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fec1db3-1218-4a65-b5c1-1e7473869742-insights-runtime-extractor-tls podName:3fec1db3-1218-4a65-b5c1-1e7473869742 nodeName:}" failed. No retries permitted until 2026-07-29 06:26:45.905531321 +0000 UTC m=+248.983163071 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3fec1db3-1218-4a65-b5c1-1e7473869742-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vt5qk" (UID: "3fec1db3-1218-4a65-b5c1-1e7473869742") : secret "insights-runtime-extractor-tls" not found
Jul 29 06:26:43.954651 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:43.954623    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rkphn_4a2a21ed-615a-4783-9a2a-b57804724f79/dns-node-resolver/0.log"
Jul 29 06:26:44.064320 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:44.064287    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gqxcv" event={"ID":"54586d7d-67b0-4934-bfb4-a9ee309e8624","Type":"ContainerStarted","Data":"b9a759deb8173d6e01b3b8ab016b8ac5643176207082af20ea0b535489db045e"}
Jul 29 06:26:45.069491 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:45.069455    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gqxcv" event={"ID":"54586d7d-67b0-4934-bfb4-a9ee309e8624","Type":"ContainerStarted","Data":"8d8414ffa17ce48ae0808b192a0c1eb2bac24aa9c1e0724bba5e381a4b80c078"}
Jul 29 06:26:45.069977 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:45.069510    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gqxcv" event={"ID":"54586d7d-67b0-4934-bfb4-a9ee309e8624","Type":"ContainerStarted","Data":"e758dddb762b2f6567818a52f80c553614c9f915f8943952428f83f6aefb672f"}
Jul 29 06:26:45.082484 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:45.082445    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gqxcv" podStartSLOduration=1.696601697 podStartE2EDuration="3.082429886s" totalImagesPullingTime="1.385828189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:26:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:26:45.081932881 +0000 UTC m=+248.159564652" watchObservedRunningTime="2026-07-29 06:26:45.082429886 +0000 UTC m=+248.160061704"
Jul 29 06:26:45.353671 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:45.353602    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qrnmf_2079ff1a-5b9e-439d-bd6c-02c73c344340/node-ca/0.log"
Jul 29 06:26:45.754960 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:45.754870    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4pk46_2f05bf26-3930-4e41-ad15-78055946eb87/serve-healthcheck-canary/0.log"
Jul 29 06:26:45.918216 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:45.918169    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1786842b-1ae8-4c6b-9c4c-0d21298a4e15-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-87wjh\" (UID: \"1786842b-1ae8-4c6b-9c4c-0d21298a4e15\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-87wjh"
Jul 29 06:26:45.918426 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:45.918258    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fec1db3-1218-4a65-b5c1-1e7473869742-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vt5qk\" (UID: \"3fec1db3-1218-4a65-b5c1-1e7473869742\") " pod="openshift-insights/insights-runtime-extractor-vt5qk"
Jul 29 06:26:45.918426 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:45.918295    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fec1db3-1218-4a65-b5c1-1e7473869742-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vt5qk\" (UID: \"3fec1db3-1218-4a65-b5c1-1e7473869742\") " pod="openshift-insights/insights-runtime-extractor-vt5qk"
Jul 29 06:26:45.918426 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:45.918338    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 06:26:45.918426 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:45.918370    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 06:26:45.918426 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:45.918371    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 06:26:45.918426 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:45.918409    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1786842b-1ae8-4c6b-9c4c-0d21298a4e15-cluster-monitoring-operator-tls podName:1786842b-1ae8-4c6b-9c4c-0d21298a4e15 nodeName:}" failed. No retries permitted until 2026-07-29 06:26:53.918391754 +0000 UTC m=+256.996023500 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1786842b-1ae8-4c6b-9c4c-0d21298a4e15-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-87wjh" (UID: "1786842b-1ae8-4c6b-9c4c-0d21298a4e15") : secret "cluster-monitoring-operator-tls" not found
Jul 29 06:26:45.918426 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:45.918430    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fec1db3-1218-4a65-b5c1-1e7473869742-insights-runtime-extractor-tls podName:3fec1db3-1218-4a65-b5c1-1e7473869742 nodeName:}" failed. No retries permitted until 2026-07-29 06:26:49.918418375 +0000 UTC m=+252.996050121 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3fec1db3-1218-4a65-b5c1-1e7473869742-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vt5qk" (UID: "3fec1db3-1218-4a65-b5c1-1e7473869742") : secret "insights-runtime-extractor-tls" not found
Jul 29 06:26:45.918770 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:45.918443    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3fec1db3-1218-4a65-b5c1-1e7473869742-kube-rbac-proxy-cm podName:3fec1db3-1218-4a65-b5c1-1e7473869742 nodeName:}" failed. No retries permitted until 2026-07-29 06:26:49.91843632 +0000 UTC m=+252.996068065 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3fec1db3-1218-4a65-b5c1-1e7473869742-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vt5qk" (UID: "3fec1db3-1218-4a65-b5c1-1e7473869742") : configmap "kube-rbac-proxy" not found
Jul 29 06:26:49.248561 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:49.248533    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-metrics-certs\") pod \"network-metrics-daemon-vcg74\" (UID: \"2ceff9c5-890d-4d2b-85ee-09bcd9e02833\") " pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:26:49.248941 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:49.248649    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 29 06:26:49.248941 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:49.248708    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-metrics-certs podName:2ceff9c5-890d-4d2b-85ee-09bcd9e02833 nodeName:}" failed. No retries permitted until 2026-07-29 06:28:51.248690418 +0000 UTC m=+374.326322165 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-metrics-certs") pod "network-metrics-daemon-vcg74" (UID: "2ceff9c5-890d-4d2b-85ee-09bcd9e02833") : secret "metrics-daemon-secret" not found
Jul 29 06:26:49.954183 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:49.954140    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fec1db3-1218-4a65-b5c1-1e7473869742-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vt5qk\" (UID: \"3fec1db3-1218-4a65-b5c1-1e7473869742\") " pod="openshift-insights/insights-runtime-extractor-vt5qk"
Jul 29 06:26:49.954373 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:49.954216    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fec1db3-1218-4a65-b5c1-1e7473869742-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vt5qk\" (UID: \"3fec1db3-1218-4a65-b5c1-1e7473869742\") " pod="openshift-insights/insights-runtime-extractor-vt5qk"
Jul 29 06:26:49.954373 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:49.954258    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 06:26:49.954373 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:49.954335    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3fec1db3-1218-4a65-b5c1-1e7473869742-kube-rbac-proxy-cm podName:3fec1db3-1218-4a65-b5c1-1e7473869742 nodeName:}" failed. No retries permitted until 2026-07-29 06:26:57.954319669 +0000 UTC m=+261.031951413 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3fec1db3-1218-4a65-b5c1-1e7473869742-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vt5qk" (UID: "3fec1db3-1218-4a65-b5c1-1e7473869742") : configmap "kube-rbac-proxy" not found
Jul 29 06:26:49.954524 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:49.954370    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 06:26:49.954524 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:49.954434    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fec1db3-1218-4a65-b5c1-1e7473869742-insights-runtime-extractor-tls podName:3fec1db3-1218-4a65-b5c1-1e7473869742 nodeName:}" failed. No retries permitted until 2026-07-29 06:26:57.95441757 +0000 UTC m=+261.032049320 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3fec1db3-1218-4a65-b5c1-1e7473869742-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vt5qk" (UID: "3fec1db3-1218-4a65-b5c1-1e7473869742") : secret "insights-runtime-extractor-tls" not found
Jul 29 06:26:53.987480 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:53.987443    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1786842b-1ae8-4c6b-9c4c-0d21298a4e15-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-87wjh\" (UID: \"1786842b-1ae8-4c6b-9c4c-0d21298a4e15\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-87wjh"
Jul 29 06:26:53.987969 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:53.987612    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 06:26:53.987969 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:53.987703    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1786842b-1ae8-4c6b-9c4c-0d21298a4e15-cluster-monitoring-operator-tls podName:1786842b-1ae8-4c6b-9c4c-0d21298a4e15 nodeName:}" failed. No retries permitted until 2026-07-29 06:27:09.987675692 +0000 UTC m=+273.065307436 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1786842b-1ae8-4c6b-9c4c-0d21298a4e15-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-87wjh" (UID: "1786842b-1ae8-4c6b-9c4c-0d21298a4e15") : secret "cluster-monitoring-operator-tls" not found
Jul 29 06:26:58.013966 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:58.013879    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fec1db3-1218-4a65-b5c1-1e7473869742-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vt5qk\" (UID: \"3fec1db3-1218-4a65-b5c1-1e7473869742\") " pod="openshift-insights/insights-runtime-extractor-vt5qk"
Jul 29 06:26:58.013966 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:58.013935    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fec1db3-1218-4a65-b5c1-1e7473869742-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vt5qk\" (UID: \"3fec1db3-1218-4a65-b5c1-1e7473869742\") " pod="openshift-insights/insights-runtime-extractor-vt5qk"
Jul 29 06:26:58.014389 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:58.014070    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 06:26:58.014389 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:26:58.014122    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fec1db3-1218-4a65-b5c1-1e7473869742-insights-runtime-extractor-tls podName:3fec1db3-1218-4a65-b5c1-1e7473869742 nodeName:}" failed. No retries permitted until 2026-07-29 06:27:14.014109782 +0000 UTC m=+277.091741528 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3fec1db3-1218-4a65-b5c1-1e7473869742-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vt5qk" (UID: "3fec1db3-1218-4a65-b5c1-1e7473869742") : secret "insights-runtime-extractor-tls" not found
Jul 29 06:26:58.014389 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:26:58.014378    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fec1db3-1218-4a65-b5c1-1e7473869742-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vt5qk\" (UID: \"3fec1db3-1218-4a65-b5c1-1e7473869742\") " pod="openshift-insights/insights-runtime-extractor-vt5qk"
Jul 29 06:27:10.001987 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:10.001946    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1786842b-1ae8-4c6b-9c4c-0d21298a4e15-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-87wjh\" (UID: \"1786842b-1ae8-4c6b-9c4c-0d21298a4e15\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-87wjh"
Jul 29 06:27:10.004493 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:10.004473    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1786842b-1ae8-4c6b-9c4c-0d21298a4e15-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-87wjh\" (UID: \"1786842b-1ae8-4c6b-9c4c-0d21298a4e15\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-87wjh"
Jul 29 06:27:10.208290 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:10.208258    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-87wjh"
Jul 29 06:27:10.329560 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:10.329530    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-87wjh"]
Jul 29 06:27:10.332725 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:27:10.332701    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1786842b_1ae8_4c6b_9c4c_0d21298a4e15.slice/crio-2372c6fc094405dfee0c13a94df21f2008397d9da20898098bb644ed720a4100 WatchSource:0}: Error finding container 2372c6fc094405dfee0c13a94df21f2008397d9da20898098bb644ed720a4100: Status 404 returned error can't find the container with id 2372c6fc094405dfee0c13a94df21f2008397d9da20898098bb644ed720a4100
Jul 29 06:27:11.142785 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:11.142751    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-87wjh" event={"ID":"1786842b-1ae8-4c6b-9c4c-0d21298a4e15","Type":"ContainerStarted","Data":"2372c6fc094405dfee0c13a94df21f2008397d9da20898098bb644ed720a4100"}
Jul 29 06:27:12.627973 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:12.627945    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2xhdp"]
Jul 29 06:27:12.630067 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:12.630051    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2xhdp"
Jul 29 06:27:12.632081 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:12.632062    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 29 06:27:12.632182 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:12.632166    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-544sq\""
Jul 29 06:27:12.637650 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:12.637629    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2xhdp"]
Jul 29 06:27:12.724599 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:12.724575    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/34133ad8-194a-4352-af5f-667836d27cb2-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-2xhdp\" (UID: \"34133ad8-194a-4352-af5f-667836d27cb2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2xhdp"
Jul 29 06:27:12.825903 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:12.825838    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/34133ad8-194a-4352-af5f-667836d27cb2-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-2xhdp\" (UID: \"34133ad8-194a-4352-af5f-667836d27cb2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2xhdp"
Jul 29 06:27:12.826056 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:27:12.826004    2563 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 29 06:27:12.826099 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:27:12.826071    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34133ad8-194a-4352-af5f-667836d27cb2-tls-certificates podName:34133ad8-194a-4352-af5f-667836d27cb2 nodeName:}" failed. No retries permitted until 2026-07-29 06:27:13.32605259 +0000 UTC m=+276.403684337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/34133ad8-194a-4352-af5f-667836d27cb2-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-2xhdp" (UID: "34133ad8-194a-4352-af5f-667836d27cb2") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 29 06:27:13.150347 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:13.150316    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-87wjh" event={"ID":"1786842b-1ae8-4c6b-9c4c-0d21298a4e15","Type":"ContainerStarted","Data":"6a1678b94f68d4a40cb667c0c9b5374c5a05fc4caacf48617a5e8f8e0a317e71"}
Jul 29 06:27:13.163808 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:13.163765    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-87wjh" podStartSLOduration=33.330745191 podStartE2EDuration="35.163751795s" totalImagesPullingTime="1.833006604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:26:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:27:13.162971489 +0000 UTC m=+276.240603256" watchObservedRunningTime="2026-07-29 06:27:13.163751795 +0000 UTC m=+276.241383539"
Jul 29 06:27:13.330408 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:13.330382    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/34133ad8-194a-4352-af5f-667836d27cb2-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-2xhdp\" (UID: \"34133ad8-194a-4352-af5f-667836d27cb2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2xhdp"
Jul 29 06:27:13.332936 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:13.332908    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/34133ad8-194a-4352-af5f-667836d27cb2-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-2xhdp\" (UID: \"34133ad8-194a-4352-af5f-667836d27cb2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2xhdp"
Jul 29 06:27:13.539439 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:13.539351    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2xhdp"
Jul 29 06:27:13.660647 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:13.660619    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2xhdp"]
Jul 29 06:27:13.663866 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:27:13.663827    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34133ad8_194a_4352_af5f_667836d27cb2.slice/crio-b1b9b5a2b6ebe8fc441afdaaf869875f535af5eb1c97aba4008c9ff489a494ae WatchSource:0}: Error finding container b1b9b5a2b6ebe8fc441afdaaf869875f535af5eb1c97aba4008c9ff489a494ae: Status 404 returned error can't find the container with id b1b9b5a2b6ebe8fc441afdaaf869875f535af5eb1c97aba4008c9ff489a494ae
Jul 29 06:27:14.036069 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:14.036032    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fec1db3-1218-4a65-b5c1-1e7473869742-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vt5qk\" (UID: \"3fec1db3-1218-4a65-b5c1-1e7473869742\") " pod="openshift-insights/insights-runtime-extractor-vt5qk"
Jul 29 06:27:14.038427 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:14.038407    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fec1db3-1218-4a65-b5c1-1e7473869742-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vt5qk\" (UID: \"3fec1db3-1218-4a65-b5c1-1e7473869742\") " pod="openshift-insights/insights-runtime-extractor-vt5qk"
Jul 29 06:27:14.155001 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:14.154959    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2xhdp" event={"ID":"34133ad8-194a-4352-af5f-667836d27cb2","Type":"ContainerStarted","Data":"b1b9b5a2b6ebe8fc441afdaaf869875f535af5eb1c97aba4008c9ff489a494ae"}
Jul 29 06:27:14.276983 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:14.276941    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vt5qk"
Jul 29 06:27:14.421796 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:14.421764    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vt5qk"]
Jul 29 06:27:14.650500 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:27:14.650468    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fec1db3_1218_4a65_b5c1_1e7473869742.slice/crio-9e6722e1a5eb69985325ad6591ee18a72754786998da6caa42f8c078a9b2bd85 WatchSource:0}: Error finding container 9e6722e1a5eb69985325ad6591ee18a72754786998da6caa42f8c078a9b2bd85: Status 404 returned error can't find the container with id 9e6722e1a5eb69985325ad6591ee18a72754786998da6caa42f8c078a9b2bd85
Jul 29 06:27:15.159708 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:15.159623    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2xhdp" event={"ID":"34133ad8-194a-4352-af5f-667836d27cb2","Type":"ContainerStarted","Data":"a114038881480f058bea1d4bc451e446caaea0dcbdd90bd2c18fd11b14715cdb"}
Jul 29 06:27:15.160149 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:15.159708    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2xhdp"
Jul 29 06:27:15.160992 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:15.160951    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vt5qk" event={"ID":"3fec1db3-1218-4a65-b5c1-1e7473869742","Type":"ContainerStarted","Data":"9e6722e1a5eb69985325ad6591ee18a72754786998da6caa42f8c078a9b2bd85"}
Jul 29 06:27:15.165279 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:15.165255    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2xhdp"
Jul 29 06:27:15.173264 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:15.173189    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2xhdp" podStartSLOduration=2.01719913 podStartE2EDuration="3.173174276s" totalImagesPullingTime="1.155975146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:27:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:27:15.172909177 +0000 UTC m=+278.250540946" watchObservedRunningTime="2026-07-29 06:27:15.173174276 +0000 UTC m=+278.250806043"
Jul 29 06:27:15.689088 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:15.689052    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-lm28j"]
Jul 29 06:27:15.691273 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:15.691245    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-lm28j"
Jul 29 06:27:15.693442 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:15.693416    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 29 06:27:15.693566 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:15.693465    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 29 06:27:15.694166 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:15.694145    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 29 06:27:15.694372 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:15.694355    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-r4qd6\""
Jul 29 06:27:15.700938 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:15.700916    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-lm28j"]
Jul 29 06:27:15.754552 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:15.754515    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c169934-87cc-43de-b7e7-2d26c36ffd35-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-lm28j\" (UID: \"9c169934-87cc-43de-b7e7-2d26c36ffd35\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lm28j"
Jul 29 06:27:15.754713 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:15.754590    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84tck\" (UniqueName: \"kubernetes.io/projected/9c169934-87cc-43de-b7e7-2d26c36ffd35-kube-api-access-84tck\") pod \"prometheus-operator-84d69f794d-lm28j\" (UID: \"9c169934-87cc-43de-b7e7-2d26c36ffd35\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lm28j"
Jul 29 06:27:15.754713 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:15.754652    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9c169934-87cc-43de-b7e7-2d26c36ffd35-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-lm28j\" (UID: \"9c169934-87cc-43de-b7e7-2d26c36ffd35\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lm28j"
Jul 29 06:27:15.754832 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:15.754720    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c169934-87cc-43de-b7e7-2d26c36ffd35-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-lm28j\" (UID: \"9c169934-87cc-43de-b7e7-2d26c36ffd35\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lm28j"
Jul 29 06:27:15.855376 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:15.855345    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9c169934-87cc-43de-b7e7-2d26c36ffd35-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-lm28j\" (UID: \"9c169934-87cc-43de-b7e7-2d26c36ffd35\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lm28j"
Jul 29 06:27:15.855540 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:15.855403    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c169934-87cc-43de-b7e7-2d26c36ffd35-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-lm28j\" (UID: \"9c169934-87cc-43de-b7e7-2d26c36ffd35\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lm28j"
Jul 29 06:27:15.855540 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:15.855489    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c169934-87cc-43de-b7e7-2d26c36ffd35-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-lm28j\" (UID: \"9c169934-87cc-43de-b7e7-2d26c36ffd35\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lm28j"
Jul 29 06:27:15.855540 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:27:15.855499    2563 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 29 06:27:15.855540 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:15.855531    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84tck\" (UniqueName: \"kubernetes.io/projected/9c169934-87cc-43de-b7e7-2d26c36ffd35-kube-api-access-84tck\") pod \"prometheus-operator-84d69f794d-lm28j\" (UID: \"9c169934-87cc-43de-b7e7-2d26c36ffd35\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lm28j"
Jul 29 06:27:15.855725 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:27:15.855565    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c169934-87cc-43de-b7e7-2d26c36ffd35-prometheus-operator-tls podName:9c169934-87cc-43de-b7e7-2d26c36ffd35 nodeName:}" failed. No retries permitted until 2026-07-29 06:27:16.355545053 +0000 UTC m=+279.433176803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/9c169934-87cc-43de-b7e7-2d26c36ffd35-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-lm28j" (UID: "9c169934-87cc-43de-b7e7-2d26c36ffd35") : secret "prometheus-operator-tls" not found
Jul 29 06:27:15.856299 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:15.856278    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c169934-87cc-43de-b7e7-2d26c36ffd35-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-lm28j\" (UID: \"9c169934-87cc-43de-b7e7-2d26c36ffd35\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lm28j"
Jul 29 06:27:15.858298 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:15.858274    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c169934-87cc-43de-b7e7-2d26c36ffd35-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-lm28j\" (UID: \"9c169934-87cc-43de-b7e7-2d26c36ffd35\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lm28j"
Jul 29 06:27:15.863278 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:15.863254    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84tck\" (UniqueName: \"kubernetes.io/projected/9c169934-87cc-43de-b7e7-2d26c36ffd35-kube-api-access-84tck\") pod \"prometheus-operator-84d69f794d-lm28j\" (UID: \"9c169934-87cc-43de-b7e7-2d26c36ffd35\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lm28j"
Jul 29 06:27:16.360806 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:16.360757    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9c169934-87cc-43de-b7e7-2d26c36ffd35-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-lm28j\" (UID: \"9c169934-87cc-43de-b7e7-2d26c36ffd35\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lm28j"
Jul 29 06:27:16.363641 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:16.363617    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9c169934-87cc-43de-b7e7-2d26c36ffd35-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-lm28j\" (UID: \"9c169934-87cc-43de-b7e7-2d26c36ffd35\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lm28j"
Jul 29 06:27:16.604029 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:16.603982    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-lm28j"
Jul 29 06:27:16.744555 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:16.744520    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-lm28j"]
Jul 29 06:27:16.747403 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:27:16.747372    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c169934_87cc_43de_b7e7_2d26c36ffd35.slice/crio-71aabe7d955f50877cdab5a344a36a69c823509658a6126c7ce99dd38bbe325d WatchSource:0}: Error finding container 71aabe7d955f50877cdab5a344a36a69c823509658a6126c7ce99dd38bbe325d: Status 404 returned error can't find the container with id 71aabe7d955f50877cdab5a344a36a69c823509658a6126c7ce99dd38bbe325d
Jul 29 06:27:17.168623 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:17.168592    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-lm28j" event={"ID":"9c169934-87cc-43de-b7e7-2d26c36ffd35","Type":"ContainerStarted","Data":"71aabe7d955f50877cdab5a344a36a69c823509658a6126c7ce99dd38bbe325d"}
Jul 29 06:27:17.846217 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:27:17.846173    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj" podUID="e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48"
Jul 29 06:27:17.846682 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:27:17.846310    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-bd28d" podUID="4f4e0417-82ca-4f78-9043-85dc8af1d71b"
Jul 29 06:27:17.846682 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:27:17.846321    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-74c9f884b7-mrklz" podUID="37763295-c6d9-4846-8891-f4c5b78ec095"
Jul 29 06:27:18.171792 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:18.171711    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-mrklz"
Jul 29 06:27:18.171987 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:18.171711    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:27:18.171987 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:18.171723    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bd28d"
Jul 29 06:27:20.181235 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:20.181206    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vt5qk" event={"ID":"3fec1db3-1218-4a65-b5c1-1e7473869742","Type":"ContainerStarted","Data":"aa8e09317f9afd078ce4b4bbe27409ce072c55fef5e53ac21a1fa2af6a646755"}
Jul 29 06:27:20.183057 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:20.183037    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-lm28j" event={"ID":"9c169934-87cc-43de-b7e7-2d26c36ffd35","Type":"ContainerStarted","Data":"46dad5dee784a628b006d03ab3ee54a5da0b2406a4a8f126a181b2c1be4461ab"}
Jul 29 06:27:20.183157 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:20.183085    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-lm28j" event={"ID":"9c169934-87cc-43de-b7e7-2d26c36ffd35","Type":"ContainerStarted","Data":"e26e18db02acd1236763479376daa15194655afc4f628dbe54fd00a622988143"}
Jul 29 06:27:20.197328 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:20.197278    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-lm28j" podStartSLOduration=1.9077169550000002 podStartE2EDuration="5.197261991s" totalImagesPullingTime="3.289545036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:27:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:27:20.196496031 +0000 UTC m=+283.274127810" watchObservedRunningTime="2026-07-29 06:27:20.197261991 +0000 UTC m=+283.274893760"
Jul 29 06:27:21.810323 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:21.810290    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-tls\") pod \"image-registry-784bf54c5f-hj6jj\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") " pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:27:21.810323 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:21.810336    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f4e0417-82ca-4f78-9043-85dc8af1d71b-metrics-tls\") pod \"dns-default-bd28d\" (UID: \"4f4e0417-82ca-4f78-9043-85dc8af1d71b\") " pod="openshift-dns/dns-default-bd28d"
Jul 29 06:27:21.810983 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:21.810401    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/37763295-c6d9-4846-8891-f4c5b78ec095-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-mrklz\" (UID: \"37763295-c6d9-4846-8891-f4c5b78ec095\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-mrklz"
Jul 29 06:27:21.813308 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:21.813277    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/37763295-c6d9-4846-8891-f4c5b78ec095-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-mrklz\" (UID: \"37763295-c6d9-4846-8891-f4c5b78ec095\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-mrklz"
Jul 29 06:27:21.813440 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:21.813416    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f4e0417-82ca-4f78-9043-85dc8af1d71b-metrics-tls\") pod \"dns-default-bd28d\" (UID: \"4f4e0417-82ca-4f78-9043-85dc8af1d71b\") " pod="openshift-dns/dns-default-bd28d"
Jul 29 06:27:21.813440 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:21.813427    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-tls\") pod \"image-registry-784bf54c5f-hj6jj\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") " pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:27:22.021094 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.021063    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-dcwzq"]
Jul 29 06:27:22.031331 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.031309    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dcwzq"
Jul 29 06:27:22.033252 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.033218    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 29 06:27:22.033376 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.033271    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 29 06:27:22.033376 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.033293    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-9tr75\""
Jul 29 06:27:22.033520 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.033446    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-x7d4r"]
Jul 29 06:27:22.036909 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.036888    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-dcwzq"]
Jul 29 06:27:22.037022 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.036960    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x7d4r"
Jul 29 06:27:22.039457 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.039258    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 29 06:27:22.039457 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.039294    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 29 06:27:22.039457 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.039377    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 29 06:27:22.039457 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.039391    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-2pjpz\""
Jul 29 06:27:22.045746 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.045725    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8nmt6"]
Jul 29 06:27:22.050640 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.050186    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-x7d4r"]
Jul 29 06:27:22.050640 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.050331    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8nmt6"
Jul 29 06:27:22.052780 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.052472    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 29 06:27:22.052780 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.052622    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-chckx\""
Jul 29 06:27:22.052780 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.052712    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 29 06:27:22.053020 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.052806    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 29 06:27:22.077529 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.077507    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vtwx9\""
Jul 29 06:27:22.078425 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.078404    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-z5482\""
Jul 29 06:27:22.078812 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.078564    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9hq6v\""
Jul 29 06:27:22.084247 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.084224    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:27:22.085390 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.085371    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bd28d"
Jul 29 06:27:22.085497 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.085410    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-mrklz"
Jul 29 06:27:22.113847 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.113814    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d2702bd-5995-4c4d-b90a-dc7e40df640d-metrics-client-ca\") pod \"node-exporter-8nmt6\" (UID: \"1d2702bd-5995-4c4d-b90a-dc7e40df640d\") " pod="openshift-monitoring/node-exporter-8nmt6"
Jul 29 06:27:22.114007 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.113889    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/46f8ec36-334c-400b-aa93-b57679124eaf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-dcwzq\" (UID: \"46f8ec36-334c-400b-aa93-b57679124eaf\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dcwzq"
Jul 29 06:27:22.114007 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.113920    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-x7d4r\" (UID: \"8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x7d4r"
Jul 29 06:27:22.114007 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.113959    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-x7d4r\" (UID: \"8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x7d4r"
Jul 29 06:27:22.114007 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.114006    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46f8ec36-334c-400b-aa93-b57679124eaf-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-dcwzq\" (UID: \"46f8ec36-334c-400b-aa93-b57679124eaf\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dcwzq"
Jul 29 06:27:22.114184 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.114028    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1d2702bd-5995-4c4d-b90a-dc7e40df640d-node-exporter-textfile\") pod \"node-exporter-8nmt6\" (UID: \"1d2702bd-5995-4c4d-b90a-dc7e40df640d\") " pod="openshift-monitoring/node-exporter-8nmt6"
Jul 29 06:27:22.114184 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.114052    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd9wb\" (UniqueName: \"kubernetes.io/projected/1d2702bd-5995-4c4d-b90a-dc7e40df640d-kube-api-access-dd9wb\") pod \"node-exporter-8nmt6\" (UID: \"1d2702bd-5995-4c4d-b90a-dc7e40df640d\") " pod="openshift-monitoring/node-exporter-8nmt6"
Jul 29 06:27:22.114184 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.114074    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gcwx\" (UniqueName: \"kubernetes.io/projected/8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8-kube-api-access-9gcwx\") pod \"kube-state-metrics-56d9f944b4-x7d4r\" (UID: \"8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x7d4r"
Jul 29 06:27:22.114184 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.114117    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d2702bd-5995-4c4d-b90a-dc7e40df640d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8nmt6\" (UID: \"1d2702bd-5995-4c4d-b90a-dc7e40df640d\") " pod="openshift-monitoring/node-exporter-8nmt6"
Jul 29 06:27:22.114184 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.114140    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1d2702bd-5995-4c4d-b90a-dc7e40df640d-node-exporter-accelerators-collector-config\") pod \"node-exporter-8nmt6\" (UID: \"1d2702bd-5995-4c4d-b90a-dc7e40df640d\") " pod="openshift-monitoring/node-exporter-8nmt6"
Jul 29 06:27:22.114184 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.114167    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1d2702bd-5995-4c4d-b90a-dc7e40df640d-root\") pod \"node-exporter-8nmt6\" (UID: \"1d2702bd-5995-4c4d-b90a-dc7e40df640d\") " pod="openshift-monitoring/node-exporter-8nmt6"
Jul 29 06:27:22.114389 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.114196    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-x7d4r\" (UID: \"8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x7d4r"
Jul 29 06:27:22.114389 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.114253    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d2702bd-5995-4c4d-b90a-dc7e40df640d-node-exporter-tls\") pod \"node-exporter-8nmt6\" (UID: \"1d2702bd-5995-4c4d-b90a-dc7e40df640d\") " pod="openshift-monitoring/node-exporter-8nmt6"
Jul 29 06:27:22.114389 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.114312    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46f8ec36-334c-400b-aa93-b57679124eaf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-dcwzq\" (UID: \"46f8ec36-334c-400b-aa93-b57679124eaf\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dcwzq"
Jul 29 06:27:22.114389 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.114337    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-x7d4r\" (UID: \"8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x7d4r"
Jul 29 06:27:22.114389 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.114364    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d2702bd-5995-4c4d-b90a-dc7e40df640d-sys\") pod \"node-exporter-8nmt6\" (UID: \"1d2702bd-5995-4c4d-b90a-dc7e40df640d\") " pod="openshift-monitoring/node-exporter-8nmt6"
Jul 29 06:27:22.114389 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.114381    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kllh7\" (UniqueName: \"kubernetes.io/projected/46f8ec36-334c-400b-aa93-b57679124eaf-kube-api-access-kllh7\") pod \"openshift-state-metrics-b8f49f5c6-dcwzq\" (UID: \"46f8ec36-334c-400b-aa93-b57679124eaf\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dcwzq"
Jul 29 06:27:22.114648 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.114407    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-x7d4r\" (UID: \"8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x7d4r"
Jul 29 06:27:22.114648 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.114496    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1d2702bd-5995-4c4d-b90a-dc7e40df640d-node-exporter-wtmp\") pod \"node-exporter-8nmt6\" (UID: \"1d2702bd-5995-4c4d-b90a-dc7e40df640d\") " pod="openshift-monitoring/node-exporter-8nmt6"
Jul 29 06:27:22.215568 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.215539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46f8ec36-334c-400b-aa93-b57679124eaf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-dcwzq\" (UID: \"46f8ec36-334c-400b-aa93-b57679124eaf\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dcwzq"
Jul 29 06:27:22.215753 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.215580    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-x7d4r\" (UID: \"8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x7d4r"
Jul 29 06:27:22.215753 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.215614    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d2702bd-5995-4c4d-b90a-dc7e40df640d-sys\") pod \"node-exporter-8nmt6\" (UID: \"1d2702bd-5995-4c4d-b90a-dc7e40df640d\") " pod="openshift-monitoring/node-exporter-8nmt6"
Jul 29 06:27:22.215753 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.215637    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kllh7\" (UniqueName: \"kubernetes.io/projected/46f8ec36-334c-400b-aa93-b57679124eaf-kube-api-access-kllh7\") pod \"openshift-state-metrics-b8f49f5c6-dcwzq\" (UID: \"46f8ec36-334c-400b-aa93-b57679124eaf\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dcwzq"
Jul 29 06:27:22.215753 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.215660    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-x7d4r\" (UID: \"8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x7d4r"
Jul 29 06:27:22.215753 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.215716    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1d2702bd-5995-4c4d-b90a-dc7e40df640d-node-exporter-wtmp\") pod \"node-exporter-8nmt6\" (UID: \"1d2702bd-5995-4c4d-b90a-dc7e40df640d\") " pod="openshift-monitoring/node-exporter-8nmt6"
Jul 29 06:27:22.215753 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.215740    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d2702bd-5995-4c4d-b90a-dc7e40df640d-metrics-client-ca\") pod \"node-exporter-8nmt6\" (UID: \"1d2702bd-5995-4c4d-b90a-dc7e40df640d\") " pod="openshift-monitoring/node-exporter-8nmt6"
Jul 29 06:27:22.216090 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.215755    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d2702bd-5995-4c4d-b90a-dc7e40df640d-sys\") pod \"node-exporter-8nmt6\" (UID: \"1d2702bd-5995-4c4d-b90a-dc7e40df640d\") " pod="openshift-monitoring/node-exporter-8nmt6"
Jul 29 06:27:22.216090 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.215788    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/46f8ec36-334c-400b-aa93-b57679124eaf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-dcwzq\" (UID: \"46f8ec36-334c-400b-aa93-b57679124eaf\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dcwzq"
Jul 29 06:27:22.216090 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.215815    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-x7d4r\" (UID: \"8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x7d4r"
Jul 29 06:27:22.216090 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.215876    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-x7d4r\" (UID: \"8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x7d4r"
Jul 29 06:27:22.216090 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.215915    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46f8ec36-334c-400b-aa93-b57679124eaf-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-dcwzq\" (UID: \"46f8ec36-334c-400b-aa93-b57679124eaf\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dcwzq"
Jul 29 06:27:22.216090 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.215940    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1d2702bd-5995-4c4d-b90a-dc7e40df640d-node-exporter-wtmp\") pod \"node-exporter-8nmt6\" (UID: \"1d2702bd-5995-4c4d-b90a-dc7e40df640d\") " pod="openshift-monitoring/node-exporter-8nmt6"
Jul 29 06:27:22.216090 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.215944    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1d2702bd-5995-4c4d-b90a-dc7e40df640d-node-exporter-textfile\") pod \"node-exporter-8nmt6\" (UID: \"1d2702bd-5995-4c4d-b90a-dc7e40df640d\") " pod="openshift-monitoring/node-exporter-8nmt6"
Jul 29 06:27:22.216090 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.215995    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd9wb\" (UniqueName: \"kubernetes.io/projected/1d2702bd-5995-4c4d-b90a-dc7e40df640d-kube-api-access-dd9wb\") pod \"node-exporter-8nmt6\" (UID: \"1d2702bd-5995-4c4d-b90a-dc7e40df640d\") " pod="openshift-monitoring/node-exporter-8nmt6"
Jul 29 06:27:22.216090 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.216026    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gcwx\" (UniqueName: \"kubernetes.io/projected/8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8-kube-api-access-9gcwx\") pod \"kube-state-metrics-56d9f944b4-x7d4r\" (UID: \"8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x7d4r"
Jul 29 06:27:22.216090 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.216059    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d2702bd-5995-4c4d-b90a-dc7e40df640d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8nmt6\" (UID: \"1d2702bd-5995-4c4d-b90a-dc7e40df640d\") " pod="openshift-monitoring/node-exporter-8nmt6"
Jul 29 06:27:22.216090 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.216090    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1d2702bd-5995-4c4d-b90a-dc7e40df640d-node-exporter-accelerators-collector-config\") pod \"node-exporter-8nmt6\" (UID: \"1d2702bd-5995-4c4d-b90a-dc7e40df640d\") " pod="openshift-monitoring/node-exporter-8nmt6"
Jul 29 06:27:22.216624 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.216119    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1d2702bd-5995-4c4d-b90a-dc7e40df640d-root\") pod \"node-exporter-8nmt6\" (UID: \"1d2702bd-5995-4c4d-b90a-dc7e40df640d\") " pod="openshift-monitoring/node-exporter-8nmt6"
Jul 29 06:27:22.216624 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.216149    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-x7d4r\" (UID: \"8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x7d4r"
Jul 29 06:27:22.216624 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.216177    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d2702bd-5995-4c4d-b90a-dc7e40df640d-node-exporter-tls\") pod \"node-exporter-8nmt6\" (UID: \"1d2702bd-5995-4c4d-b90a-dc7e40df640d\") " pod="openshift-monitoring/node-exporter-8nmt6"
Jul 29 06:27:22.216624 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.216212    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1d2702bd-5995-4c4d-b90a-dc7e40df640d-node-exporter-textfile\") pod \"node-exporter-8nmt6\" (UID: \"1d2702bd-5995-4c4d-b90a-dc7e40df640d\") " pod="openshift-monitoring/node-exporter-8nmt6"
Jul 29 06:27:22.216624 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.216380    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-x7d4r\" (UID: \"8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x7d4r"
Jul 29 06:27:22.216624 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:27:22.216470    2563 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 29 06:27:22.216624 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:27:22.216533    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46f8ec36-334c-400b-aa93-b57679124eaf-openshift-state-metrics-tls podName:46f8ec36-334c-400b-aa93-b57679124eaf nodeName:}" failed. No retries permitted until 2026-07-29 06:27:22.716510219 +0000 UTC m=+285.794141970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/46f8ec36-334c-400b-aa93-b57679124eaf-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-dcwzq" (UID: "46f8ec36-334c-400b-aa93-b57679124eaf") : secret "openshift-state-metrics-tls" not found
Jul 29 06:27:22.217012 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.216630    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d2702bd-5995-4c4d-b90a-dc7e40df640d-metrics-client-ca\") pod \"node-exporter-8nmt6\" (UID: \"1d2702bd-5995-4c4d-b90a-dc7e40df640d\") " pod="openshift-monitoring/node-exporter-8nmt6"
Jul 29 06:27:22.217012 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.216695    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1d2702bd-5995-4c4d-b90a-dc7e40df640d-root\") pod \"node-exporter-8nmt6\" (UID: \"1d2702bd-5995-4c4d-b90a-dc7e40df640d\") " pod="openshift-monitoring/node-exporter-8nmt6"
Jul 29 06:27:22.217012 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.216994    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-x7d4r\" (UID: \"8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x7d4r"
Jul 29 06:27:22.217462 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.217390    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1d2702bd-5995-4c4d-b90a-dc7e40df640d-node-exporter-accelerators-collector-config\") pod \"node-exporter-8nmt6\" (UID: \"1d2702bd-5995-4c4d-b90a-dc7e40df640d\") " pod="openshift-monitoring/node-exporter-8nmt6"
Jul 29 06:27:22.217593 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.217552    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46f8ec36-334c-400b-aa93-b57679124eaf-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-dcwzq\" (UID: \"46f8ec36-334c-400b-aa93-b57679124eaf\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dcwzq"
Jul 29 06:27:22.217593 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.217569    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-x7d4r\" (UID: \"8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x7d4r"
Jul 29 06:27:22.219056 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.219030    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-x7d4r\" (UID: \"8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x7d4r"
Jul 29 06:27:22.219056 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.219041    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46f8ec36-334c-400b-aa93-b57679124eaf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-dcwzq\" (UID: \"46f8ec36-334c-400b-aa93-b57679124eaf\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dcwzq"
Jul 29 06:27:22.219261 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.219223    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-x7d4r\" (UID: \"8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x7d4r"
Jul 29 06:27:22.220410 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.220386    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d2702bd-5995-4c4d-b90a-dc7e40df640d-node-exporter-tls\") pod \"node-exporter-8nmt6\" (UID: \"1d2702bd-5995-4c4d-b90a-dc7e40df640d\") " pod="openshift-monitoring/node-exporter-8nmt6"
Jul 29 06:27:22.220616 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.220593    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d2702bd-5995-4c4d-b90a-dc7e40df640d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8nmt6\" (UID: \"1d2702bd-5995-4c4d-b90a-dc7e40df640d\") " pod="openshift-monitoring/node-exporter-8nmt6"
Jul 29 06:27:22.227028 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.226990    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kllh7\" (UniqueName: \"kubernetes.io/projected/46f8ec36-334c-400b-aa93-b57679124eaf-kube-api-access-kllh7\") pod \"openshift-state-metrics-b8f49f5c6-dcwzq\" (UID: \"46f8ec36-334c-400b-aa93-b57679124eaf\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dcwzq"
Jul 29 06:27:22.227145 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.227113    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gcwx\" (UniqueName: \"kubernetes.io/projected/8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8-kube-api-access-9gcwx\") pod \"kube-state-metrics-56d9f944b4-x7d4r\" (UID: \"8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x7d4r"
Jul 29 06:27:22.227597 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.227576    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd9wb\" (UniqueName: \"kubernetes.io/projected/1d2702bd-5995-4c4d-b90a-dc7e40df640d-kube-api-access-dd9wb\") pod \"node-exporter-8nmt6\" (UID: \"1d2702bd-5995-4c4d-b90a-dc7e40df640d\") " pod="openshift-monitoring/node-exporter-8nmt6"
Jul 29 06:27:22.353090 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.353015    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x7d4r"
Jul 29 06:27:22.386514 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.386486    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8nmt6"
Jul 29 06:27:22.720682 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.720597    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/46f8ec36-334c-400b-aa93-b57679124eaf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-dcwzq\" (UID: \"46f8ec36-334c-400b-aa93-b57679124eaf\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dcwzq"
Jul 29 06:27:22.723411 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.723379    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/46f8ec36-334c-400b-aa93-b57679124eaf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-dcwzq\" (UID: \"46f8ec36-334c-400b-aa93-b57679124eaf\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dcwzq"
Jul 29 06:27:22.945334 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:22.945292    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dcwzq"
Jul 29 06:27:23.098150 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.098011    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 06:27:23.108309 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.108283    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.110548 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.110487    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 29 06:27:23.110548 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.110528    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 29 06:27:23.112112 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.110774    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tdf2f\""
Jul 29 06:27:23.112112 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.111035    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 29 06:27:23.112112 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.111137    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 29 06:27:23.112112 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.111205    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 29 06:27:23.112112 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.111337    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 29 06:27:23.112112 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.111403    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 29 06:27:23.112112 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.111505    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 29 06:27:23.117715 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.117695    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 29 06:27:23.119364 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.119342    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 06:27:23.225923 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.225893    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-config-volume\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.226101 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.225939    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.226101 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.226031    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09946786-61d2-4a1f-ad32-718acd329950-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.226101 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.226074    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.226249 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.226100    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/09946786-61d2-4a1f-ad32-718acd329950-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.226249 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.226126    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hvnw\" (UniqueName: \"kubernetes.io/projected/09946786-61d2-4a1f-ad32-718acd329950-kube-api-access-4hvnw\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.226249 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.226191    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.226249 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.226229    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-web-config\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.226409 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.226252    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09946786-61d2-4a1f-ad32-718acd329950-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.226409 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.226303    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.226409 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.226335    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/09946786-61d2-4a1f-ad32-718acd329950-tls-assets\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.226409 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.226391    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/09946786-61d2-4a1f-ad32-718acd329950-config-out\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.226571 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.226421    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.327404 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.327366    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-web-config\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.327586 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.327410    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09946786-61d2-4a1f-ad32-718acd329950-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.327586 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.327449    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.327586 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.327479    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/09946786-61d2-4a1f-ad32-718acd329950-tls-assets\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.327586 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.327523    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/09946786-61d2-4a1f-ad32-718acd329950-config-out\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.327586 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.327548    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.328577 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.327927    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-config-volume\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.328577 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.327978    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.328577 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.328025    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09946786-61d2-4a1f-ad32-718acd329950-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.328577 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.328057    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.328577 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.328078    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/09946786-61d2-4a1f-ad32-718acd329950-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.328577 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.328103    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hvnw\" (UniqueName: \"kubernetes.io/projected/09946786-61d2-4a1f-ad32-718acd329950-kube-api-access-4hvnw\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.328577 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.328142    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.328577 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.328395    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09946786-61d2-4a1f-ad32-718acd329950-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.329075 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.329038    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09946786-61d2-4a1f-ad32-718acd329950-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.329519 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.329466    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/09946786-61d2-4a1f-ad32-718acd329950-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.330753 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.330722    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.332579 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.332553    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/09946786-61d2-4a1f-ad32-718acd329950-config-out\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.333888 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.332915    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/09946786-61d2-4a1f-ad32-718acd329950-tls-assets\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.333888 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.333271    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.333888 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.333345    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-web-config\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.333888 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.333381    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-config-volume\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.334141 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.333960    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.334383 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.334364    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.334585 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.334558    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.337476 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.337438    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hvnw\" (UniqueName: \"kubernetes.io/projected/09946786-61d2-4a1f-ad32-718acd329950-kube-api-access-4hvnw\") pod \"alertmanager-main-0\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.424715 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.424652    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:27:23.556880 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:27:23.556830    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d2702bd_5995_4c4d_b90a_dc7e40df640d.slice/crio-d88ba5336ae892aaa6e8a9a02fcbc75a9e757604e64bb174d39c485f585bfca2 WatchSource:0}: Error finding container d88ba5336ae892aaa6e8a9a02fcbc75a9e757604e64bb174d39c485f585bfca2: Status 404 returned error can't find the container with id d88ba5336ae892aaa6e8a9a02fcbc75a9e757604e64bb174d39c485f585bfca2
Jul 29 06:27:23.747189 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.747154    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-dcwzq"]
Jul 29 06:27:23.765802 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.765736    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-mrklz"]
Jul 29 06:27:23.769489 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:23.769463    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-x7d4r"]
Jul 29 06:27:23.776410 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:27:23.776379    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8227fcbe_b2cf_4ea7_bc3d_1d51bfe0fcf8.slice/crio-631f0164f0a3d3d33ee464848a15f3d04ea0c82ac1d5630fb74217127c76465d WatchSource:0}: Error finding container 631f0164f0a3d3d33ee464848a15f3d04ea0c82ac1d5630fb74217127c76465d: Status 404 returned error can't find the container with id 631f0164f0a3d3d33ee464848a15f3d04ea0c82ac1d5630fb74217127c76465d
Jul 29 06:27:24.022272 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:24.022237    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bd28d"]
Jul 29 06:27:24.025057 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:27:24.025028    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f4e0417_82ca_4f78_9043_85dc8af1d71b.slice/crio-cfbe4593860b9549a323ad5d91bee1b5cdc2773c47a8dfa055b569cb3508dd6d WatchSource:0}: Error finding container cfbe4593860b9549a323ad5d91bee1b5cdc2773c47a8dfa055b569cb3508dd6d: Status 404 returned error can't find the container with id cfbe4593860b9549a323ad5d91bee1b5cdc2773c47a8dfa055b569cb3508dd6d
Jul 29 06:27:24.025247 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:24.025226    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-784bf54c5f-hj6jj"]
Jul 29 06:27:24.029297 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:27:24.029275    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode68b7c53_3f5b_4bcb_8d4f_a58f9d2f6c48.slice/crio-5568c8466674685e9693fd5208ea289bbc3abbce095a7fb8579db3cd4431ecb9 WatchSource:0}: Error finding container 5568c8466674685e9693fd5208ea289bbc3abbce095a7fb8579db3cd4431ecb9: Status 404 returned error can't find the container with id 5568c8466674685e9693fd5208ea289bbc3abbce095a7fb8579db3cd4431ecb9
Jul 29 06:27:24.034637 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:24.034620    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 06:27:24.038316 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:27:24.038287    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09946786_61d2_4a1f_ad32_718acd329950.slice/crio-f69ce91c388213741fb76245b0da7acfb6f17df49a060d0cd427094c5ee6b30c WatchSource:0}: Error finding container f69ce91c388213741fb76245b0da7acfb6f17df49a060d0cd427094c5ee6b30c: Status 404 returned error can't find the container with id f69ce91c388213741fb76245b0da7acfb6f17df49a060d0cd427094c5ee6b30c
Jul 29 06:27:24.200626 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:24.200586    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj" event={"ID":"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48","Type":"ContainerStarted","Data":"f41c47905180ef7c294eb0fa95949c47381e4c3bd646a655bb83a60c4226e26c"}
Jul 29 06:27:24.200806 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:24.200680    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj" event={"ID":"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48","Type":"ContainerStarted","Data":"5568c8466674685e9693fd5208ea289bbc3abbce095a7fb8579db3cd4431ecb9"}
Jul 29 06:27:24.200936 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:24.200916    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:27:24.203124 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:24.203101    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vt5qk" event={"ID":"3fec1db3-1218-4a65-b5c1-1e7473869742","Type":"ContainerStarted","Data":"1087ae19503bdb793e431aefa9f0e1d9f4d31979854fa402d1707a701b09c9ab"}
Jul 29 06:27:24.204444 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:24.204417    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"09946786-61d2-4a1f-ad32-718acd329950","Type":"ContainerStarted","Data":"f69ce91c388213741fb76245b0da7acfb6f17df49a060d0cd427094c5ee6b30c"}
Jul 29 06:27:24.205799 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:24.205767    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-mrklz" event={"ID":"37763295-c6d9-4846-8891-f4c5b78ec095","Type":"ContainerStarted","Data":"1ab28476002687cb0410d01fe4b503a7cdb757ba1996972d2d6ade897c4b79d5"}
Jul 29 06:27:24.207759 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:24.207732    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dcwzq" event={"ID":"46f8ec36-334c-400b-aa93-b57679124eaf","Type":"ContainerStarted","Data":"f0cd964a78a5284e0a928b5865a810b138567876e725d98639285d8208e7a8e8"}
Jul 29 06:27:24.207888 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:24.207786    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dcwzq" event={"ID":"46f8ec36-334c-400b-aa93-b57679124eaf","Type":"ContainerStarted","Data":"018453cfc5d804306f6d6dcab0c82936418a7d5f9779ea16b1ddf1edc98bf6ac"}
Jul 29 06:27:24.207888 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:24.207816    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dcwzq" event={"ID":"46f8ec36-334c-400b-aa93-b57679124eaf","Type":"ContainerStarted","Data":"99974556e534eaaec39e028eed8b3c64c979f0b6fd634b289be9c5f81b872da1"}
Jul 29 06:27:24.209008 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:24.208974    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8nmt6" event={"ID":"1d2702bd-5995-4c4d-b90a-dc7e40df640d","Type":"ContainerStarted","Data":"d88ba5336ae892aaa6e8a9a02fcbc75a9e757604e64bb174d39c485f585bfca2"}
Jul 29 06:27:24.210408 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:24.210378    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x7d4r" event={"ID":"8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8","Type":"ContainerStarted","Data":"631f0164f0a3d3d33ee464848a15f3d04ea0c82ac1d5630fb74217127c76465d"}
Jul 29 06:27:24.211719 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:24.211701    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bd28d" event={"ID":"4f4e0417-82ca-4f78-9043-85dc8af1d71b","Type":"ContainerStarted","Data":"cfbe4593860b9549a323ad5d91bee1b5cdc2773c47a8dfa055b569cb3508dd6d"}
Jul 29 06:27:24.219146 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:24.219093    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj" podStartSLOduration=287.219074161 podStartE2EDuration="4m47.219074161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:27:24.217659248 +0000 UTC m=+287.295291016" watchObservedRunningTime="2026-07-29 06:27:24.219074161 +0000 UTC m=+287.296705957"
Jul 29 06:27:25.223146 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:25.223058    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8nmt6" event={"ID":"1d2702bd-5995-4c4d-b90a-dc7e40df640d","Type":"ContainerStarted","Data":"9e84881cc7d85e4aa2009702a7de7f54c877a741f9d53c323589a48eb0de9608"}
Jul 29 06:27:26.227930 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:26.227895    2563 generic.go:356] "Generic (PLEG): container finished" podID="1d2702bd-5995-4c4d-b90a-dc7e40df640d" containerID="9e84881cc7d85e4aa2009702a7de7f54c877a741f9d53c323589a48eb0de9608" exitCode=0
Jul 29 06:27:26.228372 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:26.227998    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8nmt6" event={"ID":"1d2702bd-5995-4c4d-b90a-dc7e40df640d","Type":"ContainerDied","Data":"9e84881cc7d85e4aa2009702a7de7f54c877a741f9d53c323589a48eb0de9608"}
Jul 29 06:27:28.240215 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.240184    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 06:27:28.253379 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.253349    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.258630 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.258138    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 29 06:27:28.258630 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.258354    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ef390dc5dvkue\""
Jul 29 06:27:28.258630 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.258532    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-bvlrp\""
Jul 29 06:27:28.259050 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.258708    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 29 06:27:28.259050 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.258904    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 29 06:27:28.259163 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.259075    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 29 06:27:28.261463 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.260887    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 29 06:27:28.262230 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.261677    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 29 06:27:28.262230 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.261691    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 29 06:27:28.262230 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.261712    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 29 06:27:28.262230 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.261932    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 29 06:27:28.262230 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.261944    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 29 06:27:28.262230 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.261918    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 29 06:27:28.262230 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.261777    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 29 06:27:28.262230 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.262132    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 29 06:27:28.262230 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.262177    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 29 06:27:28.262749 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.262289    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 06:27:28.264121 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.263713    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 29 06:27:28.377701 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.377667    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.377885 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.377717    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.377885 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.377757    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-config\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.377885 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.377784    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ea09eb69-0c82-481a-bec4-b165ab890254-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.377885 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.377801    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.377885 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.377825    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.377885 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.377874    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.378180 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.377929    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4rqq\" (UniqueName: \"kubernetes.io/projected/ea09eb69-0c82-481a-bec4-b165ab890254-kube-api-access-x4rqq\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.378180 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.377984    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.378180 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.378011    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-web-config\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.378180 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.378042    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.378180 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.378067    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.378180 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.378106    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ea09eb69-0c82-481a-bec4-b165ab890254-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.378180 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.378137    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.378180 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.378178    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.378448 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.378196    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.378448 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.378212    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.378448 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.378236    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.378448 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.378257    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ea09eb69-0c82-481a-bec4-b165ab890254-config-out\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.378448 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.378298    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.479898 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.479838    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ea09eb69-0c82-481a-bec4-b165ab890254-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.480075 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.479917    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.480075 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.479977    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.480075 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.480003    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.480075 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.480030    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.480075 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.480058    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.480278 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.480085    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ea09eb69-0c82-481a-bec4-b165ab890254-config-out\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.480278 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.480115    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.480278 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.480168    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.480278 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.480211    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.480278 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.480242    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-config\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.480278 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.480266    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ea09eb69-0c82-481a-bec4-b165ab890254-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.480499 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.480288    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.480499 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.480315    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.480499 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.480321    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ea09eb69-0c82-481a-bec4-b165ab890254-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.480499 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.480346    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.480499 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.480375    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4rqq\" (UniqueName: \"kubernetes.io/projected/ea09eb69-0c82-481a-bec4-b165ab890254-kube-api-access-x4rqq\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.480499 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.480417    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.480499 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.480440    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-web-config\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.480499 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.480469    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.480499 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.480494    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.480895 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.480583    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.480895 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.480773    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.481626 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.481110    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.482034 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.482009    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.482872 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.482831    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.483488 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.483462    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.484376 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.484351    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.485913 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.485888    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.486019 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.485917    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.486356 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.486331    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.487656 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.487635    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ea09eb69-0c82-481a-bec4-b165ab890254-config-out\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.488029 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.487987    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.488276 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.488253    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.488368 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.488339    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-web-config\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.488947 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.488924    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-config\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.489063 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.489039    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.489830 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.489805    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ea09eb69-0c82-481a-bec4-b165ab890254-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.490644 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.490575    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.490738 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.490718    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4rqq\" (UniqueName: \"kubernetes.io/projected/ea09eb69-0c82-481a-bec4-b165ab890254-kube-api-access-x4rqq\") pod \"prometheus-k8s-0\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.568773 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.568739    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:28.790958 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:28.790829    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 06:27:28.799914 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:27:28.799836    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea09eb69_0c82_481a_bec4_b165ab890254.slice/crio-e7674e90bc99b52325da61941f1d7d10a4003c1beba1aca7637062ca08bc6432 WatchSource:0}: Error finding container e7674e90bc99b52325da61941f1d7d10a4003c1beba1aca7637062ca08bc6432: Status 404 returned error can't find the container with id e7674e90bc99b52325da61941f1d7d10a4003c1beba1aca7637062ca08bc6432
Jul 29 06:27:29.248038 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:29.247943    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vt5qk" event={"ID":"3fec1db3-1218-4a65-b5c1-1e7473869742","Type":"ContainerStarted","Data":"b861fb9afd4ce41070b346c3b20ee27cbbf8dcd9175039b3c586873cfa50bac1"}
Jul 29 06:27:29.249410 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:29.249388    2563 generic.go:356] "Generic (PLEG): container finished" podID="09946786-61d2-4a1f-ad32-718acd329950" containerID="34f36dbdfec1f9327770b8e91aab56e04c9b5ceefd5c429d849edf5151b92c2d" exitCode=0
Jul 29 06:27:29.249520 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:29.249454    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"09946786-61d2-4a1f-ad32-718acd329950","Type":"ContainerDied","Data":"34f36dbdfec1f9327770b8e91aab56e04c9b5ceefd5c429d849edf5151b92c2d"}
Jul 29 06:27:29.251157 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:29.251105    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-mrklz" event={"ID":"37763295-c6d9-4846-8891-f4c5b78ec095","Type":"ContainerStarted","Data":"dd55bdbd49a82707e0e555c035c9b8fd5e9ee1addf1692e03c3d2715582ab0dc"}
Jul 29 06:27:29.253739 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:29.253715    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dcwzq" event={"ID":"46f8ec36-334c-400b-aa93-b57679124eaf","Type":"ContainerStarted","Data":"5702a5bf791376e7311094da95a77673d3b9117cc8875905e766488c63b89ea5"}
Jul 29 06:27:29.255746 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:29.255723    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8nmt6" event={"ID":"1d2702bd-5995-4c4d-b90a-dc7e40df640d","Type":"ContainerStarted","Data":"168e179d1cb4ad33b840d7a72ce595ebd1fdf2bfdcd82cdba02abd5aa0d076fd"}
Jul 29 06:27:29.255830 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:29.255770    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8nmt6" event={"ID":"1d2702bd-5995-4c4d-b90a-dc7e40df640d","Type":"ContainerStarted","Data":"994bf9a90f31c6d113b10bf8d2da03000a6b9a1bf2cb0cc7ff5955558e58d55c"}
Jul 29 06:27:29.257787 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:29.257759    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x7d4r" event={"ID":"8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8","Type":"ContainerStarted","Data":"6d24c0f4f210f580c3591858aecf73ae1d22cec3d244d9181b91f901795010d3"}
Jul 29 06:27:29.257896 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:29.257804    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x7d4r" event={"ID":"8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8","Type":"ContainerStarted","Data":"875df0f808ae5f15bad810c01de1c7f3d20d2f9400c47969d5bfb247364ba1df"}
Jul 29 06:27:29.257896 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:29.257826    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x7d4r" event={"ID":"8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8","Type":"ContainerStarted","Data":"efba26e9d11edb05c53090695e7fb4f6417a4491f8ee47b0297b246e71a03dcc"}
Jul 29 06:27:29.259218 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:29.259196    2563 generic.go:356] "Generic (PLEG): container finished" podID="ea09eb69-0c82-481a-bec4-b165ab890254" containerID="a98353e74d92fe810caef4c77b1dfde237ba03cdb9444262e9aa9a0eb1582b0e" exitCode=0
Jul 29 06:27:29.259313 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:29.259275    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ea09eb69-0c82-481a-bec4-b165ab890254","Type":"ContainerDied","Data":"a98353e74d92fe810caef4c77b1dfde237ba03cdb9444262e9aa9a0eb1582b0e"}
Jul 29 06:27:29.259364 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:29.259329    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ea09eb69-0c82-481a-bec4-b165ab890254","Type":"ContainerStarted","Data":"e7674e90bc99b52325da61941f1d7d10a4003c1beba1aca7637062ca08bc6432"}
Jul 29 06:27:29.260940 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:29.260923    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bd28d" event={"ID":"4f4e0417-82ca-4f78-9043-85dc8af1d71b","Type":"ContainerStarted","Data":"4974734d8200dda54e7097d742acde6bda5a6ae8a4596374a10ee4b00f2c7f9e"}
Jul 29 06:27:29.261009 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:29.260960    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bd28d" event={"ID":"4f4e0417-82ca-4f78-9043-85dc8af1d71b","Type":"ContainerStarted","Data":"24478d4825e2775a77245b2b238500b7e57998343f412df4d116de4ed153f0f4"}
Jul 29 06:27:29.261009 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:29.260994    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bd28d"
Jul 29 06:27:29.263186 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:29.263059    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vt5qk" podStartSLOduration=33.428143009 podStartE2EDuration="47.263041861s" totalImagesPullingTime="13.834898852s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 06:26:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:27:29.262432303 +0000 UTC m=+292.340064075" watchObservedRunningTime="2026-07-29 06:27:29.263041861 +0000 UTC m=+292.340673632"
Jul 29 06:27:29.277120 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:29.277068    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8nmt6" podStartSLOduration=6.044882385 podStartE2EDuration="7.277054338s" totalImagesPullingTime="1.080403175s" totalInitContainerRuntime="151.768778ms" isStatefulPod=true podCreationTimestamp="2026-07-29 06:27:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:27:29.276538065 +0000 UTC m=+292.354169832" watchObservedRunningTime="2026-07-29 06:27:29.277054338 +0000 UTC m=+292.354686105"
Jul 29 06:27:29.311327 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:29.311288    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dcwzq" podStartSLOduration=2.601834969 podStartE2EDuration="7.311276389s" totalImagesPullingTime="4.70944142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:27:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:27:29.31052389 +0000 UTC m=+292.388155658" watchObservedRunningTime="2026-07-29 06:27:29.311276389 +0000 UTC m=+292.388908133"
Jul 29 06:27:29.345796 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:29.345760    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bd28d" podStartSLOduration=253.753051826 podStartE2EDuration="4m18.345747463s" totalImagesPullingTime="4.592695637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:23:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:27:29.344786101 +0000 UTC m=+292.422417868" watchObservedRunningTime="2026-07-29 06:27:29.345747463 +0000 UTC m=+292.423379234"
Jul 29 06:27:29.359460 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:29.359420    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-x7d4r" podStartSLOduration=2.5256361529999998 podStartE2EDuration="7.359408803s" totalImagesPullingTime="4.83377265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:27:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:27:29.358437564 +0000 UTC m=+292.436069332" watchObservedRunningTime="2026-07-29 06:27:29.359408803 +0000 UTC m=+292.437040594"
Jul 29 06:27:29.372239 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:29.372203    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-mrklz" podStartSLOduration=277.532685131 podStartE2EDuration="4m42.372190072s" totalImagesPullingTime="4.839504941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:27:29.3714406 +0000 UTC m=+292.449072399" watchObservedRunningTime="2026-07-29 06:27:29.372190072 +0000 UTC m=+292.449821841"
Jul 29 06:27:31.274011 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:31.273982    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"09946786-61d2-4a1f-ad32-718acd329950","Type":"ContainerStarted","Data":"ff0ab7acbef87d768faffb5bb25e75dccf53e0d67b6acb9c6bb8028e563d9c82"}
Jul 29 06:27:31.274346 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:31.274101    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"09946786-61d2-4a1f-ad32-718acd329950","Type":"ContainerStarted","Data":"9ff959711759a69efe17447c028ae977fabcd5178a6cb8874bc1884a21eeab73"}
Jul 29 06:27:31.274346 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:31.274145    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"09946786-61d2-4a1f-ad32-718acd329950","Type":"ContainerStarted","Data":"1d302fc8a84e7409f254cccd5d44db93be057509af8cebecde68979abd6d60f2"}
Jul 29 06:27:31.393090 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:27:31.393051    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-vcg74" podUID="2ceff9c5-890d-4d2b-85ee-09bcd9e02833"
Jul 29 06:27:32.282995 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:32.282956    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"09946786-61d2-4a1f-ad32-718acd329950","Type":"ContainerStarted","Data":"ad833352117bf8166db88967c536686405cd11c6cd6d5d0567b0c203b5229e2b"}
Jul 29 06:27:32.283377 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:32.283045    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"09946786-61d2-4a1f-ad32-718acd329950","Type":"ContainerStarted","Data":"a6b4847a961f7dd7a94ffd4f32aa955738d03aa81a9477769c6d3e3b97734111"}
Jul 29 06:27:33.290375 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:33.290338    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ea09eb69-0c82-481a-bec4-b165ab890254","Type":"ContainerStarted","Data":"cff13042b047413f93648d5aba80e737702e289682ccf844d0c87214b14d11af"}
Jul 29 06:27:33.290790 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:33.290418    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ea09eb69-0c82-481a-bec4-b165ab890254","Type":"ContainerStarted","Data":"9ffd9d668bb6d963817841ab645005feb14c09ed74a8c5746f509257cf403dd8"}
Jul 29 06:27:33.294227 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:33.294199    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"09946786-61d2-4a1f-ad32-718acd329950","Type":"ContainerStarted","Data":"6c15d173b0d760eb6fd9879aaa82b9974ecb21f4b1d7e4c84990649054299e64"}
Jul 29 06:27:33.319668 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:33.319623    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.243600696 podStartE2EDuration="10.31960438s" totalImagesPullingTime="8.056637022s" totalInitContainerRuntime="19.366662ms" isStatefulPod=false podCreationTimestamp="2026-07-29 06:27:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:27:33.316923555 +0000 UTC m=+296.394555325" watchObservedRunningTime="2026-07-29 06:27:33.31960438 +0000 UTC m=+296.397236147"
Jul 29 06:27:35.307934 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:35.307902    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ea09eb69-0c82-481a-bec4-b165ab890254","Type":"ContainerStarted","Data":"114cf2febc734b18fd600224331c609521b5c3a8a97315238afc5b53c818c7e6"}
Jul 29 06:27:35.308287 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:35.307989    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ea09eb69-0c82-481a-bec4-b165ab890254","Type":"ContainerStarted","Data":"7d779a9dfda4956e874922e7b72b9c8636475d04fa3959d7ffdc0b10cfe035b5"}
Jul 29 06:27:35.308287 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:35.308023    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ea09eb69-0c82-481a-bec4-b165ab890254","Type":"ContainerStarted","Data":"05faf0ff87e5e838859994cbf6a313e16089c5bc719ae6b0063b28f359ef87af"}
Jul 29 06:27:35.308287 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:35.308049    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ea09eb69-0c82-481a-bec4-b165ab890254","Type":"ContainerStarted","Data":"45e301eb6e26bc5f061aa0d3e05df1b83a200ac72b8b80999d914ee68f4822c1"}
Jul 29 06:27:35.334527 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:35.334479    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.214019563 podStartE2EDuration="7.334463163s" totalImagesPullingTime="5.078841162s" totalInitContainerRuntime="41.602438ms" isStatefulPod=false podCreationTimestamp="2026-07-29 06:27:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:27:35.332500579 +0000 UTC m=+298.410132389" watchObservedRunningTime="2026-07-29 06:27:35.334463163 +0000 UTC m=+298.412094930"
Jul 29 06:27:37.284338 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:37.284310    2563 kubelet.go:1648] "Image garbage collection succeeded"
Jul 29 06:27:38.569473 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:38.569432    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:27:39.269220 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:39.269193    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bd28d"
Jul 29 06:27:42.088822 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:42.088781    2563 patch_prober.go:28] interesting pod/image-registry-784bf54c5f-hj6jj container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 29 06:27:42.089224 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:42.088832    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj" podUID="e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 06:27:42.391299 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:42.391220    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:27:45.227511 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:45.227477    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:27:56.652163 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:27:56.652127    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-784bf54c5f-hj6jj"]
Jul 29 06:28:07.431991 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:07.431960    2563 generic.go:356] "Generic (PLEG): container finished" podID="526ca1b2-bbc3-41fc-bda0-6754c7e98e37" containerID="f93b9ad8d7af0ed5935635a07313da490ca4a3390d994125f501a4f9e271fc82" exitCode=0
Jul 29 06:28:07.432379 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:07.432040    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-npz5n" event={"ID":"526ca1b2-bbc3-41fc-bda0-6754c7e98e37","Type":"ContainerDied","Data":"f93b9ad8d7af0ed5935635a07313da490ca4a3390d994125f501a4f9e271fc82"}
Jul 29 06:28:07.432427 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:07.432380    2563 scope.go:117] "RemoveContainer" containerID="f93b9ad8d7af0ed5935635a07313da490ca4a3390d994125f501a4f9e271fc82"
Jul 29 06:28:08.439723 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:08.439685    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-npz5n" event={"ID":"526ca1b2-bbc3-41fc-bda0-6754c7e98e37","Type":"ContainerStarted","Data":"d473cb7c169e3207cfb85b7c55702f77ee3122f876953c1459c20fdd738bad27"}
Jul 29 06:28:12.456350 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:12.456313    2563 generic.go:356] "Generic (PLEG): container finished" podID="cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4" containerID="17192d10ea2a0780ba298350a0f77aa35f084e074069678045f9751fa9ea2082" exitCode=0
Jul 29 06:28:12.456769 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:12.456355    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-mr497" event={"ID":"cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4","Type":"ContainerDied","Data":"17192d10ea2a0780ba298350a0f77aa35f084e074069678045f9751fa9ea2082"}
Jul 29 06:28:12.456769 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:12.456667    2563 scope.go:117] "RemoveContainer" containerID="17192d10ea2a0780ba298350a0f77aa35f084e074069678045f9751fa9ea2082"
Jul 29 06:28:13.088806 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:13.088774    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4pk46_2f05bf26-3930-4e41-ad15-78055946eb87/serve-healthcheck-canary/0.log"
Jul 29 06:28:13.462291 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:13.462207    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-mr497" event={"ID":"cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4","Type":"ContainerStarted","Data":"42e5ac058051bcab96adf86cc9166a8e72f87862fa1803da6489bd0701285a11"}
Jul 29 06:28:21.671625 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:21.671568    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj" podUID="e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48" containerName="registry" containerID="cri-o://f41c47905180ef7c294eb0fa95949c47381e4c3bd646a655bb83a60c4226e26c" gracePeriod=30
Jul 29 06:28:22.913401 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:22.913379    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:28:23.074593 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:23.074567    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-installation-pull-secrets\") pod \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") "
Jul 29 06:28:23.074734 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:23.074604    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-bound-sa-token\") pod \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") "
Jul 29 06:28:23.074734 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:23.074622    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-ca-trust-extracted\") pod \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") "
Jul 29 06:28:23.074734 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:23.074661    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-trusted-ca\") pod \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") "
Jul 29 06:28:23.074734 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:23.074676    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-tls\") pod \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") "
Jul 29 06:28:23.074963 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:23.074745    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-image-registry-private-configuration\") pod \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") "
Jul 29 06:28:23.074963 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:23.074779    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9zxq\" (UniqueName: \"kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-kube-api-access-v9zxq\") pod \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") "
Jul 29 06:28:23.074963 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:23.074806    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-certificates\") pod \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\" (UID: \"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48\") "
Jul 29 06:28:23.075325 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:23.075175    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48" (UID: "e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 06:28:23.075443 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:23.075353    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48" (UID: "e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 06:28:23.077644 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:23.077613    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48" (UID: "e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 06:28:23.077746 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:23.077660    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48" (UID: "e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 06:28:23.077809 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:23.077749    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48" (UID: "e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 06:28:23.077809 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:23.077775    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48" (UID: "e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 06:28:23.077809 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:23.077795    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-kube-api-access-v9zxq" (OuterVolumeSpecName: "kube-api-access-v9zxq") pod "e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48" (UID: "e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48"). InnerVolumeSpecName "kube-api-access-v9zxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 06:28:23.082950 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:23.082928    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48" (UID: "e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 06:28:23.176288 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:23.176248    2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-image-registry-private-configuration\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:23.176288 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:23.176285    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9zxq\" (UniqueName: \"kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-kube-api-access-v9zxq\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:23.176288 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:23.176297    2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-certificates\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:23.176453 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:23.176306    2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-installation-pull-secrets\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:23.176453 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:23.176315    2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-bound-sa-token\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:23.176453 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:23.176323    2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-ca-trust-extracted\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:23.176453 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:23.176333    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-trusted-ca\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:23.176453 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:23.176342    2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48-registry-tls\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:23.501138 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:23.501105    2563 generic.go:356] "Generic (PLEG): container finished" podID="e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48" containerID="f41c47905180ef7c294eb0fa95949c47381e4c3bd646a655bb83a60c4226e26c" exitCode=0
Jul 29 06:28:23.501318 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:23.501152    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj"
Jul 29 06:28:23.501318 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:23.501157    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj" event={"ID":"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48","Type":"ContainerDied","Data":"f41c47905180ef7c294eb0fa95949c47381e4c3bd646a655bb83a60c4226e26c"}
Jul 29 06:28:23.501318 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:23.501216    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-784bf54c5f-hj6jj" event={"ID":"e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48","Type":"ContainerDied","Data":"5568c8466674685e9693fd5208ea289bbc3abbce095a7fb8579db3cd4431ecb9"}
Jul 29 06:28:23.502080 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:23.502060    2563 scope.go:117] "RemoveContainer" containerID="f41c47905180ef7c294eb0fa95949c47381e4c3bd646a655bb83a60c4226e26c"
Jul 29 06:28:23.511602 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:23.511584    2563 scope.go:117] "RemoveContainer" containerID="f41c47905180ef7c294eb0fa95949c47381e4c3bd646a655bb83a60c4226e26c"
Jul 29 06:28:23.512107 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:28:23.512064    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f41c47905180ef7c294eb0fa95949c47381e4c3bd646a655bb83a60c4226e26c\": container with ID starting with f41c47905180ef7c294eb0fa95949c47381e4c3bd646a655bb83a60c4226e26c not found: ID does not exist" containerID="f41c47905180ef7c294eb0fa95949c47381e4c3bd646a655bb83a60c4226e26c"
Jul 29 06:28:23.512208 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:23.512120    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f41c47905180ef7c294eb0fa95949c47381e4c3bd646a655bb83a60c4226e26c"} err="failed to get container status \"f41c47905180ef7c294eb0fa95949c47381e4c3bd646a655bb83a60c4226e26c\": rpc error: code = NotFound desc = could not find container \"f41c47905180ef7c294eb0fa95949c47381e4c3bd646a655bb83a60c4226e26c\": container with ID starting with f41c47905180ef7c294eb0fa95949c47381e4c3bd646a655bb83a60c4226e26c not found: ID does not exist"
Jul 29 06:28:23.516144 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:23.516124    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-784bf54c5f-hj6jj"]
Jul 29 06:28:23.519844 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:23.519823    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-784bf54c5f-hj6jj"]
Jul 29 06:28:25.397864 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:25.397817    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48" path="/var/lib/kubelet/pods/e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48/volumes"
Jul 29 06:28:28.569001 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:28.568915    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:28.588950 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:28.588926    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:29.549360 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:29.549333    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:42.286837 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:42.286787    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 06:28:42.287290 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:42.287123    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="09946786-61d2-4a1f-ad32-718acd329950" containerName="alertmanager" containerID="cri-o://1d302fc8a84e7409f254cccd5d44db93be057509af8cebecde68979abd6d60f2" gracePeriod=120
Jul 29 06:28:42.287290 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:42.287183    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="09946786-61d2-4a1f-ad32-718acd329950" containerName="kube-rbac-proxy-metric" containerID="cri-o://ad833352117bf8166db88967c536686405cd11c6cd6d5d0567b0c203b5229e2b" gracePeriod=120
Jul 29 06:28:42.287290 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:42.287197    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="09946786-61d2-4a1f-ad32-718acd329950" containerName="prom-label-proxy" containerID="cri-o://6c15d173b0d760eb6fd9879aaa82b9974ecb21f4b1d7e4c84990649054299e64" gracePeriod=120
Jul 29 06:28:42.287290 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:42.287227    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="09946786-61d2-4a1f-ad32-718acd329950" containerName="kube-rbac-proxy-web" containerID="cri-o://ff0ab7acbef87d768faffb5bb25e75dccf53e0d67b6acb9c6bb8028e563d9c82" gracePeriod=120
Jul 29 06:28:42.287589 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:42.287318    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="09946786-61d2-4a1f-ad32-718acd329950" containerName="kube-rbac-proxy" containerID="cri-o://a6b4847a961f7dd7a94ffd4f32aa955738d03aa81a9477769c6d3e3b97734111" gracePeriod=120
Jul 29 06:28:42.287589 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:42.287321    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="09946786-61d2-4a1f-ad32-718acd329950" containerName="config-reloader" containerID="cri-o://9ff959711759a69efe17447c028ae977fabcd5178a6cb8874bc1884a21eeab73" gracePeriod=120
Jul 29 06:28:42.585824 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:42.585798    2563 generic.go:356] "Generic (PLEG): container finished" podID="09946786-61d2-4a1f-ad32-718acd329950" containerID="6c15d173b0d760eb6fd9879aaa82b9974ecb21f4b1d7e4c84990649054299e64" exitCode=0
Jul 29 06:28:42.585824 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:42.585821    2563 generic.go:356] "Generic (PLEG): container finished" podID="09946786-61d2-4a1f-ad32-718acd329950" containerID="ad833352117bf8166db88967c536686405cd11c6cd6d5d0567b0c203b5229e2b" exitCode=0
Jul 29 06:28:42.585824 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:42.585827    2563 generic.go:356] "Generic (PLEG): container finished" podID="09946786-61d2-4a1f-ad32-718acd329950" containerID="a6b4847a961f7dd7a94ffd4f32aa955738d03aa81a9477769c6d3e3b97734111" exitCode=0
Jul 29 06:28:42.586066 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:42.585833    2563 generic.go:356] "Generic (PLEG): container finished" podID="09946786-61d2-4a1f-ad32-718acd329950" containerID="9ff959711759a69efe17447c028ae977fabcd5178a6cb8874bc1884a21eeab73" exitCode=0
Jul 29 06:28:42.586066 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:42.585838    2563 generic.go:356] "Generic (PLEG): container finished" podID="09946786-61d2-4a1f-ad32-718acd329950" containerID="1d302fc8a84e7409f254cccd5d44db93be057509af8cebecde68979abd6d60f2" exitCode=0
Jul 29 06:28:42.586066 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:42.585880    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"09946786-61d2-4a1f-ad32-718acd329950","Type":"ContainerDied","Data":"6c15d173b0d760eb6fd9879aaa82b9974ecb21f4b1d7e4c84990649054299e64"}
Jul 29 06:28:42.586066 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:42.585942    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"09946786-61d2-4a1f-ad32-718acd329950","Type":"ContainerDied","Data":"ad833352117bf8166db88967c536686405cd11c6cd6d5d0567b0c203b5229e2b"}
Jul 29 06:28:42.586066 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:42.585970    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"09946786-61d2-4a1f-ad32-718acd329950","Type":"ContainerDied","Data":"a6b4847a961f7dd7a94ffd4f32aa955738d03aa81a9477769c6d3e3b97734111"}
Jul 29 06:28:42.586066 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:42.585994    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"09946786-61d2-4a1f-ad32-718acd329950","Type":"ContainerDied","Data":"9ff959711759a69efe17447c028ae977fabcd5178a6cb8874bc1884a21eeab73"}
Jul 29 06:28:42.586066 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:42.586018    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"09946786-61d2-4a1f-ad32-718acd329950","Type":"ContainerDied","Data":"1d302fc8a84e7409f254cccd5d44db93be057509af8cebecde68979abd6d60f2"}
Jul 29 06:28:43.540704 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.540672    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:43.593450 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.593415    2563 generic.go:356] "Generic (PLEG): container finished" podID="09946786-61d2-4a1f-ad32-718acd329950" containerID="ff0ab7acbef87d768faffb5bb25e75dccf53e0d67b6acb9c6bb8028e563d9c82" exitCode=0
Jul 29 06:28:43.593638 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.593499    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:43.593638 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.593497    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"09946786-61d2-4a1f-ad32-718acd329950","Type":"ContainerDied","Data":"ff0ab7acbef87d768faffb5bb25e75dccf53e0d67b6acb9c6bb8028e563d9c82"}
Jul 29 06:28:43.593638 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.593568    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"09946786-61d2-4a1f-ad32-718acd329950","Type":"ContainerDied","Data":"f69ce91c388213741fb76245b0da7acfb6f17df49a060d0cd427094c5ee6b30c"}
Jul 29 06:28:43.593638 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.593606    2563 scope.go:117] "RemoveContainer" containerID="6c15d173b0d760eb6fd9879aaa82b9974ecb21f4b1d7e4c84990649054299e64"
Jul 29 06:28:43.602301 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.602286    2563 scope.go:117] "RemoveContainer" containerID="ad833352117bf8166db88967c536686405cd11c6cd6d5d0567b0c203b5229e2b"
Jul 29 06:28:43.609165 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.609144    2563 scope.go:117] "RemoveContainer" containerID="a6b4847a961f7dd7a94ffd4f32aa955738d03aa81a9477769c6d3e3b97734111"
Jul 29 06:28:43.616045 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.616021    2563 scope.go:117] "RemoveContainer" containerID="ff0ab7acbef87d768faffb5bb25e75dccf53e0d67b6acb9c6bb8028e563d9c82"
Jul 29 06:28:43.622688 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.622674    2563 scope.go:117] "RemoveContainer" containerID="9ff959711759a69efe17447c028ae977fabcd5178a6cb8874bc1884a21eeab73"
Jul 29 06:28:43.629090 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.629071    2563 scope.go:117] "RemoveContainer" containerID="1d302fc8a84e7409f254cccd5d44db93be057509af8cebecde68979abd6d60f2"
Jul 29 06:28:43.635628 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.635610    2563 scope.go:117] "RemoveContainer" containerID="34f36dbdfec1f9327770b8e91aab56e04c9b5ceefd5c429d849edf5151b92c2d"
Jul 29 06:28:43.642134 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.642121    2563 scope.go:117] "RemoveContainer" containerID="6c15d173b0d760eb6fd9879aaa82b9974ecb21f4b1d7e4c84990649054299e64"
Jul 29 06:28:43.642373 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:28:43.642355    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c15d173b0d760eb6fd9879aaa82b9974ecb21f4b1d7e4c84990649054299e64\": container with ID starting with 6c15d173b0d760eb6fd9879aaa82b9974ecb21f4b1d7e4c84990649054299e64 not found: ID does not exist" containerID="6c15d173b0d760eb6fd9879aaa82b9974ecb21f4b1d7e4c84990649054299e64"
Jul 29 06:28:43.642416 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.642381    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c15d173b0d760eb6fd9879aaa82b9974ecb21f4b1d7e4c84990649054299e64"} err="failed to get container status \"6c15d173b0d760eb6fd9879aaa82b9974ecb21f4b1d7e4c84990649054299e64\": rpc error: code = NotFound desc = could not find container \"6c15d173b0d760eb6fd9879aaa82b9974ecb21f4b1d7e4c84990649054299e64\": container with ID starting with 6c15d173b0d760eb6fd9879aaa82b9974ecb21f4b1d7e4c84990649054299e64 not found: ID does not exist"
Jul 29 06:28:43.642416 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.642401    2563 scope.go:117] "RemoveContainer" containerID="ad833352117bf8166db88967c536686405cd11c6cd6d5d0567b0c203b5229e2b"
Jul 29 06:28:43.642625 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:28:43.642610    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad833352117bf8166db88967c536686405cd11c6cd6d5d0567b0c203b5229e2b\": container with ID starting with ad833352117bf8166db88967c536686405cd11c6cd6d5d0567b0c203b5229e2b not found: ID does not exist" containerID="ad833352117bf8166db88967c536686405cd11c6cd6d5d0567b0c203b5229e2b"
Jul 29 06:28:43.642672 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.642632    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad833352117bf8166db88967c536686405cd11c6cd6d5d0567b0c203b5229e2b"} err="failed to get container status \"ad833352117bf8166db88967c536686405cd11c6cd6d5d0567b0c203b5229e2b\": rpc error: code = NotFound desc = could not find container \"ad833352117bf8166db88967c536686405cd11c6cd6d5d0567b0c203b5229e2b\": container with ID starting with ad833352117bf8166db88967c536686405cd11c6cd6d5d0567b0c203b5229e2b not found: ID does not exist"
Jul 29 06:28:43.642672 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.642647    2563 scope.go:117] "RemoveContainer" containerID="a6b4847a961f7dd7a94ffd4f32aa955738d03aa81a9477769c6d3e3b97734111"
Jul 29 06:28:43.642912 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:28:43.642896    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6b4847a961f7dd7a94ffd4f32aa955738d03aa81a9477769c6d3e3b97734111\": container with ID starting with a6b4847a961f7dd7a94ffd4f32aa955738d03aa81a9477769c6d3e3b97734111 not found: ID does not exist" containerID="a6b4847a961f7dd7a94ffd4f32aa955738d03aa81a9477769c6d3e3b97734111"
Jul 29 06:28:43.642960 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.642917    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6b4847a961f7dd7a94ffd4f32aa955738d03aa81a9477769c6d3e3b97734111"} err="failed to get container status \"a6b4847a961f7dd7a94ffd4f32aa955738d03aa81a9477769c6d3e3b97734111\": rpc error: code = NotFound desc = could not find container \"a6b4847a961f7dd7a94ffd4f32aa955738d03aa81a9477769c6d3e3b97734111\": container with ID starting with a6b4847a961f7dd7a94ffd4f32aa955738d03aa81a9477769c6d3e3b97734111 not found: ID does not exist"
Jul 29 06:28:43.642960 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.642931    2563 scope.go:117] "RemoveContainer" containerID="ff0ab7acbef87d768faffb5bb25e75dccf53e0d67b6acb9c6bb8028e563d9c82"
Jul 29 06:28:43.643161 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:28:43.643144    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff0ab7acbef87d768faffb5bb25e75dccf53e0d67b6acb9c6bb8028e563d9c82\": container with ID starting with ff0ab7acbef87d768faffb5bb25e75dccf53e0d67b6acb9c6bb8028e563d9c82 not found: ID does not exist" containerID="ff0ab7acbef87d768faffb5bb25e75dccf53e0d67b6acb9c6bb8028e563d9c82"
Jul 29 06:28:43.643220 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.643169    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff0ab7acbef87d768faffb5bb25e75dccf53e0d67b6acb9c6bb8028e563d9c82"} err="failed to get container status \"ff0ab7acbef87d768faffb5bb25e75dccf53e0d67b6acb9c6bb8028e563d9c82\": rpc error: code = NotFound desc = could not find container \"ff0ab7acbef87d768faffb5bb25e75dccf53e0d67b6acb9c6bb8028e563d9c82\": container with ID starting with ff0ab7acbef87d768faffb5bb25e75dccf53e0d67b6acb9c6bb8028e563d9c82 not found: ID does not exist"
Jul 29 06:28:43.643220 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.643191    2563 scope.go:117] "RemoveContainer" containerID="9ff959711759a69efe17447c028ae977fabcd5178a6cb8874bc1884a21eeab73"
Jul 29 06:28:43.643427 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:28:43.643412    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ff959711759a69efe17447c028ae977fabcd5178a6cb8874bc1884a21eeab73\": container with ID starting with 9ff959711759a69efe17447c028ae977fabcd5178a6cb8874bc1884a21eeab73 not found: ID does not exist" containerID="9ff959711759a69efe17447c028ae977fabcd5178a6cb8874bc1884a21eeab73"
Jul 29 06:28:43.643464 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.643433    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ff959711759a69efe17447c028ae977fabcd5178a6cb8874bc1884a21eeab73"} err="failed to get container status \"9ff959711759a69efe17447c028ae977fabcd5178a6cb8874bc1884a21eeab73\": rpc error: code = NotFound desc = could not find container \"9ff959711759a69efe17447c028ae977fabcd5178a6cb8874bc1884a21eeab73\": container with ID starting with 9ff959711759a69efe17447c028ae977fabcd5178a6cb8874bc1884a21eeab73 not found: ID does not exist"
Jul 29 06:28:43.643464 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.643447    2563 scope.go:117] "RemoveContainer" containerID="1d302fc8a84e7409f254cccd5d44db93be057509af8cebecde68979abd6d60f2"
Jul 29 06:28:43.643658 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:28:43.643642    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d302fc8a84e7409f254cccd5d44db93be057509af8cebecde68979abd6d60f2\": container with ID starting with 1d302fc8a84e7409f254cccd5d44db93be057509af8cebecde68979abd6d60f2 not found: ID does not exist" containerID="1d302fc8a84e7409f254cccd5d44db93be057509af8cebecde68979abd6d60f2"
Jul 29 06:28:43.643701 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.643663    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d302fc8a84e7409f254cccd5d44db93be057509af8cebecde68979abd6d60f2"} err="failed to get container status \"1d302fc8a84e7409f254cccd5d44db93be057509af8cebecde68979abd6d60f2\": rpc error: code = NotFound desc = could not find container \"1d302fc8a84e7409f254cccd5d44db93be057509af8cebecde68979abd6d60f2\": container with ID starting with 1d302fc8a84e7409f254cccd5d44db93be057509af8cebecde68979abd6d60f2 not found: ID does not exist"
Jul 29 06:28:43.643701 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.643676    2563 scope.go:117] "RemoveContainer" containerID="34f36dbdfec1f9327770b8e91aab56e04c9b5ceefd5c429d849edf5151b92c2d"
Jul 29 06:28:43.643883 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:28:43.643865    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34f36dbdfec1f9327770b8e91aab56e04c9b5ceefd5c429d849edf5151b92c2d\": container with ID starting with 34f36dbdfec1f9327770b8e91aab56e04c9b5ceefd5c429d849edf5151b92c2d not found: ID does not exist" containerID="34f36dbdfec1f9327770b8e91aab56e04c9b5ceefd5c429d849edf5151b92c2d"
Jul 29 06:28:43.643927 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.643888    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34f36dbdfec1f9327770b8e91aab56e04c9b5ceefd5c429d849edf5151b92c2d"} err="failed to get container status \"34f36dbdfec1f9327770b8e91aab56e04c9b5ceefd5c429d849edf5151b92c2d\": rpc error: code = NotFound desc = could not find container \"34f36dbdfec1f9327770b8e91aab56e04c9b5ceefd5c429d849edf5151b92c2d\": container with ID starting with 34f36dbdfec1f9327770b8e91aab56e04c9b5ceefd5c429d849edf5151b92c2d not found: ID does not exist"
Jul 29 06:28:43.654159 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.654136    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-cluster-tls-config\") pod \"09946786-61d2-4a1f-ad32-718acd329950\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") "
Jul 29 06:28:43.654245 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.654163    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-secret-alertmanager-kube-rbac-proxy\") pod \"09946786-61d2-4a1f-ad32-718acd329950\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") "
Jul 29 06:28:43.654245 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.654185    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hvnw\" (UniqueName: \"kubernetes.io/projected/09946786-61d2-4a1f-ad32-718acd329950-kube-api-access-4hvnw\") pod \"09946786-61d2-4a1f-ad32-718acd329950\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") "
Jul 29 06:28:43.654245 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.654214    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-config-volume\") pod \"09946786-61d2-4a1f-ad32-718acd329950\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") "
Jul 29 06:28:43.654389 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.654260    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/09946786-61d2-4a1f-ad32-718acd329950-tls-assets\") pod \"09946786-61d2-4a1f-ad32-718acd329950\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") "
Jul 29 06:28:43.654389 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.654286    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09946786-61d2-4a1f-ad32-718acd329950-metrics-client-ca\") pod \"09946786-61d2-4a1f-ad32-718acd329950\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") "
Jul 29 06:28:43.654389 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.654315    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09946786-61d2-4a1f-ad32-718acd329950-alertmanager-trusted-ca-bundle\") pod \"09946786-61d2-4a1f-ad32-718acd329950\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") "
Jul 29 06:28:43.654389 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.654366    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/09946786-61d2-4a1f-ad32-718acd329950-config-out\") pod \"09946786-61d2-4a1f-ad32-718acd329950\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") "
Jul 29 06:28:43.654589 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.654406    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-secret-alertmanager-main-tls\") pod \"09946786-61d2-4a1f-ad32-718acd329950\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") "
Jul 29 06:28:43.654589 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.654437    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-secret-alertmanager-kube-rbac-proxy-metric\") pod \"09946786-61d2-4a1f-ad32-718acd329950\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") "
Jul 29 06:28:43.654589 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.654476    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-web-config\") pod \"09946786-61d2-4a1f-ad32-718acd329950\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") "
Jul 29 06:28:43.654589 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.654502    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-secret-alertmanager-kube-rbac-proxy-web\") pod \"09946786-61d2-4a1f-ad32-718acd329950\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") "
Jul 29 06:28:43.654589 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.654526    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/09946786-61d2-4a1f-ad32-718acd329950-alertmanager-main-db\") pod \"09946786-61d2-4a1f-ad32-718acd329950\" (UID: \"09946786-61d2-4a1f-ad32-718acd329950\") "
Jul 29 06:28:43.654976 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.654671    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09946786-61d2-4a1f-ad32-718acd329950-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "09946786-61d2-4a1f-ad32-718acd329950" (UID: "09946786-61d2-4a1f-ad32-718acd329950"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 06:28:43.654976 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.654978    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09946786-61d2-4a1f-ad32-718acd329950-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "09946786-61d2-4a1f-ad32-718acd329950" (UID: "09946786-61d2-4a1f-ad32-718acd329950"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 06:28:43.654976 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.654985    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09946786-61d2-4a1f-ad32-718acd329950-metrics-client-ca\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:43.655351 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.655322    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09946786-61d2-4a1f-ad32-718acd329950-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "09946786-61d2-4a1f-ad32-718acd329950" (UID: "09946786-61d2-4a1f-ad32-718acd329950"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 06:28:43.657688 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.657658    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "09946786-61d2-4a1f-ad32-718acd329950" (UID: "09946786-61d2-4a1f-ad32-718acd329950"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 06:28:43.657781 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.657756    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "09946786-61d2-4a1f-ad32-718acd329950" (UID: "09946786-61d2-4a1f-ad32-718acd329950"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 06:28:43.658046 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.658009    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "09946786-61d2-4a1f-ad32-718acd329950" (UID: "09946786-61d2-4a1f-ad32-718acd329950"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 06:28:43.658293 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.658261    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09946786-61d2-4a1f-ad32-718acd329950-config-out" (OuterVolumeSpecName: "config-out") pod "09946786-61d2-4a1f-ad32-718acd329950" (UID: "09946786-61d2-4a1f-ad32-718acd329950"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 06:28:43.658293 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.658278    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09946786-61d2-4a1f-ad32-718acd329950-kube-api-access-4hvnw" (OuterVolumeSpecName: "kube-api-access-4hvnw") pod "09946786-61d2-4a1f-ad32-718acd329950" (UID: "09946786-61d2-4a1f-ad32-718acd329950"). InnerVolumeSpecName "kube-api-access-4hvnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 06:28:43.658457 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.658352    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09946786-61d2-4a1f-ad32-718acd329950-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "09946786-61d2-4a1f-ad32-718acd329950" (UID: "09946786-61d2-4a1f-ad32-718acd329950"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 06:28:43.658513 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.658472    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "09946786-61d2-4a1f-ad32-718acd329950" (UID: "09946786-61d2-4a1f-ad32-718acd329950"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 06:28:43.658642 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.658624    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-config-volume" (OuterVolumeSpecName: "config-volume") pod "09946786-61d2-4a1f-ad32-718acd329950" (UID: "09946786-61d2-4a1f-ad32-718acd329950"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 06:28:43.661632 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.661613    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "09946786-61d2-4a1f-ad32-718acd329950" (UID: "09946786-61d2-4a1f-ad32-718acd329950"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 06:28:43.669331 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.669308    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-web-config" (OuterVolumeSpecName: "web-config") pod "09946786-61d2-4a1f-ad32-718acd329950" (UID: "09946786-61d2-4a1f-ad32-718acd329950"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 06:28:43.755692 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.755667    2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-web-config\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:43.755692 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.755688    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:43.755692 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.755699    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/09946786-61d2-4a1f-ad32-718acd329950-alertmanager-main-db\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:43.755941 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.755707    2563 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-cluster-tls-config\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:43.755941 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.755719    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:43.755941 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.755728    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hvnw\" (UniqueName: \"kubernetes.io/projected/09946786-61d2-4a1f-ad32-718acd329950-kube-api-access-4hvnw\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:43.755941 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.755736    2563 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-config-volume\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:43.755941 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.755746    2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/09946786-61d2-4a1f-ad32-718acd329950-tls-assets\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:43.755941 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.755753    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09946786-61d2-4a1f-ad32-718acd329950-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:43.755941 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.755771    2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/09946786-61d2-4a1f-ad32-718acd329950-config-out\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:43.755941 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.755781    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-secret-alertmanager-main-tls\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:43.755941 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.755790    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/09946786-61d2-4a1f-ad32-718acd329950-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:43.916504 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.916474    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 06:28:43.919945 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.919911    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 06:28:43.943530 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.943508    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 06:28:43.943833 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.943819    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09946786-61d2-4a1f-ad32-718acd329950" containerName="init-config-reloader"
Jul 29 06:28:43.943890 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.943835    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="09946786-61d2-4a1f-ad32-718acd329950" containerName="init-config-reloader"
Jul 29 06:28:43.943890 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.943865    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09946786-61d2-4a1f-ad32-718acd329950" containerName="kube-rbac-proxy-metric"
Jul 29 06:28:43.943890 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.943874    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="09946786-61d2-4a1f-ad32-718acd329950" containerName="kube-rbac-proxy-metric"
Jul 29 06:28:43.943890 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.943886    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09946786-61d2-4a1f-ad32-718acd329950" containerName="alertmanager"
Jul 29 06:28:43.944010 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.943892    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="09946786-61d2-4a1f-ad32-718acd329950" containerName="alertmanager"
Jul 29 06:28:43.944010 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.943900    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09946786-61d2-4a1f-ad32-718acd329950" containerName="kube-rbac-proxy-web"
Jul 29 06:28:43.944010 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.943905    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="09946786-61d2-4a1f-ad32-718acd329950" containerName="kube-rbac-proxy-web"
Jul 29 06:28:43.944010 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.943912    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09946786-61d2-4a1f-ad32-718acd329950" containerName="kube-rbac-proxy"
Jul 29 06:28:43.944010 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.943917    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="09946786-61d2-4a1f-ad32-718acd329950" containerName="kube-rbac-proxy"
Jul 29 06:28:43.944010 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.943923    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48" containerName="registry"
Jul 29 06:28:43.944010 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.943928    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48" containerName="registry"
Jul 29 06:28:43.944010 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.943932    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09946786-61d2-4a1f-ad32-718acd329950" containerName="prom-label-proxy"
Jul 29 06:28:43.944010 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.943937    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="09946786-61d2-4a1f-ad32-718acd329950" containerName="prom-label-proxy"
Jul 29 06:28:43.944010 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.943943    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09946786-61d2-4a1f-ad32-718acd329950" containerName="config-reloader"
Jul 29 06:28:43.944010 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.943948    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="09946786-61d2-4a1f-ad32-718acd329950" containerName="config-reloader"
Jul 29 06:28:43.944010 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.943994    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e68b7c53-3f5b-4bcb-8d4f-a58f9d2f6c48" containerName="registry"
Jul 29 06:28:43.944010 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.944001    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="09946786-61d2-4a1f-ad32-718acd329950" containerName="kube-rbac-proxy"
Jul 29 06:28:43.944010 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.944007    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="09946786-61d2-4a1f-ad32-718acd329950" containerName="prom-label-proxy"
Jul 29 06:28:43.944010 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.944014    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="09946786-61d2-4a1f-ad32-718acd329950" containerName="kube-rbac-proxy-metric"
Jul 29 06:28:43.944409 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.944022    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="09946786-61d2-4a1f-ad32-718acd329950" containerName="alertmanager"
Jul 29 06:28:43.944409 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.944027    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="09946786-61d2-4a1f-ad32-718acd329950" containerName="config-reloader"
Jul 29 06:28:43.944409 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.944033    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="09946786-61d2-4a1f-ad32-718acd329950" containerName="kube-rbac-proxy-web"
Jul 29 06:28:43.947124 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.947107    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:43.949192 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.949172    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 29 06:28:43.949276 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.949173    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 29 06:28:43.949473 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.949458    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 29 06:28:43.949765 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.949749    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tdf2f\""
Jul 29 06:28:43.949765 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.949762    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 29 06:28:43.949890 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.949750    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 29 06:28:43.950063 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.950044    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 29 06:28:43.950179 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.950047    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 29 06:28:43.950179 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.950088    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 29 06:28:43.954439 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.954419    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 29 06:28:43.961326 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:43.961306    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 06:28:44.059113 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.059082    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/36636153-364d-4f22-a2d9-37834e0152c9-config-out\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.059113 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.059118    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/36636153-364d-4f22-a2d9-37834e0152c9-config-volume\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.059330 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.059142    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/36636153-364d-4f22-a2d9-37834e0152c9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.059330 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.059179    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/36636153-364d-4f22-a2d9-37834e0152c9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.059330 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.059207    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/36636153-364d-4f22-a2d9-37834e0152c9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.059330 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.059230    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/36636153-364d-4f22-a2d9-37834e0152c9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.059330 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.059244    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks4b2\" (UniqueName: \"kubernetes.io/projected/36636153-364d-4f22-a2d9-37834e0152c9-kube-api-access-ks4b2\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.059330 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.059270    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36636153-364d-4f22-a2d9-37834e0152c9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.059330 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.059290    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/36636153-364d-4f22-a2d9-37834e0152c9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.059330 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.059312    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36636153-364d-4f22-a2d9-37834e0152c9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.059649 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.059338    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/36636153-364d-4f22-a2d9-37834e0152c9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.059649 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.059424    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/36636153-364d-4f22-a2d9-37834e0152c9-web-config\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.059649 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.059471    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/36636153-364d-4f22-a2d9-37834e0152c9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.159953 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.159876    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36636153-364d-4f22-a2d9-37834e0152c9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.159953 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.159916    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/36636153-364d-4f22-a2d9-37834e0152c9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.159953 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.159937    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/36636153-364d-4f22-a2d9-37834e0152c9-web-config\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.159953 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.159955    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/36636153-364d-4f22-a2d9-37834e0152c9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.160200 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.160080    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/36636153-364d-4f22-a2d9-37834e0152c9-config-out\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.160200 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.160123    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/36636153-364d-4f22-a2d9-37834e0152c9-config-volume\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.160200 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.160164    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/36636153-364d-4f22-a2d9-37834e0152c9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.160200 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.160196    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/36636153-364d-4f22-a2d9-37834e0152c9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.160385 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.160220    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/36636153-364d-4f22-a2d9-37834e0152c9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.160385 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.160258    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/36636153-364d-4f22-a2d9-37834e0152c9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.160385 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.160282    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks4b2\" (UniqueName: \"kubernetes.io/projected/36636153-364d-4f22-a2d9-37834e0152c9-kube-api-access-ks4b2\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.160385 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.160323    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36636153-364d-4f22-a2d9-37834e0152c9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.160385 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.160358    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/36636153-364d-4f22-a2d9-37834e0152c9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.160803 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.160771    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36636153-364d-4f22-a2d9-37834e0152c9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.160941 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.160913    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/36636153-364d-4f22-a2d9-37834e0152c9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.163207 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.163188    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/36636153-364d-4f22-a2d9-37834e0152c9-config-out\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.163306 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.163284    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/36636153-364d-4f22-a2d9-37834e0152c9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.163374 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.163307    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/36636153-364d-4f22-a2d9-37834e0152c9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.163504 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.163482    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/36636153-364d-4f22-a2d9-37834e0152c9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.163577 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.163526    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/36636153-364d-4f22-a2d9-37834e0152c9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.163577 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.163566    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/36636153-364d-4f22-a2d9-37834e0152c9-web-config\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.163790 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.163754    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/36636153-364d-4f22-a2d9-37834e0152c9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.163900 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.163814    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36636153-364d-4f22-a2d9-37834e0152c9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.163900 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.163844    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/36636153-364d-4f22-a2d9-37834e0152c9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.165387 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.165370    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/36636153-364d-4f22-a2d9-37834e0152c9-config-volume\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.168284 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.168266    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks4b2\" (UniqueName: \"kubernetes.io/projected/36636153-364d-4f22-a2d9-37834e0152c9-kube-api-access-ks4b2\") pod \"alertmanager-main-0\" (UID: \"36636153-364d-4f22-a2d9-37834e0152c9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.256119 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.256090    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 06:28:44.391041 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.391012    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 06:28:44.392744 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:28:44.392716    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36636153_364d_4f22_a2d9_37834e0152c9.slice/crio-ceb81256eead45dfd55f05cc83f6092e03281bdb20f88f234c303a12872e7c13 WatchSource:0}: Error finding container ceb81256eead45dfd55f05cc83f6092e03281bdb20f88f234c303a12872e7c13: Status 404 returned error can't find the container with id ceb81256eead45dfd55f05cc83f6092e03281bdb20f88f234c303a12872e7c13
Jul 29 06:28:44.599674 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.599639    2563 generic.go:356] "Generic (PLEG): container finished" podID="36636153-364d-4f22-a2d9-37834e0152c9" containerID="76b41410d7944d711255a53a43bfe65cc8217321952e76b262b80a09d81dadfd" exitCode=0
Jul 29 06:28:44.600039 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.599701    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36636153-364d-4f22-a2d9-37834e0152c9","Type":"ContainerDied","Data":"76b41410d7944d711255a53a43bfe65cc8217321952e76b262b80a09d81dadfd"}
Jul 29 06:28:44.600039 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:44.599779    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36636153-364d-4f22-a2d9-37834e0152c9","Type":"ContainerStarted","Data":"ceb81256eead45dfd55f05cc83f6092e03281bdb20f88f234c303a12872e7c13"}
Jul 29 06:28:45.396822 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:45.396789    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09946786-61d2-4a1f-ad32-718acd329950" path="/var/lib/kubelet/pods/09946786-61d2-4a1f-ad32-718acd329950/volumes"
Jul 29 06:28:45.607252 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:45.607221    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36636153-364d-4f22-a2d9-37834e0152c9","Type":"ContainerStarted","Data":"58e744d780ef6311ad644e1ce913525bcbc250628cab5074bfaf6dd3111a3425"}
Jul 29 06:28:45.607588 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:45.607282    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36636153-364d-4f22-a2d9-37834e0152c9","Type":"ContainerStarted","Data":"727227b0a89b89022962a6793a065c8f0bf81c0376b2de872b229fc418bd668d"}
Jul 29 06:28:45.607588 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:45.607309    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36636153-364d-4f22-a2d9-37834e0152c9","Type":"ContainerStarted","Data":"2a377dd3e87295d26c30dc9862b430206de26cb60152bf2c4c1d3dc2e7853f6d"}
Jul 29 06:28:45.607588 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:45.607341    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36636153-364d-4f22-a2d9-37834e0152c9","Type":"ContainerStarted","Data":"e0b34dbba00d631f6d8dc5a6c28b9dad7bd7f5c01cc9ea20c173f456c22475d2"}
Jul 29 06:28:45.607588 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:45.607364    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36636153-364d-4f22-a2d9-37834e0152c9","Type":"ContainerStarted","Data":"9a4037355656788b0682d8d47cc00b094523a064c59e8b6eaaa9abfa3c06d276"}
Jul 29 06:28:45.607588 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:45.607387    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36636153-364d-4f22-a2d9-37834e0152c9","Type":"ContainerStarted","Data":"4a4a0509c26a183af8723322f8e51a1ae22fef44475cd81425415124fe43dd29"}
Jul 29 06:28:45.631573 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:45.631479    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.6161654050000003 podStartE2EDuration="2.631465988s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.300583ms" isStatefulPod=false podCreationTimestamp="2026-07-29 06:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:28:45.630508564 +0000 UTC m=+368.708140331" watchObservedRunningTime="2026-07-29 06:28:45.631465988 +0000 UTC m=+368.709097754"
Jul 29 06:28:46.592783 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:46.592747    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 06:28:46.593219 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:46.593151    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ea09eb69-0c82-481a-bec4-b165ab890254" containerName="prometheus" containerID="cri-o://9ffd9d668bb6d963817841ab645005feb14c09ed74a8c5746f509257cf403dd8" gracePeriod=600
Jul 29 06:28:46.593219 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:46.593159    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ea09eb69-0c82-481a-bec4-b165ab890254" containerName="kube-rbac-proxy" containerID="cri-o://7d779a9dfda4956e874922e7b72b9c8636475d04fa3959d7ffdc0b10cfe035b5" gracePeriod=600
Jul 29 06:28:46.593219 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:46.593184    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ea09eb69-0c82-481a-bec4-b165ab890254" containerName="kube-rbac-proxy-web" containerID="cri-o://05faf0ff87e5e838859994cbf6a313e16089c5bc719ae6b0063b28f359ef87af" gracePeriod=600
Jul 29 06:28:46.593451 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:46.593208    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ea09eb69-0c82-481a-bec4-b165ab890254" containerName="config-reloader" containerID="cri-o://cff13042b047413f93648d5aba80e737702e289682ccf844d0c87214b14d11af" gracePeriod=600
Jul 29 06:28:46.593451 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:46.593166    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ea09eb69-0c82-481a-bec4-b165ab890254" containerName="thanos-sidecar" containerID="cri-o://45e301eb6e26bc5f061aa0d3e05df1b83a200ac72b8b80999d914ee68f4822c1" gracePeriod=600
Jul 29 06:28:46.593451 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:46.593232    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ea09eb69-0c82-481a-bec4-b165ab890254" containerName="kube-rbac-proxy-thanos" containerID="cri-o://114cf2febc734b18fd600224331c609521b5c3a8a97315238afc5b53c818c7e6" gracePeriod=600
Jul 29 06:28:47.618781 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.618746    2563 generic.go:356] "Generic (PLEG): container finished" podID="ea09eb69-0c82-481a-bec4-b165ab890254" containerID="114cf2febc734b18fd600224331c609521b5c3a8a97315238afc5b53c818c7e6" exitCode=0
Jul 29 06:28:47.618781 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.618771    2563 generic.go:356] "Generic (PLEG): container finished" podID="ea09eb69-0c82-481a-bec4-b165ab890254" containerID="7d779a9dfda4956e874922e7b72b9c8636475d04fa3959d7ffdc0b10cfe035b5" exitCode=0
Jul 29 06:28:47.618781 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.618777    2563 generic.go:356] "Generic (PLEG): container finished" podID="ea09eb69-0c82-481a-bec4-b165ab890254" containerID="45e301eb6e26bc5f061aa0d3e05df1b83a200ac72b8b80999d914ee68f4822c1" exitCode=0
Jul 29 06:28:47.618781 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.618783    2563 generic.go:356] "Generic (PLEG): container finished" podID="ea09eb69-0c82-481a-bec4-b165ab890254" containerID="cff13042b047413f93648d5aba80e737702e289682ccf844d0c87214b14d11af" exitCode=0
Jul 29 06:28:47.618781 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.618788    2563 generic.go:356] "Generic (PLEG): container finished" podID="ea09eb69-0c82-481a-bec4-b165ab890254" containerID="9ffd9d668bb6d963817841ab645005feb14c09ed74a8c5746f509257cf403dd8" exitCode=0
Jul 29 06:28:47.619260 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.618813    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ea09eb69-0c82-481a-bec4-b165ab890254","Type":"ContainerDied","Data":"114cf2febc734b18fd600224331c609521b5c3a8a97315238afc5b53c818c7e6"}
Jul 29 06:28:47.619260 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.618889    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ea09eb69-0c82-481a-bec4-b165ab890254","Type":"ContainerDied","Data":"7d779a9dfda4956e874922e7b72b9c8636475d04fa3959d7ffdc0b10cfe035b5"}
Jul 29 06:28:47.619260 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.618917    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ea09eb69-0c82-481a-bec4-b165ab890254","Type":"ContainerDied","Data":"45e301eb6e26bc5f061aa0d3e05df1b83a200ac72b8b80999d914ee68f4822c1"}
Jul 29 06:28:47.619260 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.618940    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ea09eb69-0c82-481a-bec4-b165ab890254","Type":"ContainerDied","Data":"cff13042b047413f93648d5aba80e737702e289682ccf844d0c87214b14d11af"}
Jul 29 06:28:47.619260 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.618963    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ea09eb69-0c82-481a-bec4-b165ab890254","Type":"ContainerDied","Data":"9ffd9d668bb6d963817841ab645005feb14c09ed74a8c5746f509257cf403dd8"}
Jul 29 06:28:47.835353 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.835331    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:47.893151 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.893077    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-prometheus-k8s-rulefiles-0\") pod \"ea09eb69-0c82-481a-bec4-b165ab890254\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") "
Jul 29 06:28:47.893151 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.893118    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-configmap-kubelet-serving-ca-bundle\") pod \"ea09eb69-0c82-481a-bec4-b165ab890254\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") "
Jul 29 06:28:47.893151 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.893143    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-prometheus-k8s-tls\") pod \"ea09eb69-0c82-481a-bec4-b165ab890254\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") "
Jul 29 06:28:47.893387 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.893168    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-thanos-prometheus-http-client-file\") pod \"ea09eb69-0c82-481a-bec4-b165ab890254\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") "
Jul 29 06:28:47.893387 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.893207    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-prometheus-k8s-rulefiles-1\") pod \"ea09eb69-0c82-481a-bec4-b165ab890254\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") "
Jul 29 06:28:47.893387 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.893231    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"ea09eb69-0c82-481a-bec4-b165ab890254\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") "
Jul 29 06:28:47.893387 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.893294    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-metrics-client-certs\") pod \"ea09eb69-0c82-481a-bec4-b165ab890254\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") "
Jul 29 06:28:47.893577 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.893540    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "ea09eb69-0c82-481a-bec4-b165ab890254" (UID: "ea09eb69-0c82-481a-bec4-b165ab890254"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 06:28:47.893687 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.893613    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "ea09eb69-0c82-481a-bec4-b165ab890254" (UID: "ea09eb69-0c82-481a-bec4-b165ab890254"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 06:28:47.893687 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.893661    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ea09eb69-0c82-481a-bec4-b165ab890254-config-out\") pod \"ea09eb69-0c82-481a-bec4-b165ab890254\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") "
Jul 29 06:28:47.893788 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.893698    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-kube-rbac-proxy\") pod \"ea09eb69-0c82-481a-bec4-b165ab890254\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") "
Jul 29 06:28:47.893788 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.893729    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4rqq\" (UniqueName: \"kubernetes.io/projected/ea09eb69-0c82-481a-bec4-b165ab890254-kube-api-access-x4rqq\") pod \"ea09eb69-0c82-481a-bec4-b165ab890254\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") "
Jul 29 06:28:47.893788 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.893777    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-prometheus-k8s-rulefiles-2\") pod \"ea09eb69-0c82-481a-bec4-b165ab890254\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") "
Jul 29 06:28:47.893986 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.893816    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-grpc-tls\") pod \"ea09eb69-0c82-481a-bec4-b165ab890254\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") "
Jul 29 06:28:47.893986 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.893846    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ea09eb69-0c82-481a-bec4-b165ab890254-prometheus-k8s-db\") pod \"ea09eb69-0c82-481a-bec4-b165ab890254\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") "
Jul 29 06:28:47.893986 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.893896    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-config\") pod \"ea09eb69-0c82-481a-bec4-b165ab890254\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") "
Jul 29 06:28:47.893986 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.893921    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ea09eb69-0c82-481a-bec4-b165ab890254-tls-assets\") pod \"ea09eb69-0c82-481a-bec4-b165ab890254\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") "
Jul 29 06:28:47.893986 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.893960    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-configmap-metrics-client-ca\") pod \"ea09eb69-0c82-481a-bec4-b165ab890254\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") "
Jul 29 06:28:47.894212 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.893993    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-prometheus-trusted-ca-bundle\") pod \"ea09eb69-0c82-481a-bec4-b165ab890254\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") "
Jul 29 06:28:47.894212 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.894018    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-web-config\") pod \"ea09eb69-0c82-481a-bec4-b165ab890254\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") "
Jul 29 06:28:47.894212 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.894194    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"ea09eb69-0c82-481a-bec4-b165ab890254\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") "
Jul 29 06:28:47.894346 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.894248    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-configmap-serving-certs-ca-bundle\") pod \"ea09eb69-0c82-481a-bec4-b165ab890254\" (UID: \"ea09eb69-0c82-481a-bec4-b165ab890254\") "
Jul 29 06:28:47.894346 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.894286    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "ea09eb69-0c82-481a-bec4-b165ab890254" (UID: "ea09eb69-0c82-481a-bec4-b165ab890254"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 06:28:47.894551 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.894528    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:47.894605 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.894557    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:47.894605 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.894574    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:47.894698 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.894670    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "ea09eb69-0c82-481a-bec4-b165ab890254" (UID: "ea09eb69-0c82-481a-bec4-b165ab890254"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 06:28:47.894922 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.894895    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "ea09eb69-0c82-481a-bec4-b165ab890254" (UID: "ea09eb69-0c82-481a-bec4-b165ab890254"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 06:28:47.895545 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.895498    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "ea09eb69-0c82-481a-bec4-b165ab890254" (UID: "ea09eb69-0c82-481a-bec4-b165ab890254"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 06:28:47.896333 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.896305    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea09eb69-0c82-481a-bec4-b165ab890254-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "ea09eb69-0c82-481a-bec4-b165ab890254" (UID: "ea09eb69-0c82-481a-bec4-b165ab890254"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 06:28:47.897337 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.897066    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "ea09eb69-0c82-481a-bec4-b165ab890254" (UID: "ea09eb69-0c82-481a-bec4-b165ab890254"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 06:28:47.898106 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.898076    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "ea09eb69-0c82-481a-bec4-b165ab890254" (UID: "ea09eb69-0c82-481a-bec4-b165ab890254"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 06:28:47.901346 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.901215    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea09eb69-0c82-481a-bec4-b165ab890254-config-out" (OuterVolumeSpecName: "config-out") pod "ea09eb69-0c82-481a-bec4-b165ab890254" (UID: "ea09eb69-0c82-481a-bec4-b165ab890254"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 06:28:47.901346 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.901265    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "ea09eb69-0c82-481a-bec4-b165ab890254" (UID: "ea09eb69-0c82-481a-bec4-b165ab890254"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 06:28:47.901346 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.901319    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "ea09eb69-0c82-481a-bec4-b165ab890254" (UID: "ea09eb69-0c82-481a-bec4-b165ab890254"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 06:28:47.901346 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.901336    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "ea09eb69-0c82-481a-bec4-b165ab890254" (UID: "ea09eb69-0c82-481a-bec4-b165ab890254"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 06:28:47.901765 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.901732    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "ea09eb69-0c82-481a-bec4-b165ab890254" (UID: "ea09eb69-0c82-481a-bec4-b165ab890254"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 06:28:47.901838 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.901782    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "ea09eb69-0c82-481a-bec4-b165ab890254" (UID: "ea09eb69-0c82-481a-bec4-b165ab890254"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 06:28:47.901977 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.901954    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-config" (OuterVolumeSpecName: "config") pod "ea09eb69-0c82-481a-bec4-b165ab890254" (UID: "ea09eb69-0c82-481a-bec4-b165ab890254"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 06:28:47.902060 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.901978    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea09eb69-0c82-481a-bec4-b165ab890254-kube-api-access-x4rqq" (OuterVolumeSpecName: "kube-api-access-x4rqq") pod "ea09eb69-0c82-481a-bec4-b165ab890254" (UID: "ea09eb69-0c82-481a-bec4-b165ab890254"). InnerVolumeSpecName "kube-api-access-x4rqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 06:28:47.902060 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.901988    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea09eb69-0c82-481a-bec4-b165ab890254-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ea09eb69-0c82-481a-bec4-b165ab890254" (UID: "ea09eb69-0c82-481a-bec4-b165ab890254"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 06:28:47.902222 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.902203    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "ea09eb69-0c82-481a-bec4-b165ab890254" (UID: "ea09eb69-0c82-481a-bec4-b165ab890254"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 06:28:47.912528 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.912506    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-web-config" (OuterVolumeSpecName: "web-config") pod "ea09eb69-0c82-481a-bec4-b165ab890254" (UID: "ea09eb69-0c82-481a-bec4-b165ab890254"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 06:28:47.995363 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.995334    2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ea09eb69-0c82-481a-bec4-b165ab890254-config-out\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:47.995363 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.995359    2563 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-kube-rbac-proxy\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:47.995363 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.995369    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4rqq\" (UniqueName: \"kubernetes.io/projected/ea09eb69-0c82-481a-bec4-b165ab890254-kube-api-access-x4rqq\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:47.995556 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.995378    2563 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-grpc-tls\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:47.995556 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.995387    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ea09eb69-0c82-481a-bec4-b165ab890254-prometheus-k8s-db\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:47.995556 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.995395    2563 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-config\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:47.995556 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.995403    2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ea09eb69-0c82-481a-bec4-b165ab890254-tls-assets\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:47.995556 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.995413    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-configmap-metrics-client-ca\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:47.995556 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.995422    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:47.995556 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.995430    2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-web-config\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:47.995556 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.995438    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:47.995556 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.995447    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:47.995556 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.995456    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ea09eb69-0c82-481a-bec4-b165ab890254-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:47.995556 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.995464    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:47.995556 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.995472    2563 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:47.995556 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.995480    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:47.995556 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:47.995488    2563 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ea09eb69-0c82-481a-bec4-b165ab890254-secret-metrics-client-certs\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:28:48.626113 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.626082    2563 generic.go:356] "Generic (PLEG): container finished" podID="ea09eb69-0c82-481a-bec4-b165ab890254" containerID="05faf0ff87e5e838859994cbf6a313e16089c5bc719ae6b0063b28f359ef87af" exitCode=0
Jul 29 06:28:48.626576 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.626136    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ea09eb69-0c82-481a-bec4-b165ab890254","Type":"ContainerDied","Data":"05faf0ff87e5e838859994cbf6a313e16089c5bc719ae6b0063b28f359ef87af"}
Jul 29 06:28:48.626576 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.626156    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.626576 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.626214    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ea09eb69-0c82-481a-bec4-b165ab890254","Type":"ContainerDied","Data":"e7674e90bc99b52325da61941f1d7d10a4003c1beba1aca7637062ca08bc6432"}
Jul 29 06:28:48.626576 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.626279    2563 scope.go:117] "RemoveContainer" containerID="114cf2febc734b18fd600224331c609521b5c3a8a97315238afc5b53c818c7e6"
Jul 29 06:28:48.634657 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.634640    2563 scope.go:117] "RemoveContainer" containerID="7d779a9dfda4956e874922e7b72b9c8636475d04fa3959d7ffdc0b10cfe035b5"
Jul 29 06:28:48.641509 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.641495    2563 scope.go:117] "RemoveContainer" containerID="05faf0ff87e5e838859994cbf6a313e16089c5bc719ae6b0063b28f359ef87af"
Jul 29 06:28:48.648339 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.648322    2563 scope.go:117] "RemoveContainer" containerID="45e301eb6e26bc5f061aa0d3e05df1b83a200ac72b8b80999d914ee68f4822c1"
Jul 29 06:28:48.648767 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.648705    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 06:28:48.653739 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.653717    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 06:28:48.659393 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.659379    2563 scope.go:117] "RemoveContainer" containerID="cff13042b047413f93648d5aba80e737702e289682ccf844d0c87214b14d11af"
Jul 29 06:28:48.665788 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.665774    2563 scope.go:117] "RemoveContainer" containerID="9ffd9d668bb6d963817841ab645005feb14c09ed74a8c5746f509257cf403dd8"
Jul 29 06:28:48.672921 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.672904    2563 scope.go:117] "RemoveContainer" containerID="a98353e74d92fe810caef4c77b1dfde237ba03cdb9444262e9aa9a0eb1582b0e"
Jul 29 06:28:48.674930 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.674913    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 06:28:48.675200 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.675188    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea09eb69-0c82-481a-bec4-b165ab890254" containerName="config-reloader"
Jul 29 06:28:48.675239 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.675202    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea09eb69-0c82-481a-bec4-b165ab890254" containerName="config-reloader"
Jul 29 06:28:48.675239 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.675210    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea09eb69-0c82-481a-bec4-b165ab890254" containerName="thanos-sidecar"
Jul 29 06:28:48.675239 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.675216    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea09eb69-0c82-481a-bec4-b165ab890254" containerName="thanos-sidecar"
Jul 29 06:28:48.675239 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.675232    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea09eb69-0c82-481a-bec4-b165ab890254" containerName="kube-rbac-proxy-thanos"
Jul 29 06:28:48.675239 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.675238    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea09eb69-0c82-481a-bec4-b165ab890254" containerName="kube-rbac-proxy-thanos"
Jul 29 06:28:48.675410 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.675245    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea09eb69-0c82-481a-bec4-b165ab890254" containerName="prometheus"
Jul 29 06:28:48.675410 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.675250    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea09eb69-0c82-481a-bec4-b165ab890254" containerName="prometheus"
Jul 29 06:28:48.675410 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.675256    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea09eb69-0c82-481a-bec4-b165ab890254" containerName="kube-rbac-proxy"
Jul 29 06:28:48.675410 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.675261    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea09eb69-0c82-481a-bec4-b165ab890254" containerName="kube-rbac-proxy"
Jul 29 06:28:48.675410 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.675267    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea09eb69-0c82-481a-bec4-b165ab890254" containerName="init-config-reloader"
Jul 29 06:28:48.675410 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.675273    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea09eb69-0c82-481a-bec4-b165ab890254" containerName="init-config-reloader"
Jul 29 06:28:48.675410 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.675281    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea09eb69-0c82-481a-bec4-b165ab890254" containerName="kube-rbac-proxy-web"
Jul 29 06:28:48.675410 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.675285    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea09eb69-0c82-481a-bec4-b165ab890254" containerName="kube-rbac-proxy-web"
Jul 29 06:28:48.675410 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.675327    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea09eb69-0c82-481a-bec4-b165ab890254" containerName="prometheus"
Jul 29 06:28:48.675410 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.675335    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea09eb69-0c82-481a-bec4-b165ab890254" containerName="kube-rbac-proxy"
Jul 29 06:28:48.675410 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.675342    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea09eb69-0c82-481a-bec4-b165ab890254" containerName="thanos-sidecar"
Jul 29 06:28:48.675410 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.675348    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea09eb69-0c82-481a-bec4-b165ab890254" containerName="kube-rbac-proxy-thanos"
Jul 29 06:28:48.675410 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.675354    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea09eb69-0c82-481a-bec4-b165ab890254" containerName="config-reloader"
Jul 29 06:28:48.675410 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.675359    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea09eb69-0c82-481a-bec4-b165ab890254" containerName="kube-rbac-proxy-web"
Jul 29 06:28:48.679113 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.679097    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.680368 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.680335    2563 scope.go:117] "RemoveContainer" containerID="114cf2febc734b18fd600224331c609521b5c3a8a97315238afc5b53c818c7e6"
Jul 29 06:28:48.680608 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:28:48.680583    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"114cf2febc734b18fd600224331c609521b5c3a8a97315238afc5b53c818c7e6\": container with ID starting with 114cf2febc734b18fd600224331c609521b5c3a8a97315238afc5b53c818c7e6 not found: ID does not exist" containerID="114cf2febc734b18fd600224331c609521b5c3a8a97315238afc5b53c818c7e6"
Jul 29 06:28:48.680682 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.680619    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"114cf2febc734b18fd600224331c609521b5c3a8a97315238afc5b53c818c7e6"} err="failed to get container status \"114cf2febc734b18fd600224331c609521b5c3a8a97315238afc5b53c818c7e6\": rpc error: code = NotFound desc = could not find container \"114cf2febc734b18fd600224331c609521b5c3a8a97315238afc5b53c818c7e6\": container with ID starting with 114cf2febc734b18fd600224331c609521b5c3a8a97315238afc5b53c818c7e6 not found: ID does not exist"
Jul 29 06:28:48.680682 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.680641    2563 scope.go:117] "RemoveContainer" containerID="7d779a9dfda4956e874922e7b72b9c8636475d04fa3959d7ffdc0b10cfe035b5"
Jul 29 06:28:48.681005 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:28:48.680985    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d779a9dfda4956e874922e7b72b9c8636475d04fa3959d7ffdc0b10cfe035b5\": container with ID starting with 7d779a9dfda4956e874922e7b72b9c8636475d04fa3959d7ffdc0b10cfe035b5 not found: ID does not exist" containerID="7d779a9dfda4956e874922e7b72b9c8636475d04fa3959d7ffdc0b10cfe035b5"
Jul 29 06:28:48.681086 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.681011    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d779a9dfda4956e874922e7b72b9c8636475d04fa3959d7ffdc0b10cfe035b5"} err="failed to get container status \"7d779a9dfda4956e874922e7b72b9c8636475d04fa3959d7ffdc0b10cfe035b5\": rpc error: code = NotFound desc = could not find container \"7d779a9dfda4956e874922e7b72b9c8636475d04fa3959d7ffdc0b10cfe035b5\": container with ID starting with 7d779a9dfda4956e874922e7b72b9c8636475d04fa3959d7ffdc0b10cfe035b5 not found: ID does not exist"
Jul 29 06:28:48.681086 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.681027    2563 scope.go:117] "RemoveContainer" containerID="05faf0ff87e5e838859994cbf6a313e16089c5bc719ae6b0063b28f359ef87af"
Jul 29 06:28:48.681199 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.681117    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 29 06:28:48.681365 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:28:48.681341    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05faf0ff87e5e838859994cbf6a313e16089c5bc719ae6b0063b28f359ef87af\": container with ID starting with 05faf0ff87e5e838859994cbf6a313e16089c5bc719ae6b0063b28f359ef87af not found: ID does not exist" containerID="05faf0ff87e5e838859994cbf6a313e16089c5bc719ae6b0063b28f359ef87af"
Jul 29 06:28:48.681466 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.681369    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05faf0ff87e5e838859994cbf6a313e16089c5bc719ae6b0063b28f359ef87af"} err="failed to get container status \"05faf0ff87e5e838859994cbf6a313e16089c5bc719ae6b0063b28f359ef87af\": rpc error: code = NotFound desc = could not find container \"05faf0ff87e5e838859994cbf6a313e16089c5bc719ae6b0063b28f359ef87af\": container with ID starting with 05faf0ff87e5e838859994cbf6a313e16089c5bc719ae6b0063b28f359ef87af not found: ID does not exist"
Jul 29 06:28:48.681466 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.681384    2563 scope.go:117] "RemoveContainer" containerID="45e301eb6e26bc5f061aa0d3e05df1b83a200ac72b8b80999d914ee68f4822c1"
Jul 29 06:28:48.681466 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.681401    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 29 06:28:48.681686 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:28:48.681655    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45e301eb6e26bc5f061aa0d3e05df1b83a200ac72b8b80999d914ee68f4822c1\": container with ID starting with 45e301eb6e26bc5f061aa0d3e05df1b83a200ac72b8b80999d914ee68f4822c1 not found: ID does not exist" containerID="45e301eb6e26bc5f061aa0d3e05df1b83a200ac72b8b80999d914ee68f4822c1"
Jul 29 06:28:48.681686 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.681693    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45e301eb6e26bc5f061aa0d3e05df1b83a200ac72b8b80999d914ee68f4822c1"} err="failed to get container status \"45e301eb6e26bc5f061aa0d3e05df1b83a200ac72b8b80999d914ee68f4822c1\": rpc error: code = NotFound desc = could not find container \"45e301eb6e26bc5f061aa0d3e05df1b83a200ac72b8b80999d914ee68f4822c1\": container with ID starting with 45e301eb6e26bc5f061aa0d3e05df1b83a200ac72b8b80999d914ee68f4822c1 not found: ID does not exist"
Jul 29 06:28:48.681686 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.681674    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 29 06:28:48.681686 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.681712    2563 scope.go:117] "RemoveContainer" containerID="cff13042b047413f93648d5aba80e737702e289682ccf844d0c87214b14d11af"
Jul 29 06:28:48.681686 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.681732    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 29 06:28:48.682062 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.681743    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 29 06:28:48.682062 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.681702    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 29 06:28:48.682062 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.681922    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 29 06:28:48.682062 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.682017    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 29 06:28:48.682291 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:28:48.682173    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cff13042b047413f93648d5aba80e737702e289682ccf844d0c87214b14d11af\": container with ID starting with cff13042b047413f93648d5aba80e737702e289682ccf844d0c87214b14d11af not found: ID does not exist" containerID="cff13042b047413f93648d5aba80e737702e289682ccf844d0c87214b14d11af"
Jul 29 06:28:48.682291 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.682202    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cff13042b047413f93648d5aba80e737702e289682ccf844d0c87214b14d11af"} err="failed to get container status \"cff13042b047413f93648d5aba80e737702e289682ccf844d0c87214b14d11af\": rpc error: code = NotFound desc = could not find container \"cff13042b047413f93648d5aba80e737702e289682ccf844d0c87214b14d11af\": container with ID starting with cff13042b047413f93648d5aba80e737702e289682ccf844d0c87214b14d11af not found: ID does not exist"
Jul 29 06:28:48.682291 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.682221    2563 scope.go:117] "RemoveContainer" containerID="9ffd9d668bb6d963817841ab645005feb14c09ed74a8c5746f509257cf403dd8"
Jul 29 06:28:48.682291 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.682260    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 29 06:28:48.682291 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.682222    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 29 06:28:48.682291 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.682284    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ef390dc5dvkue\""
Jul 29 06:28:48.682626 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.682387    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 29 06:28:48.682626 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.682407    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-bvlrp\""
Jul 29 06:28:48.682626 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.682530    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 29 06:28:48.682626 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.682542    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 29 06:28:48.682878 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:28:48.682695    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ffd9d668bb6d963817841ab645005feb14c09ed74a8c5746f509257cf403dd8\": container with ID starting with 9ffd9d668bb6d963817841ab645005feb14c09ed74a8c5746f509257cf403dd8 not found: ID does not exist" containerID="9ffd9d668bb6d963817841ab645005feb14c09ed74a8c5746f509257cf403dd8"
Jul 29 06:28:48.682878 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.682721    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ffd9d668bb6d963817841ab645005feb14c09ed74a8c5746f509257cf403dd8"} err="failed to get container status \"9ffd9d668bb6d963817841ab645005feb14c09ed74a8c5746f509257cf403dd8\": rpc error: code = NotFound desc = could not find container \"9ffd9d668bb6d963817841ab645005feb14c09ed74a8c5746f509257cf403dd8\": container with ID starting with 9ffd9d668bb6d963817841ab645005feb14c09ed74a8c5746f509257cf403dd8 not found: ID does not exist"
Jul 29 06:28:48.682878 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.682740    2563 scope.go:117] "RemoveContainer" containerID="a98353e74d92fe810caef4c77b1dfde237ba03cdb9444262e9aa9a0eb1582b0e"
Jul 29 06:28:48.683100 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:28:48.683046    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a98353e74d92fe810caef4c77b1dfde237ba03cdb9444262e9aa9a0eb1582b0e\": container with ID starting with a98353e74d92fe810caef4c77b1dfde237ba03cdb9444262e9aa9a0eb1582b0e not found: ID does not exist" containerID="a98353e74d92fe810caef4c77b1dfde237ba03cdb9444262e9aa9a0eb1582b0e"
Jul 29 06:28:48.683100 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.683072    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a98353e74d92fe810caef4c77b1dfde237ba03cdb9444262e9aa9a0eb1582b0e"} err="failed to get container status \"a98353e74d92fe810caef4c77b1dfde237ba03cdb9444262e9aa9a0eb1582b0e\": rpc error: code = NotFound desc = could not find container \"a98353e74d92fe810caef4c77b1dfde237ba03cdb9444262e9aa9a0eb1582b0e\": container with ID starting with a98353e74d92fe810caef4c77b1dfde237ba03cdb9444262e9aa9a0eb1582b0e not found: ID does not exist"
Jul 29 06:28:48.686840 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.686812    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 29 06:28:48.688944 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.688924    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 29 06:28:48.693652 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.693634    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 06:28:48.801225 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.801189    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/526902c0-ec0e-4384-b718-24d9387d9fee-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.801369 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.801238    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/526902c0-ec0e-4384-b718-24d9387d9fee-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.801369 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.801259    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/526902c0-ec0e-4384-b718-24d9387d9fee-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.801369 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.801278    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/526902c0-ec0e-4384-b718-24d9387d9fee-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.801369 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.801294    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/526902c0-ec0e-4384-b718-24d9387d9fee-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.801369 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.801326    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/526902c0-ec0e-4384-b718-24d9387d9fee-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.801538 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.801375    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/526902c0-ec0e-4384-b718-24d9387d9fee-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.801538 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.801403    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/526902c0-ec0e-4384-b718-24d9387d9fee-config-out\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.801538 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.801437    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/526902c0-ec0e-4384-b718-24d9387d9fee-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.801538 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.801476    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b77l9\" (UniqueName: \"kubernetes.io/projected/526902c0-ec0e-4384-b718-24d9387d9fee-kube-api-access-b77l9\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.801538 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.801498    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/526902c0-ec0e-4384-b718-24d9387d9fee-config\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.801538 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.801513    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/526902c0-ec0e-4384-b718-24d9387d9fee-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.801707 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.801554    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/526902c0-ec0e-4384-b718-24d9387d9fee-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.801707 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.801575    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/526902c0-ec0e-4384-b718-24d9387d9fee-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.801707 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.801617    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/526902c0-ec0e-4384-b718-24d9387d9fee-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.801707 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.801644    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/526902c0-ec0e-4384-b718-24d9387d9fee-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.801707 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.801657    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/526902c0-ec0e-4384-b718-24d9387d9fee-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.801845 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.801710    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/526902c0-ec0e-4384-b718-24d9387d9fee-web-config\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.801845 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.801726    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/526902c0-ec0e-4384-b718-24d9387d9fee-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.801845 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.801740    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/526902c0-ec0e-4384-b718-24d9387d9fee-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.903317 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.903231    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/526902c0-ec0e-4384-b718-24d9387d9fee-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.903317 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.903272    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/526902c0-ec0e-4384-b718-24d9387d9fee-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.903317 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.903292    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/526902c0-ec0e-4384-b718-24d9387d9fee-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.903317 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.903308    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/526902c0-ec0e-4384-b718-24d9387d9fee-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.903633 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.903332    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/526902c0-ec0e-4384-b718-24d9387d9fee-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.903633 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.903353    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/526902c0-ec0e-4384-b718-24d9387d9fee-config-out\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.903633 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.903367    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/526902c0-ec0e-4384-b718-24d9387d9fee-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.903633 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.903392    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b77l9\" (UniqueName: \"kubernetes.io/projected/526902c0-ec0e-4384-b718-24d9387d9fee-kube-api-access-b77l9\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.903633 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.903412    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/526902c0-ec0e-4384-b718-24d9387d9fee-config\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.903633 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.903430    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/526902c0-ec0e-4384-b718-24d9387d9fee-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.903633 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.903466    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/526902c0-ec0e-4384-b718-24d9387d9fee-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.903633 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.903493    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/526902c0-ec0e-4384-b718-24d9387d9fee-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.903633 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.903519    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/526902c0-ec0e-4384-b718-24d9387d9fee-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.903633 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.903543    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/526902c0-ec0e-4384-b718-24d9387d9fee-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.903633 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.903565    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/526902c0-ec0e-4384-b718-24d9387d9fee-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.903633 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.903603    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/526902c0-ec0e-4384-b718-24d9387d9fee-web-config\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.903633 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.903627    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/526902c0-ec0e-4384-b718-24d9387d9fee-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.904283 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.903647    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/526902c0-ec0e-4384-b718-24d9387d9fee-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.904283 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.903680    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/526902c0-ec0e-4384-b718-24d9387d9fee-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.904283 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.903717    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/526902c0-ec0e-4384-b718-24d9387d9fee-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.904283 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.904070    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/526902c0-ec0e-4384-b718-24d9387d9fee-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.904283 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.904103    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/526902c0-ec0e-4384-b718-24d9387d9fee-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.904538 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.904360    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/526902c0-ec0e-4384-b718-24d9387d9fee-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.904831 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.904665    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/526902c0-ec0e-4384-b718-24d9387d9fee-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.905482 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.905460    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/526902c0-ec0e-4384-b718-24d9387d9fee-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.906640 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.906614    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/526902c0-ec0e-4384-b718-24d9387d9fee-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.906839 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.906814    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/526902c0-ec0e-4384-b718-24d9387d9fee-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.907054 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.907030    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/526902c0-ec0e-4384-b718-24d9387d9fee-web-config\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.907830 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.907315    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/526902c0-ec0e-4384-b718-24d9387d9fee-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.907830 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.906826    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/526902c0-ec0e-4384-b718-24d9387d9fee-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.907830 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.907376    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/526902c0-ec0e-4384-b718-24d9387d9fee-config-out\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.907830 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.907743    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/526902c0-ec0e-4384-b718-24d9387d9fee-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.908196 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.908148    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/526902c0-ec0e-4384-b718-24d9387d9fee-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.908807 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.908783    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/526902c0-ec0e-4384-b718-24d9387d9fee-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.908921 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.908883    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/526902c0-ec0e-4384-b718-24d9387d9fee-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.909300 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.909277    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/526902c0-ec0e-4384-b718-24d9387d9fee-config\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.909398 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.909376    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/526902c0-ec0e-4384-b718-24d9387d9fee-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.910080 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.910061    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/526902c0-ec0e-4384-b718-24d9387d9fee-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.910144 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.910087    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/526902c0-ec0e-4384-b718-24d9387d9fee-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.911198 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.911180    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b77l9\" (UniqueName: \"kubernetes.io/projected/526902c0-ec0e-4384-b718-24d9387d9fee-kube-api-access-b77l9\") pod \"prometheus-k8s-0\" (UID: \"526902c0-ec0e-4384-b718-24d9387d9fee\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:48.990696 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:48.990669    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:28:49.128275 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:49.128201    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 06:28:49.130357 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:28:49.130337    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod526902c0_ec0e_4384_b718_24d9387d9fee.slice/crio-063738d0b563984f93e30abb451e249e02fbaf99f30ff873fd928f3666757e00 WatchSource:0}: Error finding container 063738d0b563984f93e30abb451e249e02fbaf99f30ff873fd928f3666757e00: Status 404 returned error can't find the container with id 063738d0b563984f93e30abb451e249e02fbaf99f30ff873fd928f3666757e00
Jul 29 06:28:49.396662 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:49.396631    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea09eb69-0c82-481a-bec4-b165ab890254" path="/var/lib/kubelet/pods/ea09eb69-0c82-481a-bec4-b165ab890254/volumes"
Jul 29 06:28:49.631965 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:49.631932    2563 generic.go:356] "Generic (PLEG): container finished" podID="526902c0-ec0e-4384-b718-24d9387d9fee" containerID="63afb57e182398231d9c1134b27111c9be76901b43a7f64f74269a0afa27e2ba" exitCode=0
Jul 29 06:28:49.632359 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:49.632018    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"526902c0-ec0e-4384-b718-24d9387d9fee","Type":"ContainerDied","Data":"63afb57e182398231d9c1134b27111c9be76901b43a7f64f74269a0afa27e2ba"}
Jul 29 06:28:49.632359 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:49.632104    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"526902c0-ec0e-4384-b718-24d9387d9fee","Type":"ContainerStarted","Data":"063738d0b563984f93e30abb451e249e02fbaf99f30ff873fd928f3666757e00"}
Jul 29 06:28:50.641430 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:50.641399    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"526902c0-ec0e-4384-b718-24d9387d9fee","Type":"ContainerStarted","Data":"211fb1b4fdccebc17633622519b0f786a0c5e48e71929abb5bc14bc0e53352f8"}
Jul 29 06:28:50.641808 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:50.641458    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"526902c0-ec0e-4384-b718-24d9387d9fee","Type":"ContainerStarted","Data":"ee924629e3dbaf97c72359e29e0fdbe814e5ab356e4dcab4ae2feaa378ba1941"}
Jul 29 06:28:50.641808 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:50.641485    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"526902c0-ec0e-4384-b718-24d9387d9fee","Type":"ContainerStarted","Data":"87709cbcb0455bde31b1e0e70234c119c85a18a6b68b26ccf09bcb82dfa5c42b"}
Jul 29 06:28:50.641808 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:50.641514    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"526902c0-ec0e-4384-b718-24d9387d9fee","Type":"ContainerStarted","Data":"e5d2e48eb78bc046d6f66a09075169c8c28a40c5ace986c3ccfa52495babcb32"}
Jul 29 06:28:50.641808 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:50.641539    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"526902c0-ec0e-4384-b718-24d9387d9fee","Type":"ContainerStarted","Data":"a69ef176a3eec265173d2ea57b160f560fba8e171c05c0b631c02e062a998397"}
Jul 29 06:28:50.641808 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:50.641560    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"526902c0-ec0e-4384-b718-24d9387d9fee","Type":"ContainerStarted","Data":"02e08bf28cdf7440d937727ce39b4fe92d085a70f759021c0ccf9e5240a3d129"}
Jul 29 06:28:50.666584 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:50.666506    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.64887871 podStartE2EDuration="2.666483075s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.604365ms" isStatefulPod=false podCreationTimestamp="2026-07-29 06:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:28:50.664573637 +0000 UTC m=+373.742205468" watchObservedRunningTime="2026-07-29 06:28:50.666483075 +0000 UTC m=+373.744114846"
Jul 29 06:28:51.326203 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:51.326167    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-metrics-certs\") pod \"network-metrics-daemon-vcg74\" (UID: \"2ceff9c5-890d-4d2b-85ee-09bcd9e02833\") " pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:28:51.328700 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:51.328671    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ceff9c5-890d-4d2b-85ee-09bcd9e02833-metrics-certs\") pod \"network-metrics-daemon-vcg74\" (UID: \"2ceff9c5-890d-4d2b-85ee-09bcd9e02833\") " pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:28:51.393824 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:51.393791    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-btmpr\""
Jul 29 06:28:51.401769 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:51.401746    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcg74"
Jul 29 06:28:51.546312 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:51.546284    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vcg74"]
Jul 29 06:28:51.549694 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:28:51.549667    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ceff9c5_890d_4d2b_85ee_09bcd9e02833.slice/crio-5e26fff8da07c1a2362dbd944445802369d433968ab2a6ce9bf83b7c1ba6e5dd WatchSource:0}: Error finding container 5e26fff8da07c1a2362dbd944445802369d433968ab2a6ce9bf83b7c1ba6e5dd: Status 404 returned error can't find the container with id 5e26fff8da07c1a2362dbd944445802369d433968ab2a6ce9bf83b7c1ba6e5dd
Jul 29 06:28:51.551873 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:51.551839    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 06:28:51.646953 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:51.646869    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vcg74" event={"ID":"2ceff9c5-890d-4d2b-85ee-09bcd9e02833","Type":"ContainerStarted","Data":"5e26fff8da07c1a2362dbd944445802369d433968ab2a6ce9bf83b7c1ba6e5dd"}
Jul 29 06:28:52.652199 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:52.652121    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vcg74" event={"ID":"2ceff9c5-890d-4d2b-85ee-09bcd9e02833","Type":"ContainerStarted","Data":"1e9968cd3b54e9a981f8aff09a5282c53764497f67bd97ac703f313ccd44fb94"}
Jul 29 06:28:52.652199 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:52.652169    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vcg74" event={"ID":"2ceff9c5-890d-4d2b-85ee-09bcd9e02833","Type":"ContainerStarted","Data":"44c9ea4f7dd838605cdee8e74320f6de839c2c6bc1da991356601bd275002a2e"}
Jul 29 06:28:52.667045 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:52.667000    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vcg74" podStartSLOduration=374.830132507 podStartE2EDuration="6m15.666983903s" totalImagesPullingTime="836.851396ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:28:52.666199069 +0000 UTC m=+375.743830836" watchObservedRunningTime="2026-07-29 06:28:52.666983903 +0000 UTC m=+375.744615670"
Jul 29 06:28:53.991922 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:28:53.991888    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:29:48.991341 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:29:48.991304    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:29:49.007257 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:29:49.007231    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:29:49.897835 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:29:49.897797    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 06:31:53.130589 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:31:53.130551    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-fwsh9"]
Jul 29 06:31:53.133507 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:31:53.133490    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fwsh9"
Jul 29 06:31:53.135569 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:31:53.135549    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 29 06:31:53.135676 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:31:53.135611    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 29 06:31:53.135676 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:31:53.135628    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-k5xg6\""
Jul 29 06:31:53.142758 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:31:53.142734    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-fwsh9"]
Jul 29 06:31:53.210025 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:31:53.209990    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h59gd\" (UniqueName: \"kubernetes.io/projected/56256e69-14ec-45c4-8619-59530c769e5d-kube-api-access-h59gd\") pod \"cert-manager-cainjector-559cdd8c87-fwsh9\" (UID: \"56256e69-14ec-45c4-8619-59530c769e5d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fwsh9"
Jul 29 06:31:53.210199 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:31:53.210088    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56256e69-14ec-45c4-8619-59530c769e5d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-fwsh9\" (UID: \"56256e69-14ec-45c4-8619-59530c769e5d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fwsh9"
Jul 29 06:31:53.311369 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:31:53.311322    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56256e69-14ec-45c4-8619-59530c769e5d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-fwsh9\" (UID: \"56256e69-14ec-45c4-8619-59530c769e5d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fwsh9"
Jul 29 06:31:53.311562 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:31:53.311414    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h59gd\" (UniqueName: \"kubernetes.io/projected/56256e69-14ec-45c4-8619-59530c769e5d-kube-api-access-h59gd\") pod \"cert-manager-cainjector-559cdd8c87-fwsh9\" (UID: \"56256e69-14ec-45c4-8619-59530c769e5d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fwsh9"
Jul 29 06:31:53.318630 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:31:53.318602    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56256e69-14ec-45c4-8619-59530c769e5d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-fwsh9\" (UID: \"56256e69-14ec-45c4-8619-59530c769e5d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fwsh9"
Jul 29 06:31:53.319143 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:31:53.319124    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h59gd\" (UniqueName: \"kubernetes.io/projected/56256e69-14ec-45c4-8619-59530c769e5d-kube-api-access-h59gd\") pod \"cert-manager-cainjector-559cdd8c87-fwsh9\" (UID: \"56256e69-14ec-45c4-8619-59530c769e5d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fwsh9"
Jul 29 06:31:53.457142 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:31:53.457058    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fwsh9"
Jul 29 06:31:53.582030 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:31:53.581973    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-fwsh9"]
Jul 29 06:31:53.584735 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:31:53.584710    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56256e69_14ec_45c4_8619_59530c769e5d.slice/crio-698bb35dc57fa02b9702ac0f6090f5770387a82d79abdc3ffbe6cd2e14c9e3df WatchSource:0}: Error finding container 698bb35dc57fa02b9702ac0f6090f5770387a82d79abdc3ffbe6cd2e14c9e3df: Status 404 returned error can't find the container with id 698bb35dc57fa02b9702ac0f6090f5770387a82d79abdc3ffbe6cd2e14c9e3df
Jul 29 06:31:54.361428 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:31:54.361392    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fwsh9" event={"ID":"56256e69-14ec-45c4-8619-59530c769e5d","Type":"ContainerStarted","Data":"698bb35dc57fa02b9702ac0f6090f5770387a82d79abdc3ffbe6cd2e14c9e3df"}
Jul 29 06:31:57.377512 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:31:57.377476    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fwsh9" event={"ID":"56256e69-14ec-45c4-8619-59530c769e5d","Type":"ContainerStarted","Data":"4badc0e4492f3eaff7e342ac3305001b0b955a032c0f94ef7da3c5b6ce8862bf"}
Jul 29 06:31:57.390415 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:31:57.390365    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fwsh9" podStartSLOduration=1.417016349 podStartE2EDuration="4.390346798s" totalImagesPullingTime="2.973330449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:31:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:31:57.389771518 +0000 UTC m=+560.467403285" watchObservedRunningTime="2026-07-29 06:31:57.390346798 +0000 UTC m=+560.467978592"
Jul 29 06:32:06.967868 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:06.967830    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bgbfp"]
Jul 29 06:32:06.971004 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:06.970988    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-bgbfp"
Jul 29 06:32:06.972734 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:06.972714    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-jq7vm\""
Jul 29 06:32:06.978139 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:06.978121    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bgbfp"]
Jul 29 06:32:07.027745 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:07.027708    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4665x\" (UniqueName: \"kubernetes.io/projected/a9304b17-be0a-44c1-b0e9-0d4506347e75-kube-api-access-4665x\") pod \"cert-manager-55b4d954d5-bgbfp\" (UID: \"a9304b17-be0a-44c1-b0e9-0d4506347e75\") " pod="cert-manager/cert-manager-55b4d954d5-bgbfp"
Jul 29 06:32:07.027911 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:07.027790    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a9304b17-be0a-44c1-b0e9-0d4506347e75-bound-sa-token\") pod \"cert-manager-55b4d954d5-bgbfp\" (UID: \"a9304b17-be0a-44c1-b0e9-0d4506347e75\") " pod="cert-manager/cert-manager-55b4d954d5-bgbfp"
Jul 29 06:32:07.128872 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:07.128821    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4665x\" (UniqueName: \"kubernetes.io/projected/a9304b17-be0a-44c1-b0e9-0d4506347e75-kube-api-access-4665x\") pod \"cert-manager-55b4d954d5-bgbfp\" (UID: \"a9304b17-be0a-44c1-b0e9-0d4506347e75\") " pod="cert-manager/cert-manager-55b4d954d5-bgbfp"
Jul 29 06:32:07.128997 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:07.128903    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a9304b17-be0a-44c1-b0e9-0d4506347e75-bound-sa-token\") pod \"cert-manager-55b4d954d5-bgbfp\" (UID: \"a9304b17-be0a-44c1-b0e9-0d4506347e75\") " pod="cert-manager/cert-manager-55b4d954d5-bgbfp"
Jul 29 06:32:07.136682 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:07.136650    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a9304b17-be0a-44c1-b0e9-0d4506347e75-bound-sa-token\") pod \"cert-manager-55b4d954d5-bgbfp\" (UID: \"a9304b17-be0a-44c1-b0e9-0d4506347e75\") " pod="cert-manager/cert-manager-55b4d954d5-bgbfp"
Jul 29 06:32:07.136682 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:07.136669    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4665x\" (UniqueName: \"kubernetes.io/projected/a9304b17-be0a-44c1-b0e9-0d4506347e75-kube-api-access-4665x\") pod \"cert-manager-55b4d954d5-bgbfp\" (UID: \"a9304b17-be0a-44c1-b0e9-0d4506347e75\") " pod="cert-manager/cert-manager-55b4d954d5-bgbfp"
Jul 29 06:32:07.280175 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:07.280101    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-bgbfp"
Jul 29 06:32:07.403704 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:07.403677    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bgbfp"]
Jul 29 06:32:07.406825 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:32:07.406799    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9304b17_be0a_44c1_b0e9_0d4506347e75.slice/crio-17855f6eed85a4e8feef30b32cb180c462d43b494859f7f78e529e1fc71fd1a1 WatchSource:0}: Error finding container 17855f6eed85a4e8feef30b32cb180c462d43b494859f7f78e529e1fc71fd1a1: Status 404 returned error can't find the container with id 17855f6eed85a4e8feef30b32cb180c462d43b494859f7f78e529e1fc71fd1a1
Jul 29 06:32:07.418822 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:07.418797    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-bgbfp" event={"ID":"a9304b17-be0a-44c1-b0e9-0d4506347e75","Type":"ContainerStarted","Data":"17855f6eed85a4e8feef30b32cb180c462d43b494859f7f78e529e1fc71fd1a1"}
Jul 29 06:32:08.425173 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:08.425136    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-bgbfp" event={"ID":"a9304b17-be0a-44c1-b0e9-0d4506347e75","Type":"ContainerStarted","Data":"b042f1b53ce0a6431f883246dfb0da0ea3a27ff855ab030c556be50aa7a52764"}
Jul 29 06:32:08.439242 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:08.439197    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-bgbfp" podStartSLOduration=2.439182664 podStartE2EDuration="2.439182664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:32:08.437298677 +0000 UTC m=+571.514930446" watchObservedRunningTime="2026-07-29 06:32:08.439182664 +0000 UTC m=+571.516814430"
Jul 29 06:32:22.812843 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:22.812802    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-t79x9"]
Jul 29 06:32:22.817700 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:22.817676    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-t79x9"
Jul 29 06:32:22.820504 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:22.820484    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 29 06:32:22.820773 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:22.820759    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 29 06:32:22.821214 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:22.821198    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 29 06:32:22.821318 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:22.821202    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 29 06:32:22.821608 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:22.821592    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-fdzx8\""
Jul 29 06:32:22.833462 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:22.833437    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-t79x9"]
Jul 29 06:32:22.833870 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:22.833830    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b7ac749e-0dfb-498c-9294-01f55a0f2bb2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d6c7945d7-t79x9\" (UID: \"b7ac749e-0dfb-498c-9294-01f55a0f2bb2\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-t79x9"
Jul 29 06:32:22.833945 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:22.833884    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b7ac749e-0dfb-498c-9294-01f55a0f2bb2-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d6c7945d7-t79x9\" (UID: \"b7ac749e-0dfb-498c-9294-01f55a0f2bb2\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-t79x9"
Jul 29 06:32:22.834001 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:22.833945    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7ac749e-0dfb-498c-9294-01f55a0f2bb2-tmp\") pod \"opendatahub-operator-controller-manager-7d6c7945d7-t79x9\" (UID: \"b7ac749e-0dfb-498c-9294-01f55a0f2bb2\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-t79x9"
Jul 29 06:32:22.834038 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:22.834009    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b7ac749e-0dfb-498c-9294-01f55a0f2bb2-manifests\") pod \"opendatahub-operator-controller-manager-7d6c7945d7-t79x9\" (UID: \"b7ac749e-0dfb-498c-9294-01f55a0f2bb2\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-t79x9"
Jul 29 06:32:22.834077 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:22.834050    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lgv8\" (UniqueName: \"kubernetes.io/projected/b7ac749e-0dfb-498c-9294-01f55a0f2bb2-kube-api-access-5lgv8\") pod \"opendatahub-operator-controller-manager-7d6c7945d7-t79x9\" (UID: \"b7ac749e-0dfb-498c-9294-01f55a0f2bb2\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-t79x9"
Jul 29 06:32:22.935061 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:22.935029    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b7ac749e-0dfb-498c-9294-01f55a0f2bb2-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d6c7945d7-t79x9\" (UID: \"b7ac749e-0dfb-498c-9294-01f55a0f2bb2\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-t79x9"
Jul 29 06:32:22.935253 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:22.935094    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7ac749e-0dfb-498c-9294-01f55a0f2bb2-tmp\") pod \"opendatahub-operator-controller-manager-7d6c7945d7-t79x9\" (UID: \"b7ac749e-0dfb-498c-9294-01f55a0f2bb2\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-t79x9"
Jul 29 06:32:22.935253 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:22.935118    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b7ac749e-0dfb-498c-9294-01f55a0f2bb2-manifests\") pod \"opendatahub-operator-controller-manager-7d6c7945d7-t79x9\" (UID: \"b7ac749e-0dfb-498c-9294-01f55a0f2bb2\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-t79x9"
Jul 29 06:32:22.935253 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:22.935152    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lgv8\" (UniqueName: \"kubernetes.io/projected/b7ac749e-0dfb-498c-9294-01f55a0f2bb2-kube-api-access-5lgv8\") pod \"opendatahub-operator-controller-manager-7d6c7945d7-t79x9\" (UID: \"b7ac749e-0dfb-498c-9294-01f55a0f2bb2\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-t79x9"
Jul 29 06:32:22.935253 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:22.935185    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b7ac749e-0dfb-498c-9294-01f55a0f2bb2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d6c7945d7-t79x9\" (UID: \"b7ac749e-0dfb-498c-9294-01f55a0f2bb2\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-t79x9"
Jul 29 06:32:22.935707 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:22.935669    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b7ac749e-0dfb-498c-9294-01f55a0f2bb2-manifests\") pod \"opendatahub-operator-controller-manager-7d6c7945d7-t79x9\" (UID: \"b7ac749e-0dfb-498c-9294-01f55a0f2bb2\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-t79x9"
Jul 29 06:32:22.938156 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:22.938132    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7ac749e-0dfb-498c-9294-01f55a0f2bb2-tmp\") pod \"opendatahub-operator-controller-manager-7d6c7945d7-t79x9\" (UID: \"b7ac749e-0dfb-498c-9294-01f55a0f2bb2\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-t79x9"
Jul 29 06:32:22.938292 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:22.938136    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b7ac749e-0dfb-498c-9294-01f55a0f2bb2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d6c7945d7-t79x9\" (UID: \"b7ac749e-0dfb-498c-9294-01f55a0f2bb2\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-t79x9"
Jul 29 06:32:22.938292 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:22.938263    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b7ac749e-0dfb-498c-9294-01f55a0f2bb2-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d6c7945d7-t79x9\" (UID: \"b7ac749e-0dfb-498c-9294-01f55a0f2bb2\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-t79x9"
Jul 29 06:32:22.943183 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:22.943165    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lgv8\" (UniqueName: \"kubernetes.io/projected/b7ac749e-0dfb-498c-9294-01f55a0f2bb2-kube-api-access-5lgv8\") pod \"opendatahub-operator-controller-manager-7d6c7945d7-t79x9\" (UID: \"b7ac749e-0dfb-498c-9294-01f55a0f2bb2\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-t79x9"
Jul 29 06:32:23.128371 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:23.128279    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-t79x9"
Jul 29 06:32:23.266459 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:23.266436    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-t79x9"]
Jul 29 06:32:23.269174 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:32:23.269145    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7ac749e_0dfb_498c_9294_01f55a0f2bb2.slice/crio-00e278f3b929a5edff2ad32757213778b3ce864bd4e7780cd1b8e5cf67914f07 WatchSource:0}: Error finding container 00e278f3b929a5edff2ad32757213778b3ce864bd4e7780cd1b8e5cf67914f07: Status 404 returned error can't find the container with id 00e278f3b929a5edff2ad32757213778b3ce864bd4e7780cd1b8e5cf67914f07
Jul 29 06:32:23.486190 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:23.486101    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-t79x9" event={"ID":"b7ac749e-0dfb-498c-9294-01f55a0f2bb2","Type":"ContainerStarted","Data":"00e278f3b929a5edff2ad32757213778b3ce864bd4e7780cd1b8e5cf67914f07"}
Jul 29 06:32:26.504375 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:26.504340    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-t79x9" event={"ID":"b7ac749e-0dfb-498c-9294-01f55a0f2bb2","Type":"ContainerStarted","Data":"9ec4d7450ec01fee1a14326cfaad6d54d131b21e453d3a98b74aa331564ab7a1"}
Jul 29 06:32:27.509960 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:27.509922    2563 generic.go:356] "Generic (PLEG): container finished" podID="b7ac749e-0dfb-498c-9294-01f55a0f2bb2" containerID="9ec4d7450ec01fee1a14326cfaad6d54d131b21e453d3a98b74aa331564ab7a1" exitCode=0
Jul 29 06:32:27.510357 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:27.510012    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-t79x9" event={"ID":"b7ac749e-0dfb-498c-9294-01f55a0f2bb2","Type":"ContainerDied","Data":"9ec4d7450ec01fee1a14326cfaad6d54d131b21e453d3a98b74aa331564ab7a1"}
Jul 29 06:32:28.516065 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:28.516031    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-t79x9" event={"ID":"b7ac749e-0dfb-498c-9294-01f55a0f2bb2","Type":"ContainerStarted","Data":"8edf4bd187d7b5e5c77a87509bac2f63ea28d64cc1ed3f3a6ed5f9cf96267fed"}
Jul 29 06:32:28.516528 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:28.516130    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-t79x9"
Jul 29 06:32:28.535411 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:28.535358    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-t79x9" podStartSLOduration=2.031610268 podStartE2EDuration="6.53533817s" totalImagesPullingTime="3.409678704s" totalInitContainerRuntime="1.094049198s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:32:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:32:28.534509868 +0000 UTC m=+591.612141634" watchObservedRunningTime="2026-07-29 06:32:28.53533817 +0000 UTC m=+591.612969941"
Jul 29 06:32:39.523007 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:39.522980    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7d6c7945d7-t79x9"
Jul 29 06:32:39.663303 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:39.663271    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-68c9b6dddc-gk4hv"]
Jul 29 06:32:39.665518 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:39.665503    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-gk4hv"
Jul 29 06:32:39.667676 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:39.667654    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 29 06:32:39.667805 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:39.667714    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 29 06:32:39.668013 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:39.667994    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-8wnsf\""
Jul 29 06:32:39.668075 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:39.668038    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 29 06:32:39.668321 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:39.668304    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 29 06:32:39.670212 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:39.670194    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 29 06:32:39.674747 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:39.674723    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68c9b6dddc-gk4hv"]
Jul 29 06:32:39.782819 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:39.782720    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9862f729-e68d-4647-bdf1-9fac3162c4de-cert\") pod \"lws-controller-manager-68c9b6dddc-gk4hv\" (UID: \"9862f729-e68d-4647-bdf1-9fac3162c4de\") " pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-gk4hv"
Jul 29 06:32:39.782819 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:39.782768    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h44x5\" (UniqueName: \"kubernetes.io/projected/9862f729-e68d-4647-bdf1-9fac3162c4de-kube-api-access-h44x5\") pod \"lws-controller-manager-68c9b6dddc-gk4hv\" (UID: \"9862f729-e68d-4647-bdf1-9fac3162c4de\") " pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-gk4hv"
Jul 29 06:32:39.783034 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:39.782842    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9862f729-e68d-4647-bdf1-9fac3162c4de-manager-config\") pod \"lws-controller-manager-68c9b6dddc-gk4hv\" (UID: \"9862f729-e68d-4647-bdf1-9fac3162c4de\") " pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-gk4hv"
Jul 29 06:32:39.783034 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:39.782931    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9862f729-e68d-4647-bdf1-9fac3162c4de-metrics-cert\") pod \"lws-controller-manager-68c9b6dddc-gk4hv\" (UID: \"9862f729-e68d-4647-bdf1-9fac3162c4de\") " pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-gk4hv"
Jul 29 06:32:39.883797 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:39.883766    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9862f729-e68d-4647-bdf1-9fac3162c4de-cert\") pod \"lws-controller-manager-68c9b6dddc-gk4hv\" (UID: \"9862f729-e68d-4647-bdf1-9fac3162c4de\") " pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-gk4hv"
Jul 29 06:32:39.883978 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:39.883805    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h44x5\" (UniqueName: \"kubernetes.io/projected/9862f729-e68d-4647-bdf1-9fac3162c4de-kube-api-access-h44x5\") pod \"lws-controller-manager-68c9b6dddc-gk4hv\" (UID: \"9862f729-e68d-4647-bdf1-9fac3162c4de\") " pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-gk4hv"
Jul 29 06:32:39.883978 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:39.883949    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9862f729-e68d-4647-bdf1-9fac3162c4de-manager-config\") pod \"lws-controller-manager-68c9b6dddc-gk4hv\" (UID: \"9862f729-e68d-4647-bdf1-9fac3162c4de\") " pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-gk4hv"
Jul 29 06:32:39.884071 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:39.884043    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9862f729-e68d-4647-bdf1-9fac3162c4de-metrics-cert\") pod \"lws-controller-manager-68c9b6dddc-gk4hv\" (UID: \"9862f729-e68d-4647-bdf1-9fac3162c4de\") " pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-gk4hv"
Jul 29 06:32:39.884600 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:39.884569    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9862f729-e68d-4647-bdf1-9fac3162c4de-manager-config\") pod \"lws-controller-manager-68c9b6dddc-gk4hv\" (UID: \"9862f729-e68d-4647-bdf1-9fac3162c4de\") " pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-gk4hv"
Jul 29 06:32:39.886642 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:39.886621    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9862f729-e68d-4647-bdf1-9fac3162c4de-cert\") pod \"lws-controller-manager-68c9b6dddc-gk4hv\" (UID: \"9862f729-e68d-4647-bdf1-9fac3162c4de\") " pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-gk4hv"
Jul 29 06:32:39.886642 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:39.886632    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9862f729-e68d-4647-bdf1-9fac3162c4de-metrics-cert\") pod \"lws-controller-manager-68c9b6dddc-gk4hv\" (UID: \"9862f729-e68d-4647-bdf1-9fac3162c4de\") " pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-gk4hv"
Jul 29 06:32:39.897711 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:39.897684    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h44x5\" (UniqueName: \"kubernetes.io/projected/9862f729-e68d-4647-bdf1-9fac3162c4de-kube-api-access-h44x5\") pod \"lws-controller-manager-68c9b6dddc-gk4hv\" (UID: \"9862f729-e68d-4647-bdf1-9fac3162c4de\") " pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-gk4hv"
Jul 29 06:32:39.975441 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:39.975410    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-gk4hv"
Jul 29 06:32:40.105001 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:40.104976    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68c9b6dddc-gk4hv"]
Jul 29 06:32:40.107583 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:32:40.107555    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9862f729_e68d_4647_bdf1_9fac3162c4de.slice/crio-aa96465c1a2fd4b74de6a4356ca733d13fb8c5a7a0543c2aef56ec40e69448f6 WatchSource:0}: Error finding container aa96465c1a2fd4b74de6a4356ca733d13fb8c5a7a0543c2aef56ec40e69448f6: Status 404 returned error can't find the container with id aa96465c1a2fd4b74de6a4356ca733d13fb8c5a7a0543c2aef56ec40e69448f6
Jul 29 06:32:40.573528 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:40.573493    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-gk4hv" event={"ID":"9862f729-e68d-4647-bdf1-9fac3162c4de","Type":"ContainerStarted","Data":"aa96465c1a2fd4b74de6a4356ca733d13fb8c5a7a0543c2aef56ec40e69448f6"}
Jul 29 06:32:43.595308 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:43.595272    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-gk4hv" event={"ID":"9862f729-e68d-4647-bdf1-9fac3162c4de","Type":"ContainerStarted","Data":"5af0af193d34653a0f143d7cca1bdbd74068f3757999d3a4a1fc70188df87e66"}
Jul 29 06:32:43.595715 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:43.595387    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-gk4hv"
Jul 29 06:32:43.608768 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:43.608682    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-gk4hv" podStartSLOduration=2.101959465 podStartE2EDuration="4.608659227s" totalImagesPullingTime="2.506699762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:32:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:32:43.608193804 +0000 UTC m=+606.685825570" watchObservedRunningTime="2026-07-29 06:32:43.608659227 +0000 UTC m=+606.686290996"
Jul 29 06:32:52.965933 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:52.965900    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5c857974d4-g2dnm"]
Jul 29 06:32:52.971260 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:52.971237    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5c857974d4-g2dnm"
Jul 29 06:32:52.974391 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:52.974366    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 29 06:32:52.974517 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:52.974476    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-qnzwk\""
Jul 29 06:32:52.974586 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:52.974518    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 29 06:32:52.975098 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:52.975081    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 29 06:32:52.975309 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:52.975291    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 29 06:32:52.996297 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:52.996268    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5c857974d4-g2dnm"]
Jul 29 06:32:53.109280 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:53.109239    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ldfz\" (UniqueName: \"kubernetes.io/projected/d60bb996-2dc5-4113-b14c-131e20292ed9-kube-api-access-9ldfz\") pod \"kube-auth-proxy-5c857974d4-g2dnm\" (UID: \"d60bb996-2dc5-4113-b14c-131e20292ed9\") " pod="openshift-ingress/kube-auth-proxy-5c857974d4-g2dnm"
Jul 29 06:32:53.109460 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:53.109321    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d60bb996-2dc5-4113-b14c-131e20292ed9-tmp\") pod \"kube-auth-proxy-5c857974d4-g2dnm\" (UID: \"d60bb996-2dc5-4113-b14c-131e20292ed9\") " pod="openshift-ingress/kube-auth-proxy-5c857974d4-g2dnm"
Jul 29 06:32:53.109460 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:53.109352    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d60bb996-2dc5-4113-b14c-131e20292ed9-tls-certs\") pod \"kube-auth-proxy-5c857974d4-g2dnm\" (UID: \"d60bb996-2dc5-4113-b14c-131e20292ed9\") " pod="openshift-ingress/kube-auth-proxy-5c857974d4-g2dnm"
Jul 29 06:32:53.210513 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:53.210464    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ldfz\" (UniqueName: \"kubernetes.io/projected/d60bb996-2dc5-4113-b14c-131e20292ed9-kube-api-access-9ldfz\") pod \"kube-auth-proxy-5c857974d4-g2dnm\" (UID: \"d60bb996-2dc5-4113-b14c-131e20292ed9\") " pod="openshift-ingress/kube-auth-proxy-5c857974d4-g2dnm"
Jul 29 06:32:53.210696 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:53.210536    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d60bb996-2dc5-4113-b14c-131e20292ed9-tmp\") pod \"kube-auth-proxy-5c857974d4-g2dnm\" (UID: \"d60bb996-2dc5-4113-b14c-131e20292ed9\") " pod="openshift-ingress/kube-auth-proxy-5c857974d4-g2dnm"
Jul 29 06:32:53.210696 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:53.210565    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d60bb996-2dc5-4113-b14c-131e20292ed9-tls-certs\") pod \"kube-auth-proxy-5c857974d4-g2dnm\" (UID: \"d60bb996-2dc5-4113-b14c-131e20292ed9\") " pod="openshift-ingress/kube-auth-proxy-5c857974d4-g2dnm"
Jul 29 06:32:53.210696 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:32:53.210671    2563 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 29 06:32:53.210798 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:32:53.210734    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d60bb996-2dc5-4113-b14c-131e20292ed9-tls-certs podName:d60bb996-2dc5-4113-b14c-131e20292ed9 nodeName:}" failed. No retries permitted until 2026-07-29 06:32:53.710718625 +0000 UTC m=+616.788350370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d60bb996-2dc5-4113-b14c-131e20292ed9-tls-certs") pod "kube-auth-proxy-5c857974d4-g2dnm" (UID: "d60bb996-2dc5-4113-b14c-131e20292ed9") : secret "kube-auth-proxy-tls" not found
Jul 29 06:32:53.212986 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:53.212962    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d60bb996-2dc5-4113-b14c-131e20292ed9-tmp\") pod \"kube-auth-proxy-5c857974d4-g2dnm\" (UID: \"d60bb996-2dc5-4113-b14c-131e20292ed9\") " pod="openshift-ingress/kube-auth-proxy-5c857974d4-g2dnm"
Jul 29 06:32:53.219680 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:53.219616    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ldfz\" (UniqueName: \"kubernetes.io/projected/d60bb996-2dc5-4113-b14c-131e20292ed9-kube-api-access-9ldfz\") pod \"kube-auth-proxy-5c857974d4-g2dnm\" (UID: \"d60bb996-2dc5-4113-b14c-131e20292ed9\") " pod="openshift-ingress/kube-auth-proxy-5c857974d4-g2dnm"
Jul 29 06:32:53.716793 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:53.716754    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d60bb996-2dc5-4113-b14c-131e20292ed9-tls-certs\") pod \"kube-auth-proxy-5c857974d4-g2dnm\" (UID: \"d60bb996-2dc5-4113-b14c-131e20292ed9\") " pod="openshift-ingress/kube-auth-proxy-5c857974d4-g2dnm"
Jul 29 06:32:53.719316 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:53.719290    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d60bb996-2dc5-4113-b14c-131e20292ed9-tls-certs\") pod \"kube-auth-proxy-5c857974d4-g2dnm\" (UID: \"d60bb996-2dc5-4113-b14c-131e20292ed9\") " pod="openshift-ingress/kube-auth-proxy-5c857974d4-g2dnm"
Jul 29 06:32:53.881824 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:53.881788    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5c857974d4-g2dnm"
Jul 29 06:32:54.029216 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:54.029183    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5c857974d4-g2dnm"]
Jul 29 06:32:54.032502 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:32:54.032475    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd60bb996_2dc5_4113_b14c_131e20292ed9.slice/crio-08650b25c81ca49067fbe6bd09203ba4e7e5bc8619a177813e8a97529eab2eec WatchSource:0}: Error finding container 08650b25c81ca49067fbe6bd09203ba4e7e5bc8619a177813e8a97529eab2eec: Status 404 returned error can't find the container with id 08650b25c81ca49067fbe6bd09203ba4e7e5bc8619a177813e8a97529eab2eec
Jul 29 06:32:54.601978 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:54.601952    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-68c9b6dddc-gk4hv"
Jul 29 06:32:54.645501 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:54.645466    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5c857974d4-g2dnm" event={"ID":"d60bb996-2dc5-4113-b14c-131e20292ed9","Type":"ContainerStarted","Data":"08650b25c81ca49067fbe6bd09203ba4e7e5bc8619a177813e8a97529eab2eec"}
Jul 29 06:32:57.671319 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:57.671282    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5c857974d4-g2dnm" event={"ID":"d60bb996-2dc5-4113-b14c-131e20292ed9","Type":"ContainerStarted","Data":"534fb952e8c7c9957fcf075a64fb2aa83904e6b9b447118a44b1cb43159280c1"}
Jul 29 06:32:57.686760 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:32:57.686715    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5c857974d4-g2dnm" podStartSLOduration=2.767804374 podStartE2EDuration="5.686700858s" totalImagesPullingTime="2.918896484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:32:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:32:57.6851329 +0000 UTC m=+620.762764670" watchObservedRunningTime="2026-07-29 06:32:57.686700858 +0000 UTC m=+620.764332624"
Jul 29 06:35:29.090843 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:29.090812    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ktdkd"]
Jul 29 06:35:29.094114 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:29.094098    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-ktdkd"
Jul 29 06:35:29.096158 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:29.096131    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 29 06:35:29.096158 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:29.096143    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 29 06:35:29.096804 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:29.096788    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 29 06:35:29.096908 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:29.096894    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6llvq\""
Jul 29 06:35:29.103489 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:29.103463    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ktdkd"]
Jul 29 06:35:29.189467 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:29.189433    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ktdkd"]
Jul 29 06:35:29.242991 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:29.242953    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62l6b\" (UniqueName: \"kubernetes.io/projected/966b170d-3214-4248-b9a7-96b12070ba94-kube-api-access-62l6b\") pod \"limitador-limitador-64c94fd64f-ktdkd\" (UID: \"966b170d-3214-4248-b9a7-96b12070ba94\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ktdkd"
Jul 29 06:35:29.242991 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:29.242996    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/966b170d-3214-4248-b9a7-96b12070ba94-config-file\") pod \"limitador-limitador-64c94fd64f-ktdkd\" (UID: \"966b170d-3214-4248-b9a7-96b12070ba94\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ktdkd"
Jul 29 06:35:29.343519 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:29.343438    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62l6b\" (UniqueName: \"kubernetes.io/projected/966b170d-3214-4248-b9a7-96b12070ba94-kube-api-access-62l6b\") pod \"limitador-limitador-64c94fd64f-ktdkd\" (UID: \"966b170d-3214-4248-b9a7-96b12070ba94\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ktdkd"
Jul 29 06:35:29.343519 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:29.343473    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/966b170d-3214-4248-b9a7-96b12070ba94-config-file\") pod \"limitador-limitador-64c94fd64f-ktdkd\" (UID: \"966b170d-3214-4248-b9a7-96b12070ba94\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ktdkd"
Jul 29 06:35:29.344084 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:29.344067    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/966b170d-3214-4248-b9a7-96b12070ba94-config-file\") pod \"limitador-limitador-64c94fd64f-ktdkd\" (UID: \"966b170d-3214-4248-b9a7-96b12070ba94\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ktdkd"
Jul 29 06:35:29.351665 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:29.351641    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62l6b\" (UniqueName: \"kubernetes.io/projected/966b170d-3214-4248-b9a7-96b12070ba94-kube-api-access-62l6b\") pod \"limitador-limitador-64c94fd64f-ktdkd\" (UID: \"966b170d-3214-4248-b9a7-96b12070ba94\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ktdkd"
Jul 29 06:35:29.410065 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:29.410032    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-ktdkd"
Jul 29 06:35:29.537554 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:29.537431    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ktdkd"]
Jul 29 06:35:29.540289 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:35:29.540258    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod966b170d_3214_4248_b9a7_96b12070ba94.slice/crio-ad08c2e6ab4e08c6644df06fbdf040386ab1a562b64d5836c168d87dab5dbfab WatchSource:0}: Error finding container ad08c2e6ab4e08c6644df06fbdf040386ab1a562b64d5836c168d87dab5dbfab: Status 404 returned error can't find the container with id ad08c2e6ab4e08c6644df06fbdf040386ab1a562b64d5836c168d87dab5dbfab
Jul 29 06:35:29.542151 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:29.542133    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 06:35:29.966840 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:29.966803    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-4rgqw"]
Jul 29 06:35:29.971642 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:29.971618    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-4rgqw"
Jul 29 06:35:29.973737 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:29.973713    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-82x8j\""
Jul 29 06:35:29.974944 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:29.974922    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-4rgqw"]
Jul 29 06:35:30.050230 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:30.050198    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mts5m\" (UniqueName: \"kubernetes.io/projected/784add0f-49b7-4618-9c87-d988af95969b-kube-api-access-mts5m\") pod \"authorino-75d556bd77-4rgqw\" (UID: \"784add0f-49b7-4618-9c87-d988af95969b\") " pod="kuadrant-system/authorino-75d556bd77-4rgqw"
Jul 29 06:35:30.151585 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:30.151551    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mts5m\" (UniqueName: \"kubernetes.io/projected/784add0f-49b7-4618-9c87-d988af95969b-kube-api-access-mts5m\") pod \"authorino-75d556bd77-4rgqw\" (UID: \"784add0f-49b7-4618-9c87-d988af95969b\") " pod="kuadrant-system/authorino-75d556bd77-4rgqw"
Jul 29 06:35:30.159220 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:30.159187    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mts5m\" (UniqueName: \"kubernetes.io/projected/784add0f-49b7-4618-9c87-d988af95969b-kube-api-access-mts5m\") pod \"authorino-75d556bd77-4rgqw\" (UID: \"784add0f-49b7-4618-9c87-d988af95969b\") " pod="kuadrant-system/authorino-75d556bd77-4rgqw"
Jul 29 06:35:30.282867 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:30.282572    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-4rgqw"
Jul 29 06:35:30.372628 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:30.372589    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-ktdkd" event={"ID":"966b170d-3214-4248-b9a7-96b12070ba94","Type":"ContainerStarted","Data":"ad08c2e6ab4e08c6644df06fbdf040386ab1a562b64d5836c168d87dab5dbfab"}
Jul 29 06:35:30.466988 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:30.466960    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-4rgqw"]
Jul 29 06:35:30.472721 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:35:30.471991    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod784add0f_49b7_4618_9c87_d988af95969b.slice/crio-dfacdfdd4cc180a9cf23f0d2d97114a70f370a5a5336310ec19157c6ad188daf WatchSource:0}: Error finding container dfacdfdd4cc180a9cf23f0d2d97114a70f370a5a5336310ec19157c6ad188daf: Status 404 returned error can't find the container with id dfacdfdd4cc180a9cf23f0d2d97114a70f370a5a5336310ec19157c6ad188daf
Jul 29 06:35:31.382279 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:31.382215    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-4rgqw" event={"ID":"784add0f-49b7-4618-9c87-d988af95969b","Type":"ContainerStarted","Data":"dfacdfdd4cc180a9cf23f0d2d97114a70f370a5a5336310ec19157c6ad188daf"}
Jul 29 06:35:34.400457 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:34.400372    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-ktdkd" event={"ID":"966b170d-3214-4248-b9a7-96b12070ba94","Type":"ContainerStarted","Data":"97929f27d1e6536a3d8058eacb0319cf6b745c8d715ffb9cad1a8a4dc3edef15"}
Jul 29 06:35:34.400457 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:34.400423    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-ktdkd"
Jul 29 06:35:34.401806 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:34.401783    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-4rgqw" event={"ID":"784add0f-49b7-4618-9c87-d988af95969b","Type":"ContainerStarted","Data":"a2cc6e1ad86532ef325f03b759a8088376800fa2eda5896c3f02691e8c8d6159"}
Jul 29 06:35:34.415142 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:34.415100    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-ktdkd" podStartSLOduration=0.838765291 podStartE2EDuration="5.415086729s" totalImagesPullingTime="4.576321438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:35:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:35:34.414053631 +0000 UTC m=+777.491685397" watchObservedRunningTime="2026-07-29 06:35:34.415086729 +0000 UTC m=+777.492718495"
Jul 29 06:35:34.427003 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:34.426966    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-4rgqw" podStartSLOduration=1.778606495 podStartE2EDuration="5.426952733s" totalImagesPullingTime="3.648346238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:35:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:35:34.425408181 +0000 UTC m=+777.503039958" watchObservedRunningTime="2026-07-29 06:35:34.426952733 +0000 UTC m=+777.504584499"
Jul 29 06:35:44.213216 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:44.213180    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ktdkd"]
Jul 29 06:35:44.213661 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:44.213373    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-ktdkd" podUID="966b170d-3214-4248-b9a7-96b12070ba94" containerName="limitador" containerID="cri-o://97929f27d1e6536a3d8058eacb0319cf6b745c8d715ffb9cad1a8a4dc3edef15" gracePeriod=30
Jul 29 06:35:44.214085 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:44.214009    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-ktdkd"
Jul 29 06:35:44.757948 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:44.757928    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-ktdkd"
Jul 29 06:35:44.879869 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:44.879834    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/966b170d-3214-4248-b9a7-96b12070ba94-config-file\") pod \"966b170d-3214-4248-b9a7-96b12070ba94\" (UID: \"966b170d-3214-4248-b9a7-96b12070ba94\") "
Jul 29 06:35:44.880016 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:44.879898    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62l6b\" (UniqueName: \"kubernetes.io/projected/966b170d-3214-4248-b9a7-96b12070ba94-kube-api-access-62l6b\") pod \"966b170d-3214-4248-b9a7-96b12070ba94\" (UID: \"966b170d-3214-4248-b9a7-96b12070ba94\") "
Jul 29 06:35:44.880238 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:44.880213    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/966b170d-3214-4248-b9a7-96b12070ba94-config-file" (OuterVolumeSpecName: "config-file") pod "966b170d-3214-4248-b9a7-96b12070ba94" (UID: "966b170d-3214-4248-b9a7-96b12070ba94"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 06:35:44.882193 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:44.882171    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/966b170d-3214-4248-b9a7-96b12070ba94-kube-api-access-62l6b" (OuterVolumeSpecName: "kube-api-access-62l6b") pod "966b170d-3214-4248-b9a7-96b12070ba94" (UID: "966b170d-3214-4248-b9a7-96b12070ba94"). InnerVolumeSpecName "kube-api-access-62l6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 06:35:44.980789 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:44.980760    2563 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/966b170d-3214-4248-b9a7-96b12070ba94-config-file\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:35:44.980789 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:44.980785    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62l6b\" (UniqueName: \"kubernetes.io/projected/966b170d-3214-4248-b9a7-96b12070ba94-kube-api-access-62l6b\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:35:45.450525 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:45.450494    2563 generic.go:356] "Generic (PLEG): container finished" podID="966b170d-3214-4248-b9a7-96b12070ba94" containerID="97929f27d1e6536a3d8058eacb0319cf6b745c8d715ffb9cad1a8a4dc3edef15" exitCode=0
Jul 29 06:35:45.450525 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:45.450530    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-ktdkd" event={"ID":"966b170d-3214-4248-b9a7-96b12070ba94","Type":"ContainerDied","Data":"97929f27d1e6536a3d8058eacb0319cf6b745c8d715ffb9cad1a8a4dc3edef15"}
Jul 29 06:35:45.451054 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:45.450544    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-ktdkd"
Jul 29 06:35:45.451054 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:45.450566    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-ktdkd" event={"ID":"966b170d-3214-4248-b9a7-96b12070ba94","Type":"ContainerDied","Data":"ad08c2e6ab4e08c6644df06fbdf040386ab1a562b64d5836c168d87dab5dbfab"}
Jul 29 06:35:45.451054 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:45.450586    2563 scope.go:117] "RemoveContainer" containerID="97929f27d1e6536a3d8058eacb0319cf6b745c8d715ffb9cad1a8a4dc3edef15"
Jul 29 06:35:45.458976 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:45.458958    2563 scope.go:117] "RemoveContainer" containerID="97929f27d1e6536a3d8058eacb0319cf6b745c8d715ffb9cad1a8a4dc3edef15"
Jul 29 06:35:45.459242 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:35:45.459214    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97929f27d1e6536a3d8058eacb0319cf6b745c8d715ffb9cad1a8a4dc3edef15\": container with ID starting with 97929f27d1e6536a3d8058eacb0319cf6b745c8d715ffb9cad1a8a4dc3edef15 not found: ID does not exist" containerID="97929f27d1e6536a3d8058eacb0319cf6b745c8d715ffb9cad1a8a4dc3edef15"
Jul 29 06:35:45.459341 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:45.459244    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97929f27d1e6536a3d8058eacb0319cf6b745c8d715ffb9cad1a8a4dc3edef15"} err="failed to get container status \"97929f27d1e6536a3d8058eacb0319cf6b745c8d715ffb9cad1a8a4dc3edef15\": rpc error: code = NotFound desc = could not find container \"97929f27d1e6536a3d8058eacb0319cf6b745c8d715ffb9cad1a8a4dc3edef15\": container with ID starting with 97929f27d1e6536a3d8058eacb0319cf6b745c8d715ffb9cad1a8a4dc3edef15 not found: ID does not exist"
Jul 29 06:35:45.464695 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:45.464675    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ktdkd"]
Jul 29 06:35:45.467649 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:45.467630    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ktdkd"]
Jul 29 06:35:47.397460 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:47.397417    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="966b170d-3214-4248-b9a7-96b12070ba94" path="/var/lib/kubelet/pods/966b170d-3214-4248-b9a7-96b12070ba94/volumes"
Jul 29 06:35:49.583131 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:49.583089    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-2dm84"]
Jul 29 06:35:49.583655 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:49.583635    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="966b170d-3214-4248-b9a7-96b12070ba94" containerName="limitador"
Jul 29 06:35:49.583742 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:49.583659    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="966b170d-3214-4248-b9a7-96b12070ba94" containerName="limitador"
Jul 29 06:35:49.583816 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:49.583803    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="966b170d-3214-4248-b9a7-96b12070ba94" containerName="limitador"
Jul 29 06:35:49.588442 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:49.588423    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2dm84"
Jul 29 06:35:49.590807 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:49.590777    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 29 06:35:49.591270 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:49.591176    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-6674t\""
Jul 29 06:35:49.591270 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:49.591235    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 06:35:49.591455 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:49.591235    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 06:35:49.592652 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:49.592629    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-2dm84"]
Jul 29 06:35:49.621595 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:49.621569    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bd49080d-e4f0-4620-8e4c-0632e2d426e2-data\") pod \"postgres-69fb4594cf-2dm84\" (UID: \"bd49080d-e4f0-4620-8e4c-0632e2d426e2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2dm84"
Jul 29 06:35:49.621688 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:49.621623    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c96vf\" (UniqueName: \"kubernetes.io/projected/bd49080d-e4f0-4620-8e4c-0632e2d426e2-kube-api-access-c96vf\") pod \"postgres-69fb4594cf-2dm84\" (UID: \"bd49080d-e4f0-4620-8e4c-0632e2d426e2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2dm84"
Jul 29 06:35:49.722719 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:49.722691    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bd49080d-e4f0-4620-8e4c-0632e2d426e2-data\") pod \"postgres-69fb4594cf-2dm84\" (UID: \"bd49080d-e4f0-4620-8e4c-0632e2d426e2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2dm84"
Jul 29 06:35:49.722912 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:49.722733    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c96vf\" (UniqueName: \"kubernetes.io/projected/bd49080d-e4f0-4620-8e4c-0632e2d426e2-kube-api-access-c96vf\") pod \"postgres-69fb4594cf-2dm84\" (UID: \"bd49080d-e4f0-4620-8e4c-0632e2d426e2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2dm84"
Jul 29 06:35:49.723186 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:49.723164    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bd49080d-e4f0-4620-8e4c-0632e2d426e2-data\") pod \"postgres-69fb4594cf-2dm84\" (UID: \"bd49080d-e4f0-4620-8e4c-0632e2d426e2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2dm84"
Jul 29 06:35:49.730667 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:49.730646    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c96vf\" (UniqueName: \"kubernetes.io/projected/bd49080d-e4f0-4620-8e4c-0632e2d426e2-kube-api-access-c96vf\") pod \"postgres-69fb4594cf-2dm84\" (UID: \"bd49080d-e4f0-4620-8e4c-0632e2d426e2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2dm84"
Jul 29 06:35:49.901961 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:49.901842    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2dm84"
Jul 29 06:35:50.032590 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:50.032534    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-2dm84"]
Jul 29 06:35:50.034941 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:35:50.034915    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd49080d_e4f0_4620_8e4c_0632e2d426e2.slice/crio-721bd4c8a938f2e039d627bcd0320da0fa8958f45d1d09a334e8118f786f88b6 WatchSource:0}: Error finding container 721bd4c8a938f2e039d627bcd0320da0fa8958f45d1d09a334e8118f786f88b6: Status 404 returned error can't find the container with id 721bd4c8a938f2e039d627bcd0320da0fa8958f45d1d09a334e8118f786f88b6
Jul 29 06:35:50.475020 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:50.474986    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2dm84" event={"ID":"bd49080d-e4f0-4620-8e4c-0632e2d426e2","Type":"ContainerStarted","Data":"721bd4c8a938f2e039d627bcd0320da0fa8958f45d1d09a334e8118f786f88b6"}
Jul 29 06:35:55.506178 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:55.506143    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2dm84" event={"ID":"bd49080d-e4f0-4620-8e4c-0632e2d426e2","Type":"ContainerStarted","Data":"55b53d6ecfec61d8b8df5ae2af9671d2909d6d9e0203e9a6fd45f62f42ec9568"}
Jul 29 06:35:55.506602 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:55.506272    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2dm84"
Jul 29 06:35:55.521187 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:35:55.521135    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2dm84" podStartSLOduration=1.707333346 podStartE2EDuration="6.521121001s" totalImagesPullingTime="4.813787655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:35:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:35:55.518657573 +0000 UTC m=+798.596289341" watchObservedRunningTime="2026-07-29 06:35:55.521121001 +0000 UTC m=+798.598752793"
Jul 29 06:36:01.540203 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:01.540129    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2dm84"
Jul 29 06:36:02.327310 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:02.327275    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-84c55877b8-wqxh8"]
Jul 29 06:36:02.330508 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:02.330489    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84c55877b8-wqxh8"
Jul 29 06:36:02.334218 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:02.334197    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84c55877b8-wqxh8"]
Jul 29 06:36:02.348234 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:02.348211    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-84c55877b8-wqxh8"]
Jul 29 06:36:02.348475 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:36:02.348457    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-6dlws], unattached volumes=[], failed to process volumes=[kube-api-access-6dlws]: context canceled" pod="kuadrant-system/authorino-84c55877b8-wqxh8" podUID="d7f2a037-496b-496d-aa5f-f9610591bd01"
Jul 29 06:36:02.432842 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:02.432811    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dlws\" (UniqueName: \"kubernetes.io/projected/d7f2a037-496b-496d-aa5f-f9610591bd01-kube-api-access-6dlws\") pod \"authorino-84c55877b8-wqxh8\" (UID: \"d7f2a037-496b-496d-aa5f-f9610591bd01\") " pod="kuadrant-system/authorino-84c55877b8-wqxh8"
Jul 29 06:36:02.534335 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:02.534310    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dlws\" (UniqueName: \"kubernetes.io/projected/d7f2a037-496b-496d-aa5f-f9610591bd01-kube-api-access-6dlws\") pod \"authorino-84c55877b8-wqxh8\" (UID: \"d7f2a037-496b-496d-aa5f-f9610591bd01\") " pod="kuadrant-system/authorino-84c55877b8-wqxh8"
Jul 29 06:36:02.537782 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:02.537761    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84c55877b8-wqxh8"
Jul 29 06:36:02.541538 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:02.541514    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dlws\" (UniqueName: \"kubernetes.io/projected/d7f2a037-496b-496d-aa5f-f9610591bd01-kube-api-access-6dlws\") pod \"authorino-84c55877b8-wqxh8\" (UID: \"d7f2a037-496b-496d-aa5f-f9610591bd01\") " pod="kuadrant-system/authorino-84c55877b8-wqxh8"
Jul 29 06:36:02.544277 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:02.544261    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84c55877b8-wqxh8"
Jul 29 06:36:02.635239 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:02.635164    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dlws\" (UniqueName: \"kubernetes.io/projected/d7f2a037-496b-496d-aa5f-f9610591bd01-kube-api-access-6dlws\") pod \"d7f2a037-496b-496d-aa5f-f9610591bd01\" (UID: \"d7f2a037-496b-496d-aa5f-f9610591bd01\") "
Jul 29 06:36:02.637359 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:02.637339    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f2a037-496b-496d-aa5f-f9610591bd01-kube-api-access-6dlws" (OuterVolumeSpecName: "kube-api-access-6dlws") pod "d7f2a037-496b-496d-aa5f-f9610591bd01" (UID: "d7f2a037-496b-496d-aa5f-f9610591bd01"). InnerVolumeSpecName "kube-api-access-6dlws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 06:36:02.736067 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:02.736033    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dlws\" (UniqueName: \"kubernetes.io/projected/d7f2a037-496b-496d-aa5f-f9610591bd01-kube-api-access-6dlws\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:36:03.542400 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:03.542368    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84c55877b8-wqxh8"
Jul 29 06:36:03.563551 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:03.563516    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-84c55877b8-wqxh8"]
Jul 29 06:36:03.566825 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:03.566801    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-84c55877b8-wqxh8"]
Jul 29 06:36:03.872684 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:03.872652    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-4rgqw"]
Jul 29 06:36:03.872863 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:03.872818    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-4rgqw" podUID="784add0f-49b7-4618-9c87-d988af95969b" containerName="authorino" containerID="cri-o://a2cc6e1ad86532ef325f03b759a8088376800fa2eda5896c3f02691e8c8d6159" gracePeriod=30
Jul 29 06:36:04.133436 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:04.133378    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-4rgqw"
Jul 29 06:36:04.249111 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:04.249081    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mts5m\" (UniqueName: \"kubernetes.io/projected/784add0f-49b7-4618-9c87-d988af95969b-kube-api-access-mts5m\") pod \"784add0f-49b7-4618-9c87-d988af95969b\" (UID: \"784add0f-49b7-4618-9c87-d988af95969b\") "
Jul 29 06:36:04.251281 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:04.251251    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784add0f-49b7-4618-9c87-d988af95969b-kube-api-access-mts5m" (OuterVolumeSpecName: "kube-api-access-mts5m") pod "784add0f-49b7-4618-9c87-d988af95969b" (UID: "784add0f-49b7-4618-9c87-d988af95969b"). InnerVolumeSpecName "kube-api-access-mts5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 06:36:04.350581 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:04.350552    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mts5m\" (UniqueName: \"kubernetes.io/projected/784add0f-49b7-4618-9c87-d988af95969b-kube-api-access-mts5m\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:36:04.548657 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:04.548568    2563 generic.go:356] "Generic (PLEG): container finished" podID="784add0f-49b7-4618-9c87-d988af95969b" containerID="a2cc6e1ad86532ef325f03b759a8088376800fa2eda5896c3f02691e8c8d6159" exitCode=0
Jul 29 06:36:04.548657 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:04.548603    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-4rgqw"
Jul 29 06:36:04.549120 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:04.548654    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-4rgqw" event={"ID":"784add0f-49b7-4618-9c87-d988af95969b","Type":"ContainerDied","Data":"a2cc6e1ad86532ef325f03b759a8088376800fa2eda5896c3f02691e8c8d6159"}
Jul 29 06:36:04.549120 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:04.548710    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-4rgqw" event={"ID":"784add0f-49b7-4618-9c87-d988af95969b","Type":"ContainerDied","Data":"dfacdfdd4cc180a9cf23f0d2d97114a70f370a5a5336310ec19157c6ad188daf"}
Jul 29 06:36:04.549120 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:04.548737    2563 scope.go:117] "RemoveContainer" containerID="a2cc6e1ad86532ef325f03b759a8088376800fa2eda5896c3f02691e8c8d6159"
Jul 29 06:36:04.558142 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:04.558124    2563 scope.go:117] "RemoveContainer" containerID="a2cc6e1ad86532ef325f03b759a8088376800fa2eda5896c3f02691e8c8d6159"
Jul 29 06:36:04.558415 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:36:04.558397    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2cc6e1ad86532ef325f03b759a8088376800fa2eda5896c3f02691e8c8d6159\": container with ID starting with a2cc6e1ad86532ef325f03b759a8088376800fa2eda5896c3f02691e8c8d6159 not found: ID does not exist" containerID="a2cc6e1ad86532ef325f03b759a8088376800fa2eda5896c3f02691e8c8d6159"
Jul 29 06:36:04.558471 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:04.558426    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2cc6e1ad86532ef325f03b759a8088376800fa2eda5896c3f02691e8c8d6159"} err="failed to get container status \"a2cc6e1ad86532ef325f03b759a8088376800fa2eda5896c3f02691e8c8d6159\": rpc error: code = NotFound desc = could not find container \"a2cc6e1ad86532ef325f03b759a8088376800fa2eda5896c3f02691e8c8d6159\": container with ID starting with a2cc6e1ad86532ef325f03b759a8088376800fa2eda5896c3f02691e8c8d6159 not found: ID does not exist"
Jul 29 06:36:04.574454 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:04.574433    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-4rgqw"]
Jul 29 06:36:04.580742 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:04.580722    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-4rgqw"]
Jul 29 06:36:05.397327 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:05.397295    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="784add0f-49b7-4618-9c87-d988af95969b" path="/var/lib/kubelet/pods/784add0f-49b7-4618-9c87-d988af95969b/volumes"
Jul 29 06:36:05.397622 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:05.397609    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7f2a037-496b-496d-aa5f-f9610591bd01" path="/var/lib/kubelet/pods/d7f2a037-496b-496d-aa5f-f9610591bd01/volumes"
Jul 29 06:36:08.562933 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:08.562903    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5984f4f444-j2cfv"]
Jul 29 06:36:08.563307 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:08.563241    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="784add0f-49b7-4618-9c87-d988af95969b" containerName="authorino"
Jul 29 06:36:08.563307 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:08.563252    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="784add0f-49b7-4618-9c87-d988af95969b" containerName="authorino"
Jul 29 06:36:08.563392 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:08.563313    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="784add0f-49b7-4618-9c87-d988af95969b" containerName="authorino"
Jul 29 06:36:08.714661 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:08.714626    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5984f4f444-j2cfv"]
Jul 29 06:36:08.714812 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:08.714731    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5984f4f444-j2cfv"
Jul 29 06:36:08.717325 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:08.717298    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 29 06:36:08.717325 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:08.717322    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 29 06:36:08.717524 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:08.717305    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kkv5x\""
Jul 29 06:36:08.729558 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:08.729539    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-677c488c9b-dxrl5"]
Jul 29 06:36:08.763695 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:08.763665    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-677c488c9b-dxrl5"]
Jul 29 06:36:08.763836 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:08.763761    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-677c488c9b-dxrl5"
Jul 29 06:36:08.788160 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:08.788134    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmft2\" (UniqueName: \"kubernetes.io/projected/c705e623-4127-4d64-af84-8671c019599e-kube-api-access-kmft2\") pod \"maas-controller-5984f4f444-j2cfv\" (UID: \"c705e623-4127-4d64-af84-8671c019599e\") " pod="opendatahub/maas-controller-5984f4f444-j2cfv"
Jul 29 06:36:08.788284 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:08.788180    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c705e623-4127-4d64-af84-8671c019599e-webhook-cert\") pod \"maas-controller-5984f4f444-j2cfv\" (UID: \"c705e623-4127-4d64-af84-8671c019599e\") " pod="opendatahub/maas-controller-5984f4f444-j2cfv"
Jul 29 06:36:08.888783 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:08.888714    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9c49d6dd-a863-460a-ae03-88b3948337a1-webhook-cert\") pod \"maas-controller-677c488c9b-dxrl5\" (UID: \"9c49d6dd-a863-460a-ae03-88b3948337a1\") " pod="opendatahub/maas-controller-677c488c9b-dxrl5"
Jul 29 06:36:08.888783 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:08.888748    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7l2r\" (UniqueName: \"kubernetes.io/projected/9c49d6dd-a863-460a-ae03-88b3948337a1-kube-api-access-z7l2r\") pod \"maas-controller-677c488c9b-dxrl5\" (UID: \"9c49d6dd-a863-460a-ae03-88b3948337a1\") " pod="opendatahub/maas-controller-677c488c9b-dxrl5"
Jul 29 06:36:08.888783 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:08.888782    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmft2\" (UniqueName: \"kubernetes.io/projected/c705e623-4127-4d64-af84-8671c019599e-kube-api-access-kmft2\") pod \"maas-controller-5984f4f444-j2cfv\" (UID: \"c705e623-4127-4d64-af84-8671c019599e\") " pod="opendatahub/maas-controller-5984f4f444-j2cfv"
Jul 29 06:36:08.889035 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:08.888902    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c705e623-4127-4d64-af84-8671c019599e-webhook-cert\") pod \"maas-controller-5984f4f444-j2cfv\" (UID: \"c705e623-4127-4d64-af84-8671c019599e\") " pod="opendatahub/maas-controller-5984f4f444-j2cfv"
Jul 29 06:36:08.891529 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:08.891496    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c705e623-4127-4d64-af84-8671c019599e-webhook-cert\") pod \"maas-controller-5984f4f444-j2cfv\" (UID: \"c705e623-4127-4d64-af84-8671c019599e\") " pod="opendatahub/maas-controller-5984f4f444-j2cfv"
Jul 29 06:36:08.897018 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:08.896997    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmft2\" (UniqueName: \"kubernetes.io/projected/c705e623-4127-4d64-af84-8671c019599e-kube-api-access-kmft2\") pod \"maas-controller-5984f4f444-j2cfv\" (UID: \"c705e623-4127-4d64-af84-8671c019599e\") " pod="opendatahub/maas-controller-5984f4f444-j2cfv"
Jul 29 06:36:08.989409 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:08.989371    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9c49d6dd-a863-460a-ae03-88b3948337a1-webhook-cert\") pod \"maas-controller-677c488c9b-dxrl5\" (UID: \"9c49d6dd-a863-460a-ae03-88b3948337a1\") " pod="opendatahub/maas-controller-677c488c9b-dxrl5"
Jul 29 06:36:08.989409 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:08.989411    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7l2r\" (UniqueName: \"kubernetes.io/projected/9c49d6dd-a863-460a-ae03-88b3948337a1-kube-api-access-z7l2r\") pod \"maas-controller-677c488c9b-dxrl5\" (UID: \"9c49d6dd-a863-460a-ae03-88b3948337a1\") " pod="opendatahub/maas-controller-677c488c9b-dxrl5"
Jul 29 06:36:08.992179 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:08.992152    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9c49d6dd-a863-460a-ae03-88b3948337a1-webhook-cert\") pod \"maas-controller-677c488c9b-dxrl5\" (UID: \"9c49d6dd-a863-460a-ae03-88b3948337a1\") " pod="opendatahub/maas-controller-677c488c9b-dxrl5"
Jul 29 06:36:08.997087 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:08.997062    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7l2r\" (UniqueName: \"kubernetes.io/projected/9c49d6dd-a863-460a-ae03-88b3948337a1-kube-api-access-z7l2r\") pod \"maas-controller-677c488c9b-dxrl5\" (UID: \"9c49d6dd-a863-460a-ae03-88b3948337a1\") " pod="opendatahub/maas-controller-677c488c9b-dxrl5"
Jul 29 06:36:09.025365 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:09.025339    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5984f4f444-j2cfv"
Jul 29 06:36:09.073779 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:09.073754    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-677c488c9b-dxrl5"
Jul 29 06:36:09.197387 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:09.197356    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5984f4f444-j2cfv"]
Jul 29 06:36:09.201377 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:36:09.201345    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc705e623_4127_4d64_af84_8671c019599e.slice/crio-5ea8b12c9378ceeb09bd78fdb8aaba19d3c57164d208cbdfc997fde5e3254feb WatchSource:0}: Error finding container 5ea8b12c9378ceeb09bd78fdb8aaba19d3c57164d208cbdfc997fde5e3254feb: Status 404 returned error can't find the container with id 5ea8b12c9378ceeb09bd78fdb8aaba19d3c57164d208cbdfc997fde5e3254feb
Jul 29 06:36:09.236254 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:09.236232    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-677c488c9b-dxrl5"]
Jul 29 06:36:09.238760 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:36:09.238720    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c49d6dd_a863_460a_ae03_88b3948337a1.slice/crio-5c092da2238358288160fcdf527f68383a423aa633d6f4103627d490c6b7b4be WatchSource:0}: Error finding container 5c092da2238358288160fcdf527f68383a423aa633d6f4103627d490c6b7b4be: Status 404 returned error can't find the container with id 5c092da2238358288160fcdf527f68383a423aa633d6f4103627d490c6b7b4be
Jul 29 06:36:09.574338 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:09.574308    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5984f4f444-j2cfv" event={"ID":"c705e623-4127-4d64-af84-8671c019599e","Type":"ContainerStarted","Data":"5ea8b12c9378ceeb09bd78fdb8aaba19d3c57164d208cbdfc997fde5e3254feb"}
Jul 29 06:36:09.575441 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:09.575418    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-677c488c9b-dxrl5" event={"ID":"9c49d6dd-a863-460a-ae03-88b3948337a1","Type":"ContainerStarted","Data":"5c092da2238358288160fcdf527f68383a423aa633d6f4103627d490c6b7b4be"}
Jul 29 06:36:12.593112 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:12.593075    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-677c488c9b-dxrl5" event={"ID":"9c49d6dd-a863-460a-ae03-88b3948337a1","Type":"ContainerStarted","Data":"07e1498e57b8030d70f00164f9ce2bba9e35be7e08cce0ff24ba056d1554ed20"}
Jul 29 06:36:12.593526 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:12.593197    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-677c488c9b-dxrl5"
Jul 29 06:36:12.594809 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:12.594782    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5984f4f444-j2cfv" event={"ID":"c705e623-4127-4d64-af84-8671c019599e","Type":"ContainerStarted","Data":"44f01a860606a7cb1149472cdb37759fa1f6a65e2d36f4a6137a1f0f5626a83e"}
Jul 29 06:36:12.594934 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:12.594909    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5984f4f444-j2cfv"
Jul 29 06:36:12.608490 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:12.608448    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-677c488c9b-dxrl5" podStartSLOduration=2.004728617 podStartE2EDuration="4.608435947s" totalImagesPullingTime="2.60370733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:36:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:36:12.606603067 +0000 UTC m=+815.684234833" watchObservedRunningTime="2026-07-29 06:36:12.608435947 +0000 UTC m=+815.686067692"
Jul 29 06:36:12.621140 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:12.621092    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5984f4f444-j2cfv" podStartSLOduration=1.9743261140000001 podStartE2EDuration="4.621075795s" totalImagesPullingTime="2.646749681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:36:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:36:12.620751312 +0000 UTC m=+815.698383077" watchObservedRunningTime="2026-07-29 06:36:12.621075795 +0000 UTC m=+815.698707563"
Jul 29 06:36:23.601247 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:23.601218    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-677c488c9b-dxrl5"
Jul 29 06:36:23.601602 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:23.601312    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5984f4f444-j2cfv"
Jul 29 06:36:23.656148 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:23.656119    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5984f4f444-j2cfv"]
Jul 29 06:36:23.656348 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:23.656319    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5984f4f444-j2cfv" podUID="c705e623-4127-4d64-af84-8671c019599e" containerName="manager" containerID="cri-o://44f01a860606a7cb1149472cdb37759fa1f6a65e2d36f4a6137a1f0f5626a83e" gracePeriod=30
Jul 29 06:36:23.907278 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:23.907256    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5984f4f444-j2cfv"
Jul 29 06:36:24.025657 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:24.025632    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c705e623-4127-4d64-af84-8671c019599e-webhook-cert\") pod \"c705e623-4127-4d64-af84-8671c019599e\" (UID: \"c705e623-4127-4d64-af84-8671c019599e\") "
Jul 29 06:36:24.025828 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:24.025686    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmft2\" (UniqueName: \"kubernetes.io/projected/c705e623-4127-4d64-af84-8671c019599e-kube-api-access-kmft2\") pod \"c705e623-4127-4d64-af84-8671c019599e\" (UID: \"c705e623-4127-4d64-af84-8671c019599e\") "
Jul 29 06:36:24.027816 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:24.027793    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c705e623-4127-4d64-af84-8671c019599e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "c705e623-4127-4d64-af84-8671c019599e" (UID: "c705e623-4127-4d64-af84-8671c019599e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 06:36:24.027938 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:24.027868    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c705e623-4127-4d64-af84-8671c019599e-kube-api-access-kmft2" (OuterVolumeSpecName: "kube-api-access-kmft2") pod "c705e623-4127-4d64-af84-8671c019599e" (UID: "c705e623-4127-4d64-af84-8671c019599e"). InnerVolumeSpecName "kube-api-access-kmft2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 06:36:24.126595 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:24.126556    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c705e623-4127-4d64-af84-8671c019599e-webhook-cert\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:36:24.126595 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:24.126593    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmft2\" (UniqueName: \"kubernetes.io/projected/c705e623-4127-4d64-af84-8671c019599e-kube-api-access-kmft2\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:36:24.649465 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:24.649429    2563 generic.go:356] "Generic (PLEG): container finished" podID="c705e623-4127-4d64-af84-8671c019599e" containerID="44f01a860606a7cb1149472cdb37759fa1f6a65e2d36f4a6137a1f0f5626a83e" exitCode=0
Jul 29 06:36:24.649807 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:24.649489    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5984f4f444-j2cfv"
Jul 29 06:36:24.649807 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:24.649513    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5984f4f444-j2cfv" event={"ID":"c705e623-4127-4d64-af84-8671c019599e","Type":"ContainerDied","Data":"44f01a860606a7cb1149472cdb37759fa1f6a65e2d36f4a6137a1f0f5626a83e"}
Jul 29 06:36:24.649807 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:24.649597    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5984f4f444-j2cfv" event={"ID":"c705e623-4127-4d64-af84-8671c019599e","Type":"ContainerDied","Data":"5ea8b12c9378ceeb09bd78fdb8aaba19d3c57164d208cbdfc997fde5e3254feb"}
Jul 29 06:36:24.649807 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:24.649626    2563 scope.go:117] "RemoveContainer" containerID="44f01a860606a7cb1149472cdb37759fa1f6a65e2d36f4a6137a1f0f5626a83e"
Jul 29 06:36:24.659292 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:24.659135    2563 scope.go:117] "RemoveContainer" containerID="44f01a860606a7cb1149472cdb37759fa1f6a65e2d36f4a6137a1f0f5626a83e"
Jul 29 06:36:24.659409 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:36:24.659389    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44f01a860606a7cb1149472cdb37759fa1f6a65e2d36f4a6137a1f0f5626a83e\": container with ID starting with 44f01a860606a7cb1149472cdb37759fa1f6a65e2d36f4a6137a1f0f5626a83e not found: ID does not exist" containerID="44f01a860606a7cb1149472cdb37759fa1f6a65e2d36f4a6137a1f0f5626a83e"
Jul 29 06:36:24.659452 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:24.659418    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44f01a860606a7cb1149472cdb37759fa1f6a65e2d36f4a6137a1f0f5626a83e"} err="failed to get container status \"44f01a860606a7cb1149472cdb37759fa1f6a65e2d36f4a6137a1f0f5626a83e\": rpc error: code = NotFound desc = could not find container \"44f01a860606a7cb1149472cdb37759fa1f6a65e2d36f4a6137a1f0f5626a83e\": container with ID starting with 44f01a860606a7cb1149472cdb37759fa1f6a65e2d36f4a6137a1f0f5626a83e not found: ID does not exist"
Jul 29 06:36:24.669223 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:24.669205    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5984f4f444-j2cfv"]
Jul 29 06:36:24.672484 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:24.672465    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5984f4f444-j2cfv"]
Jul 29 06:36:25.397266 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:25.397231    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c705e623-4127-4d64-af84-8671c019599e" path="/var/lib/kubelet/pods/c705e623-4127-4d64-af84-8671c019599e/volumes"
Jul 29 06:36:44.364785 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:44.364758    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6bd9f8c959-scl8p"]
Jul 29 06:36:44.365440 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:44.365137    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c705e623-4127-4d64-af84-8671c019599e" containerName="manager"
Jul 29 06:36:44.365440 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:44.365148    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c705e623-4127-4d64-af84-8671c019599e" containerName="manager"
Jul 29 06:36:44.365440 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:44.365200    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c705e623-4127-4d64-af84-8671c019599e" containerName="manager"
Jul 29 06:36:44.369613 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:44.369596    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6bd9f8c959-scl8p"
Jul 29 06:36:44.371615 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:44.371586    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 29 06:36:44.371746 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:44.371644    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sx9bv\""
Jul 29 06:36:44.377841 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:44.377821    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6bd9f8c959-scl8p"]
Jul 29 06:36:44.397900 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:44.397719    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e970836d-8708-475d-82a2-55b2acbf2a6f-maas-api-tls\") pod \"maas-api-6bd9f8c959-scl8p\" (UID: \"e970836d-8708-475d-82a2-55b2acbf2a6f\") " pod="odh-ai-gateway-infra/maas-api-6bd9f8c959-scl8p"
Jul 29 06:36:44.398038 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:44.397965    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7f6m\" (UniqueName: \"kubernetes.io/projected/e970836d-8708-475d-82a2-55b2acbf2a6f-kube-api-access-z7f6m\") pod \"maas-api-6bd9f8c959-scl8p\" (UID: \"e970836d-8708-475d-82a2-55b2acbf2a6f\") " pod="odh-ai-gateway-infra/maas-api-6bd9f8c959-scl8p"
Jul 29 06:36:44.398038 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:44.398002    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e970836d-8708-475d-82a2-55b2acbf2a6f-openshift-service-ca\") pod \"maas-api-6bd9f8c959-scl8p\" (UID: \"e970836d-8708-475d-82a2-55b2acbf2a6f\") " pod="odh-ai-gateway-infra/maas-api-6bd9f8c959-scl8p"
Jul 29 06:36:44.498581 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:44.498549    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e970836d-8708-475d-82a2-55b2acbf2a6f-maas-api-tls\") pod \"maas-api-6bd9f8c959-scl8p\" (UID: \"e970836d-8708-475d-82a2-55b2acbf2a6f\") " pod="odh-ai-gateway-infra/maas-api-6bd9f8c959-scl8p"
Jul 29 06:36:44.498581 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:44.498587    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7f6m\" (UniqueName: \"kubernetes.io/projected/e970836d-8708-475d-82a2-55b2acbf2a6f-kube-api-access-z7f6m\") pod \"maas-api-6bd9f8c959-scl8p\" (UID: \"e970836d-8708-475d-82a2-55b2acbf2a6f\") " pod="odh-ai-gateway-infra/maas-api-6bd9f8c959-scl8p"
Jul 29 06:36:44.498784 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:44.498605    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e970836d-8708-475d-82a2-55b2acbf2a6f-openshift-service-ca\") pod \"maas-api-6bd9f8c959-scl8p\" (UID: \"e970836d-8708-475d-82a2-55b2acbf2a6f\") " pod="odh-ai-gateway-infra/maas-api-6bd9f8c959-scl8p"
Jul 29 06:36:44.498784 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:36:44.498693    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Jul 29 06:36:44.498784 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:36:44.498782    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e970836d-8708-475d-82a2-55b2acbf2a6f-maas-api-tls podName:e970836d-8708-475d-82a2-55b2acbf2a6f nodeName:}" failed. No retries permitted until 2026-07-29 06:36:44.998760317 +0000 UTC m=+848.076392078 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e970836d-8708-475d-82a2-55b2acbf2a6f-maas-api-tls") pod "maas-api-6bd9f8c959-scl8p" (UID: "e970836d-8708-475d-82a2-55b2acbf2a6f") : secret "maas-api-serving-cert" not found
Jul 29 06:36:44.499289 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:44.499270    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e970836d-8708-475d-82a2-55b2acbf2a6f-openshift-service-ca\") pod \"maas-api-6bd9f8c959-scl8p\" (UID: \"e970836d-8708-475d-82a2-55b2acbf2a6f\") " pod="odh-ai-gateway-infra/maas-api-6bd9f8c959-scl8p"
Jul 29 06:36:44.509444 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:44.509422    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7f6m\" (UniqueName: \"kubernetes.io/projected/e970836d-8708-475d-82a2-55b2acbf2a6f-kube-api-access-z7f6m\") pod \"maas-api-6bd9f8c959-scl8p\" (UID: \"e970836d-8708-475d-82a2-55b2acbf2a6f\") " pod="odh-ai-gateway-infra/maas-api-6bd9f8c959-scl8p"
Jul 29 06:36:44.822499 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:44.822471    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-v55gs"]
Jul 29 06:36:44.824961 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:44.824946    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-v55gs"
Jul 29 06:36:44.827868 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:44.827821    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 29 06:36:44.827868 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:44.827821    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-k2gbg\""
Jul 29 06:36:44.835448 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:44.835425    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-v55gs"]
Jul 29 06:36:44.847755 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:44.847723    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-csc8g"]
Jul 29 06:36:44.850091 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:44.850074    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-csc8g"
Jul 29 06:36:44.862300 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:44.862279    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-csc8g"]
Jul 29 06:36:44.901907 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:44.901877    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmn8z\" (UniqueName: \"kubernetes.io/projected/75e1377f-ca60-47c3-b91b-f328413bd9f6-kube-api-access-gmn8z\") pod \"payload-processing-5cc6cd444b-csc8g\" (UID: \"75e1377f-ca60-47c3-b91b-f328413bd9f6\") " pod="openshift-ingress/payload-processing-5cc6cd444b-csc8g"
Jul 29 06:36:44.902071 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:44.901920    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvcdn\" (UniqueName: \"kubernetes.io/projected/c2f9abbe-999d-4a04-b435-4cad541c4ff3-kube-api-access-qvcdn\") pod \"payload-pre-processing-687dbb6cf6-v55gs\" (UID: \"c2f9abbe-999d-4a04-b435-4cad541c4ff3\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-v55gs"
Jul 29 06:36:44.902071 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:44.902037    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2f9abbe-999d-4a04-b435-4cad541c4ff3-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-v55gs\" (UID: \"c2f9abbe-999d-4a04-b435-4cad541c4ff3\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-v55gs"
Jul 29 06:36:44.902179 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:44.902071    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75e1377f-ca60-47c3-b91b-f328413bd9f6-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-csc8g\" (UID: \"75e1377f-ca60-47c3-b91b-f328413bd9f6\") " pod="openshift-ingress/payload-processing-5cc6cd444b-csc8g"
Jul 29 06:36:45.002772 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:45.002675    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e970836d-8708-475d-82a2-55b2acbf2a6f-maas-api-tls\") pod \"maas-api-6bd9f8c959-scl8p\" (UID: \"e970836d-8708-475d-82a2-55b2acbf2a6f\") " pod="odh-ai-gateway-infra/maas-api-6bd9f8c959-scl8p"
Jul 29 06:36:45.002953 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:45.002792    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmn8z\" (UniqueName: \"kubernetes.io/projected/75e1377f-ca60-47c3-b91b-f328413bd9f6-kube-api-access-gmn8z\") pod \"payload-processing-5cc6cd444b-csc8g\" (UID: \"75e1377f-ca60-47c3-b91b-f328413bd9f6\") " pod="openshift-ingress/payload-processing-5cc6cd444b-csc8g"
Jul 29 06:36:45.002953 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:45.002846    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvcdn\" (UniqueName: \"kubernetes.io/projected/c2f9abbe-999d-4a04-b435-4cad541c4ff3-kube-api-access-qvcdn\") pod \"payload-pre-processing-687dbb6cf6-v55gs\" (UID: \"c2f9abbe-999d-4a04-b435-4cad541c4ff3\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-v55gs"
Jul 29 06:36:45.002953 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:45.002943    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2f9abbe-999d-4a04-b435-4cad541c4ff3-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-v55gs\" (UID: \"c2f9abbe-999d-4a04-b435-4cad541c4ff3\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-v55gs"
Jul 29 06:36:45.003086 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:45.002967    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75e1377f-ca60-47c3-b91b-f328413bd9f6-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-csc8g\" (UID: \"75e1377f-ca60-47c3-b91b-f328413bd9f6\") " pod="openshift-ingress/payload-processing-5cc6cd444b-csc8g"
Jul 29 06:36:45.003846 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:45.003820    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75e1377f-ca60-47c3-b91b-f328413bd9f6-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-csc8g\" (UID: \"75e1377f-ca60-47c3-b91b-f328413bd9f6\") " pod="openshift-ingress/payload-processing-5cc6cd444b-csc8g"
Jul 29 06:36:45.004039 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:45.004018    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2f9abbe-999d-4a04-b435-4cad541c4ff3-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-v55gs\" (UID: \"c2f9abbe-999d-4a04-b435-4cad541c4ff3\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-v55gs"
Jul 29 06:36:45.005493 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:45.005469    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e970836d-8708-475d-82a2-55b2acbf2a6f-maas-api-tls\") pod \"maas-api-6bd9f8c959-scl8p\" (UID: \"e970836d-8708-475d-82a2-55b2acbf2a6f\") " pod="odh-ai-gateway-infra/maas-api-6bd9f8c959-scl8p"
Jul 29 06:36:45.010558 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:45.010538    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvcdn\" (UniqueName: \"kubernetes.io/projected/c2f9abbe-999d-4a04-b435-4cad541c4ff3-kube-api-access-qvcdn\") pod \"payload-pre-processing-687dbb6cf6-v55gs\" (UID: \"c2f9abbe-999d-4a04-b435-4cad541c4ff3\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-v55gs"
Jul 29 06:36:45.010952 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:45.010932    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmn8z\" (UniqueName: \"kubernetes.io/projected/75e1377f-ca60-47c3-b91b-f328413bd9f6-kube-api-access-gmn8z\") pod \"payload-processing-5cc6cd444b-csc8g\" (UID: \"75e1377f-ca60-47c3-b91b-f328413bd9f6\") " pod="openshift-ingress/payload-processing-5cc6cd444b-csc8g"
Jul 29 06:36:45.137569 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:45.137495    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-v55gs"
Jul 29 06:36:45.160626 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:45.160594    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-csc8g"
Jul 29 06:36:45.281334 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:45.281300    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6bd9f8c959-scl8p"
Jul 29 06:36:45.282358 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:45.282291    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-v55gs"]
Jul 29 06:36:45.285146 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:36:45.285116    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2f9abbe_999d_4a04_b435_4cad541c4ff3.slice/crio-a71927618ee62f34eacaba854eeb12198dbde69ae1419b355e0f21c2966176af WatchSource:0}: Error finding container a71927618ee62f34eacaba854eeb12198dbde69ae1419b355e0f21c2966176af: Status 404 returned error can't find the container with id a71927618ee62f34eacaba854eeb12198dbde69ae1419b355e0f21c2966176af
Jul 29 06:36:45.321163 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:45.313935    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-csc8g"]
Jul 29 06:36:45.321163 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:36:45.319240    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e1377f_ca60_47c3_b91b_f328413bd9f6.slice/crio-f8c829700149afbc88f8186e0807831a387ee5ee96132fc4f4db653fb2adf2ec WatchSource:0}: Error finding container f8c829700149afbc88f8186e0807831a387ee5ee96132fc4f4db653fb2adf2ec: Status 404 returned error can't find the container with id f8c829700149afbc88f8186e0807831a387ee5ee96132fc4f4db653fb2adf2ec
Jul 29 06:36:45.443034 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:45.443003    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6bd9f8c959-scl8p"]
Jul 29 06:36:45.446750 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:36:45.446726    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode970836d_8708_475d_82a2_55b2acbf2a6f.slice/crio-aef219d524ace09eee496680e573e5db0e3b9e265e715f3b8f758ff0104caa41 WatchSource:0}: Error finding container aef219d524ace09eee496680e573e5db0e3b9e265e715f3b8f758ff0104caa41: Status 404 returned error can't find the container with id aef219d524ace09eee496680e573e5db0e3b9e265e715f3b8f758ff0104caa41
Jul 29 06:36:45.747956 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:45.747845    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-v55gs" event={"ID":"c2f9abbe-999d-4a04-b435-4cad541c4ff3","Type":"ContainerStarted","Data":"a71927618ee62f34eacaba854eeb12198dbde69ae1419b355e0f21c2966176af"}
Jul 29 06:36:45.749004 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:45.748985    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6bd9f8c959-scl8p" event={"ID":"e970836d-8708-475d-82a2-55b2acbf2a6f","Type":"ContainerStarted","Data":"aef219d524ace09eee496680e573e5db0e3b9e265e715f3b8f758ff0104caa41"}
Jul 29 06:36:45.750010 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:45.749993    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-csc8g" event={"ID":"75e1377f-ca60-47c3-b91b-f328413bd9f6","Type":"ContainerStarted","Data":"f8c829700149afbc88f8186e0807831a387ee5ee96132fc4f4db653fb2adf2ec"}
Jul 29 06:36:47.765146 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:47.765063    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-v55gs" event={"ID":"c2f9abbe-999d-4a04-b435-4cad541c4ff3","Type":"ContainerStarted","Data":"4bbaa8c58f665f5af8fc5adea52721020432f61a936b92ecb891afe0b27ad24e"}
Jul 29 06:36:47.765555 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:47.765197    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-v55gs"
Jul 29 06:36:47.766670 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:47.766649    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6bd9f8c959-scl8p" event={"ID":"e970836d-8708-475d-82a2-55b2acbf2a6f","Type":"ContainerStarted","Data":"0c35a641aa3e2c47b260af1c9f1927b44ddbfda4b7ff1601851ea385f639fc89"}
Jul 29 06:36:47.766796 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:47.766778    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6bd9f8c959-scl8p"
Jul 29 06:36:47.768259 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:47.768239    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-csc8g" event={"ID":"75e1377f-ca60-47c3-b91b-f328413bd9f6","Type":"ContainerStarted","Data":"03661ac2fab39128cf9049aa81516c5a9590760e44c545f1080ce01f227b9027"}
Jul 29 06:36:47.768345 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:47.768291    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-csc8g"
Jul 29 06:36:47.778446 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:47.778396    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-v55gs" podStartSLOduration=1.552658775 podStartE2EDuration="3.778379765s" totalImagesPullingTime="2.22572099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:36:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:36:47.777902243 +0000 UTC m=+850.855534013" watchObservedRunningTime="2026-07-29 06:36:47.778379765 +0000 UTC m=+850.856011533"
Jul 29 06:36:47.790658 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:47.790610    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-csc8g" podStartSLOduration=1.608304773 podStartE2EDuration="3.790593597s" totalImagesPullingTime="2.182288824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:36:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:36:47.790083885 +0000 UTC m=+850.867715664" watchObservedRunningTime="2026-07-29 06:36:47.790593597 +0000 UTC m=+850.868225367"
Jul 29 06:36:47.804516 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:47.804472    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6bd9f8c959-scl8p" podStartSLOduration=1.749874121 podStartE2EDuration="3.804458112s" totalImagesPullingTime="2.054583991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:36:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:36:47.803092926 +0000 UTC m=+850.880724697" watchObservedRunningTime="2026-07-29 06:36:47.804458112 +0000 UTC m=+850.882089913"
Jul 29 06:36:53.779014 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:53.778987    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6bd9f8c959-scl8p"
Jul 29 06:36:58.775318 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:58.775285    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-csc8g"
Jul 29 06:36:58.775745 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:36:58.775378    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-v55gs"
Jul 29 06:37:01.405311 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:01.405282    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6"]
Jul 29 06:37:01.407932 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:01.407910    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6"
Jul 29 06:37:01.409907 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:01.409888    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 29 06:37:01.409979 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:01.409887    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 29 06:37:01.409979 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:01.409893    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kkwjc\""
Jul 29 06:37:01.410290 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:01.410275    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 29 06:37:01.418059 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:01.418036    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6"]
Jul 29 06:37:01.452278 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:01.452248    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6101e8c2-fdd5-4de5-8fed-9d250dd10c9a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6\" (UID: \"6101e8c2-fdd5-4de5-8fed-9d250dd10c9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6"
Jul 29 06:37:01.452429 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:01.452280    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc6qf\" (UniqueName: \"kubernetes.io/projected/6101e8c2-fdd5-4de5-8fed-9d250dd10c9a-kube-api-access-mc6qf\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6\" (UID: \"6101e8c2-fdd5-4de5-8fed-9d250dd10c9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6"
Jul 29 06:37:01.452429 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:01.452346    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6101e8c2-fdd5-4de5-8fed-9d250dd10c9a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6\" (UID: \"6101e8c2-fdd5-4de5-8fed-9d250dd10c9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6"
Jul 29 06:37:01.452429 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:01.452411    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6101e8c2-fdd5-4de5-8fed-9d250dd10c9a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6\" (UID: \"6101e8c2-fdd5-4de5-8fed-9d250dd10c9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6"
Jul 29 06:37:01.452614 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:01.452553    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6101e8c2-fdd5-4de5-8fed-9d250dd10c9a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6\" (UID: \"6101e8c2-fdd5-4de5-8fed-9d250dd10c9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6"
Jul 29 06:37:01.452670 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:01.452613    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6101e8c2-fdd5-4de5-8fed-9d250dd10c9a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6\" (UID: \"6101e8c2-fdd5-4de5-8fed-9d250dd10c9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6"
Jul 29 06:37:01.553527 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:01.553493    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6101e8c2-fdd5-4de5-8fed-9d250dd10c9a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6\" (UID: \"6101e8c2-fdd5-4de5-8fed-9d250dd10c9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6"
Jul 29 06:37:01.553694 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:01.553532    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc6qf\" (UniqueName: \"kubernetes.io/projected/6101e8c2-fdd5-4de5-8fed-9d250dd10c9a-kube-api-access-mc6qf\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6\" (UID: \"6101e8c2-fdd5-4de5-8fed-9d250dd10c9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6"
Jul 29 06:37:01.553694 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:01.553553    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6101e8c2-fdd5-4de5-8fed-9d250dd10c9a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6\" (UID: \"6101e8c2-fdd5-4de5-8fed-9d250dd10c9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6"
Jul 29 06:37:01.553694 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:01.553576    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6101e8c2-fdd5-4de5-8fed-9d250dd10c9a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6\" (UID: \"6101e8c2-fdd5-4de5-8fed-9d250dd10c9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6"
Jul 29 06:37:01.553694 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:01.553614    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6101e8c2-fdd5-4de5-8fed-9d250dd10c9a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6\" (UID: \"6101e8c2-fdd5-4de5-8fed-9d250dd10c9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6"
Jul 29 06:37:01.553694 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:01.553654    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6101e8c2-fdd5-4de5-8fed-9d250dd10c9a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6\" (UID: \"6101e8c2-fdd5-4de5-8fed-9d250dd10c9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6"
Jul 29 06:37:01.554044 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:01.554017    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6101e8c2-fdd5-4de5-8fed-9d250dd10c9a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6\" (UID: \"6101e8c2-fdd5-4de5-8fed-9d250dd10c9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6"
Jul 29 06:37:01.554168 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:01.554058    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6101e8c2-fdd5-4de5-8fed-9d250dd10c9a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6\" (UID: \"6101e8c2-fdd5-4de5-8fed-9d250dd10c9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6"
Jul 29 06:37:01.554168 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:01.554083    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6101e8c2-fdd5-4de5-8fed-9d250dd10c9a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6\" (UID: \"6101e8c2-fdd5-4de5-8fed-9d250dd10c9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6"
Jul 29 06:37:01.555793 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:01.555772    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6101e8c2-fdd5-4de5-8fed-9d250dd10c9a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6\" (UID: \"6101e8c2-fdd5-4de5-8fed-9d250dd10c9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6"
Jul 29 06:37:01.556101 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:01.556086    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6101e8c2-fdd5-4de5-8fed-9d250dd10c9a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6\" (UID: \"6101e8c2-fdd5-4de5-8fed-9d250dd10c9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6"
Jul 29 06:37:01.560604 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:01.560578    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc6qf\" (UniqueName: \"kubernetes.io/projected/6101e8c2-fdd5-4de5-8fed-9d250dd10c9a-kube-api-access-mc6qf\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6\" (UID: \"6101e8c2-fdd5-4de5-8fed-9d250dd10c9a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6"
Jul 29 06:37:01.718939 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:01.718872    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6"
Jul 29 06:37:01.850301 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:01.850279    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6"]
Jul 29 06:37:01.852792 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:37:01.852755    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6101e8c2_fdd5_4de5_8fed_9d250dd10c9a.slice/crio-c87955f13c9005e1b7612d0906f684235231270d696706294c590aa9cc62c431 WatchSource:0}: Error finding container c87955f13c9005e1b7612d0906f684235231270d696706294c590aa9cc62c431: Status 404 returned error can't find the container with id c87955f13c9005e1b7612d0906f684235231270d696706294c590aa9cc62c431
Jul 29 06:37:02.845872 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:02.845809    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6" event={"ID":"6101e8c2-fdd5-4de5-8fed-9d250dd10c9a","Type":"ContainerStarted","Data":"c87955f13c9005e1b7612d0906f684235231270d696706294c590aa9cc62c431"}
Jul 29 06:37:03.852023 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:03.851988    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6" event={"ID":"6101e8c2-fdd5-4de5-8fed-9d250dd10c9a","Type":"ContainerStarted","Data":"c718c01c659b1bafea37855f5b0b3244d8e0e51e8c9c726ab8d858a09883a55c"}
Jul 29 06:37:03.866715 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:03.866663    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6" podStartSLOduration=1.7684170369999999 podStartE2EDuration="2.86664318s" totalImagesPullingTime="1.098226143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:37:03.865911245 +0000 UTC m=+866.943543012" watchObservedRunningTime="2026-07-29 06:37:03.86664318 +0000 UTC m=+866.944274952"
Jul 29 06:37:11.719640 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:11.719604    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6"
Jul 29 06:37:11.720271 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:11.719711    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6"
Jul 29 06:37:11.732242 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:11.732219    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6"
Jul 29 06:37:11.898803 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:11.898780    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6"
Jul 29 06:37:29.805371 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:29.805293    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg"]
Jul 29 06:37:29.811873 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:29.811833    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg"
Jul 29 06:37:29.813864 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:29.813828    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 29 06:37:29.816781 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:29.816761    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg"]
Jul 29 06:37:29.904733 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:29.904705    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5805ac5a-18bf-4ac2-a496-54323bd42721-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg\" (UID: \"5805ac5a-18bf-4ac2-a496-54323bd42721\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg"
Jul 29 06:37:29.904922 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:29.904751    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5805ac5a-18bf-4ac2-a496-54323bd42721-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg\" (UID: \"5805ac5a-18bf-4ac2-a496-54323bd42721\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg"
Jul 29 06:37:29.904922 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:29.904816    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bswrc\" (UniqueName: \"kubernetes.io/projected/5805ac5a-18bf-4ac2-a496-54323bd42721-kube-api-access-bswrc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg\" (UID: \"5805ac5a-18bf-4ac2-a496-54323bd42721\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg"
Jul 29 06:37:29.904922 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:29.904884    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5805ac5a-18bf-4ac2-a496-54323bd42721-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg\" (UID: \"5805ac5a-18bf-4ac2-a496-54323bd42721\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg"
Jul 29 06:37:29.904922 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:29.904907    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5805ac5a-18bf-4ac2-a496-54323bd42721-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg\" (UID: \"5805ac5a-18bf-4ac2-a496-54323bd42721\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg"
Jul 29 06:37:29.905094 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:29.904935    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5805ac5a-18bf-4ac2-a496-54323bd42721-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg\" (UID: \"5805ac5a-18bf-4ac2-a496-54323bd42721\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg"
Jul 29 06:37:30.006108 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:30.006072    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5805ac5a-18bf-4ac2-a496-54323bd42721-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg\" (UID: \"5805ac5a-18bf-4ac2-a496-54323bd42721\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg"
Jul 29 06:37:30.006296 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:30.006133    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bswrc\" (UniqueName: \"kubernetes.io/projected/5805ac5a-18bf-4ac2-a496-54323bd42721-kube-api-access-bswrc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg\" (UID: \"5805ac5a-18bf-4ac2-a496-54323bd42721\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg"
Jul 29 06:37:30.006296 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:30.006161    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5805ac5a-18bf-4ac2-a496-54323bd42721-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg\" (UID: \"5805ac5a-18bf-4ac2-a496-54323bd42721\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg"
Jul 29 06:37:30.006296 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:30.006182    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5805ac5a-18bf-4ac2-a496-54323bd42721-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg\" (UID: \"5805ac5a-18bf-4ac2-a496-54323bd42721\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg"
Jul 29 06:37:30.006296 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:30.006208    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5805ac5a-18bf-4ac2-a496-54323bd42721-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg\" (UID: \"5805ac5a-18bf-4ac2-a496-54323bd42721\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg"
Jul 29 06:37:30.006483 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:30.006307    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5805ac5a-18bf-4ac2-a496-54323bd42721-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg\" (UID: \"5805ac5a-18bf-4ac2-a496-54323bd42721\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg"
Jul 29 06:37:30.006607 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:30.006572    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5805ac5a-18bf-4ac2-a496-54323bd42721-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg\" (UID: \"5805ac5a-18bf-4ac2-a496-54323bd42721\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg"
Jul 29 06:37:30.006730 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:30.006643    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5805ac5a-18bf-4ac2-a496-54323bd42721-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg\" (UID: \"5805ac5a-18bf-4ac2-a496-54323bd42721\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg"
Jul 29 06:37:30.006730 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:30.006675    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5805ac5a-18bf-4ac2-a496-54323bd42721-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg\" (UID: \"5805ac5a-18bf-4ac2-a496-54323bd42721\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg"
Jul 29 06:37:30.008682 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:30.008660    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5805ac5a-18bf-4ac2-a496-54323bd42721-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg\" (UID: \"5805ac5a-18bf-4ac2-a496-54323bd42721\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg"
Jul 29 06:37:30.008950 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:30.008933    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5805ac5a-18bf-4ac2-a496-54323bd42721-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg\" (UID: \"5805ac5a-18bf-4ac2-a496-54323bd42721\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg"
Jul 29 06:37:30.013062 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:30.013041    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bswrc\" (UniqueName: \"kubernetes.io/projected/5805ac5a-18bf-4ac2-a496-54323bd42721-kube-api-access-bswrc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg\" (UID: \"5805ac5a-18bf-4ac2-a496-54323bd42721\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg"
Jul 29 06:37:30.140952 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:30.140916    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg"
Jul 29 06:37:30.271400 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:30.271340    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg"]
Jul 29 06:37:30.273731 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:37:30.273709    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5805ac5a_18bf_4ac2_a496_54323bd42721.slice/crio-842d8cf80c83dab4999976deb46c3519712edfd9952ea866f40f039f3263e829 WatchSource:0}: Error finding container 842d8cf80c83dab4999976deb46c3519712edfd9952ea866f40f039f3263e829: Status 404 returned error can't find the container with id 842d8cf80c83dab4999976deb46c3519712edfd9952ea866f40f039f3263e829
Jul 29 06:37:30.978263 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:30.978229    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg" event={"ID":"5805ac5a-18bf-4ac2-a496-54323bd42721","Type":"ContainerStarted","Data":"e260a3bd56fc86869b7c0bc4b91ef192a2e4bd8edc0f7879693e131ac83bc1bb"}
Jul 29 06:37:30.978679 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:30.978279    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg" event={"ID":"5805ac5a-18bf-4ac2-a496-54323bd42721","Type":"ContainerStarted","Data":"842d8cf80c83dab4999976deb46c3519712edfd9952ea866f40f039f3263e829"}
Jul 29 06:37:30.993470 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:30.993422    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg" podStartSLOduration=1.802489684 podStartE2EDuration="1.993408066s" totalImagesPullingTime="190.918382ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:37:30.992410274 +0000 UTC m=+894.070042044" watchObservedRunningTime="2026-07-29 06:37:30.993408066 +0000 UTC m=+894.071039868"
Jul 29 06:37:40.141245 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:40.141210    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg"
Jul 29 06:37:40.143664 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:40.141291    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg"
Jul 29 06:37:40.154059 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:40.154035    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg"
Jul 29 06:37:41.039076 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:37:41.039047    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg"
Jul 29 06:39:03.283558 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:03.283483    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-586ff9c89-4tgtz"]
Jul 29 06:39:03.286949 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:03.286928    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-586ff9c89-4tgtz"
Jul 29 06:39:03.288869 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:03.288835    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 29 06:39:03.288989 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:03.288884    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 29 06:39:03.289434 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:03.289420    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-82x8j\""
Jul 29 06:39:03.289512 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:03.289419    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 29 06:39:03.292752 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:03.292730    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-586ff9c89-4tgtz"]
Jul 29 06:39:03.437521 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:03.437495    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnnvs\" (UniqueName: \"kubernetes.io/projected/31c2224d-b518-44e9-97cf-2fc2cff44b23-kube-api-access-xnnvs\") pod \"authorino-586ff9c89-4tgtz\" (UID: \"31c2224d-b518-44e9-97cf-2fc2cff44b23\") " pod="kuadrant-system/authorino-586ff9c89-4tgtz"
Jul 29 06:39:03.437703 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:03.437529    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/31c2224d-b518-44e9-97cf-2fc2cff44b23-tls-cert\") pod \"authorino-586ff9c89-4tgtz\" (UID: \"31c2224d-b518-44e9-97cf-2fc2cff44b23\") " pod="kuadrant-system/authorino-586ff9c89-4tgtz"
Jul 29 06:39:03.538912 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:03.538802    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/31c2224d-b518-44e9-97cf-2fc2cff44b23-tls-cert\") pod \"authorino-586ff9c89-4tgtz\" (UID: \"31c2224d-b518-44e9-97cf-2fc2cff44b23\") " pod="kuadrant-system/authorino-586ff9c89-4tgtz"
Jul 29 06:39:03.539044 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:03.538926    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnnvs\" (UniqueName: \"kubernetes.io/projected/31c2224d-b518-44e9-97cf-2fc2cff44b23-kube-api-access-xnnvs\") pod \"authorino-586ff9c89-4tgtz\" (UID: \"31c2224d-b518-44e9-97cf-2fc2cff44b23\") " pod="kuadrant-system/authorino-586ff9c89-4tgtz"
Jul 29 06:39:03.541412 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:03.541382    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/31c2224d-b518-44e9-97cf-2fc2cff44b23-tls-cert\") pod \"authorino-586ff9c89-4tgtz\" (UID: \"31c2224d-b518-44e9-97cf-2fc2cff44b23\") " pod="kuadrant-system/authorino-586ff9c89-4tgtz"
Jul 29 06:39:03.545898 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:03.545875    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnnvs\" (UniqueName: \"kubernetes.io/projected/31c2224d-b518-44e9-97cf-2fc2cff44b23-kube-api-access-xnnvs\") pod \"authorino-586ff9c89-4tgtz\" (UID: \"31c2224d-b518-44e9-97cf-2fc2cff44b23\") " pod="kuadrant-system/authorino-586ff9c89-4tgtz"
Jul 29 06:39:03.597964 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:03.597940    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-586ff9c89-4tgtz"
Jul 29 06:39:03.730863 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:03.730825    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-586ff9c89-4tgtz"]
Jul 29 06:39:03.733473 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:39:03.733450    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31c2224d_b518_44e9_97cf_2fc2cff44b23.slice/crio-39e2721eb4c56ea50c5c0630e55d0d01d462d5b513179702baf1770f425166d0 WatchSource:0}: Error finding container 39e2721eb4c56ea50c5c0630e55d0d01d462d5b513179702baf1770f425166d0: Status 404 returned error can't find the container with id 39e2721eb4c56ea50c5c0630e55d0d01d462d5b513179702baf1770f425166d0
Jul 29 06:39:04.440190 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:04.440157    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-586ff9c89-4tgtz" event={"ID":"31c2224d-b518-44e9-97cf-2fc2cff44b23","Type":"ContainerStarted","Data":"49800384aac25a694a5ab814d740d472b967e40d167166095d9dd7f5fe0e4f4c"}
Jul 29 06:39:04.440638 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:04.440202    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-586ff9c89-4tgtz" event={"ID":"31c2224d-b518-44e9-97cf-2fc2cff44b23","Type":"ContainerStarted","Data":"39e2721eb4c56ea50c5c0630e55d0d01d462d5b513179702baf1770f425166d0"}
Jul 29 06:39:04.453661 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:04.453618    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-586ff9c89-4tgtz" podStartSLOduration=0.915642556 podStartE2EDuration="1.453603171s" totalImagesPullingTime="537.960615ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:39:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:39:04.45278282 +0000 UTC m=+987.530414591" watchObservedRunningTime="2026-07-29 06:39:04.453603171 +0000 UTC m=+987.531234938"
Jul 29 06:39:51.854041 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:51.854005    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-677c488c9b-dxrl5"]
Jul 29 06:39:51.854531 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:51.854258    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-677c488c9b-dxrl5" podUID="9c49d6dd-a863-460a-ae03-88b3948337a1" containerName="manager" containerID="cri-o://07e1498e57b8030d70f00164f9ce2bba9e35be7e08cce0ff24ba056d1554ed20" gracePeriod=30
Jul 29 06:39:52.104305 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:52.104241    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-677c488c9b-dxrl5"
Jul 29 06:39:52.298427 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:52.298391    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9c49d6dd-a863-460a-ae03-88b3948337a1-webhook-cert\") pod \"9c49d6dd-a863-460a-ae03-88b3948337a1\" (UID: \"9c49d6dd-a863-460a-ae03-88b3948337a1\") "
Jul 29 06:39:52.298622 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:52.298447    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7l2r\" (UniqueName: \"kubernetes.io/projected/9c49d6dd-a863-460a-ae03-88b3948337a1-kube-api-access-z7l2r\") pod \"9c49d6dd-a863-460a-ae03-88b3948337a1\" (UID: \"9c49d6dd-a863-460a-ae03-88b3948337a1\") "
Jul 29 06:39:52.300638 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:52.300607    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c49d6dd-a863-460a-ae03-88b3948337a1-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9c49d6dd-a863-460a-ae03-88b3948337a1" (UID: "9c49d6dd-a863-460a-ae03-88b3948337a1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 06:39:52.300746 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:52.300708    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c49d6dd-a863-460a-ae03-88b3948337a1-kube-api-access-z7l2r" (OuterVolumeSpecName: "kube-api-access-z7l2r") pod "9c49d6dd-a863-460a-ae03-88b3948337a1" (UID: "9c49d6dd-a863-460a-ae03-88b3948337a1"). InnerVolumeSpecName "kube-api-access-z7l2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 06:39:52.399578 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:52.399504    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7l2r\" (UniqueName: \"kubernetes.io/projected/9c49d6dd-a863-460a-ae03-88b3948337a1-kube-api-access-z7l2r\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:39:52.399578 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:52.399529    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9c49d6dd-a863-460a-ae03-88b3948337a1-webhook-cert\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:39:52.672333 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:52.672240    2563 generic.go:356] "Generic (PLEG): container finished" podID="9c49d6dd-a863-460a-ae03-88b3948337a1" containerID="07e1498e57b8030d70f00164f9ce2bba9e35be7e08cce0ff24ba056d1554ed20" exitCode=0
Jul 29 06:39:52.672333 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:52.672296    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-677c488c9b-dxrl5" event={"ID":"9c49d6dd-a863-460a-ae03-88b3948337a1","Type":"ContainerDied","Data":"07e1498e57b8030d70f00164f9ce2bba9e35be7e08cce0ff24ba056d1554ed20"}
Jul 29 06:39:52.672507 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:52.672381    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-677c488c9b-dxrl5" event={"ID":"9c49d6dd-a863-460a-ae03-88b3948337a1","Type":"ContainerDied","Data":"5c092da2238358288160fcdf527f68383a423aa633d6f4103627d490c6b7b4be"}
Jul 29 06:39:52.672507 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:52.672310    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-677c488c9b-dxrl5"
Jul 29 06:39:52.672507 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:52.672404    2563 scope.go:117] "RemoveContainer" containerID="07e1498e57b8030d70f00164f9ce2bba9e35be7e08cce0ff24ba056d1554ed20"
Jul 29 06:39:52.681974 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:52.681957    2563 scope.go:117] "RemoveContainer" containerID="07e1498e57b8030d70f00164f9ce2bba9e35be7e08cce0ff24ba056d1554ed20"
Jul 29 06:39:52.682231 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:39:52.682212    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07e1498e57b8030d70f00164f9ce2bba9e35be7e08cce0ff24ba056d1554ed20\": container with ID starting with 07e1498e57b8030d70f00164f9ce2bba9e35be7e08cce0ff24ba056d1554ed20 not found: ID does not exist" containerID="07e1498e57b8030d70f00164f9ce2bba9e35be7e08cce0ff24ba056d1554ed20"
Jul 29 06:39:52.682285 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:52.682239    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07e1498e57b8030d70f00164f9ce2bba9e35be7e08cce0ff24ba056d1554ed20"} err="failed to get container status \"07e1498e57b8030d70f00164f9ce2bba9e35be7e08cce0ff24ba056d1554ed20\": rpc error: code = NotFound desc = could not find container \"07e1498e57b8030d70f00164f9ce2bba9e35be7e08cce0ff24ba056d1554ed20\": container with ID starting with 07e1498e57b8030d70f00164f9ce2bba9e35be7e08cce0ff24ba056d1554ed20 not found: ID does not exist"
Jul 29 06:39:52.691892 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:52.691870    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-677c488c9b-dxrl5"]
Jul 29 06:39:52.695597 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:52.695578    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-677c488c9b-dxrl5"]
Jul 29 06:39:53.397464 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:39:53.397428    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c49d6dd-a863-460a-ae03-88b3948337a1" path="/var/lib/kubelet/pods/9c49d6dd-a863-460a-ae03-88b3948337a1/volumes"
Jul 29 06:40:53.471063 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:40:53.471030    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-677c488c9b-9lkwd"]
Jul 29 06:40:53.471868 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:40:53.471815    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c49d6dd-a863-460a-ae03-88b3948337a1" containerName="manager"
Jul 29 06:40:53.471868 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:40:53.471870    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c49d6dd-a863-460a-ae03-88b3948337a1" containerName="manager"
Jul 29 06:40:53.474292 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:40:53.472314    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c49d6dd-a863-460a-ae03-88b3948337a1" containerName="manager"
Jul 29 06:40:53.478172 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:40:53.478146    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-677c488c9b-9lkwd"
Jul 29 06:40:53.481029 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:40:53.481009    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 29 06:40:53.481330 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:40:53.481035    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kkv5x\""
Jul 29 06:40:53.481590 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:40:53.481036    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 29 06:40:53.482488 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:40:53.482465    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-677c488c9b-9lkwd"]
Jul 29 06:40:53.551833 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:40:53.551794    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a458285-736b-42bf-8364-658ec80ab42a-webhook-cert\") pod \"maas-controller-677c488c9b-9lkwd\" (UID: \"8a458285-736b-42bf-8364-658ec80ab42a\") " pod="opendatahub/maas-controller-677c488c9b-9lkwd"
Jul 29 06:40:53.552020 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:40:53.551899    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98kb9\" (UniqueName: \"kubernetes.io/projected/8a458285-736b-42bf-8364-658ec80ab42a-kube-api-access-98kb9\") pod \"maas-controller-677c488c9b-9lkwd\" (UID: \"8a458285-736b-42bf-8364-658ec80ab42a\") " pod="opendatahub/maas-controller-677c488c9b-9lkwd"
Jul 29 06:40:53.652922 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:40:53.652884    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a458285-736b-42bf-8364-658ec80ab42a-webhook-cert\") pod \"maas-controller-677c488c9b-9lkwd\" (UID: \"8a458285-736b-42bf-8364-658ec80ab42a\") " pod="opendatahub/maas-controller-677c488c9b-9lkwd"
Jul 29 06:40:53.653104 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:40:53.652950    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98kb9\" (UniqueName: \"kubernetes.io/projected/8a458285-736b-42bf-8364-658ec80ab42a-kube-api-access-98kb9\") pod \"maas-controller-677c488c9b-9lkwd\" (UID: \"8a458285-736b-42bf-8364-658ec80ab42a\") " pod="opendatahub/maas-controller-677c488c9b-9lkwd"
Jul 29 06:40:53.655420 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:40:53.655397    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a458285-736b-42bf-8364-658ec80ab42a-webhook-cert\") pod \"maas-controller-677c488c9b-9lkwd\" (UID: \"8a458285-736b-42bf-8364-658ec80ab42a\") " pod="opendatahub/maas-controller-677c488c9b-9lkwd"
Jul 29 06:40:53.660174 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:40:53.660152    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98kb9\" (UniqueName: \"kubernetes.io/projected/8a458285-736b-42bf-8364-658ec80ab42a-kube-api-access-98kb9\") pod \"maas-controller-677c488c9b-9lkwd\" (UID: \"8a458285-736b-42bf-8364-658ec80ab42a\") " pod="opendatahub/maas-controller-677c488c9b-9lkwd"
Jul 29 06:40:53.790734 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:40:53.790647    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-677c488c9b-9lkwd"
Jul 29 06:40:54.132658 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:40:54.132636    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-677c488c9b-9lkwd"]
Jul 29 06:40:54.135187 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:40:54.135154    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a458285_736b_42bf_8364_658ec80ab42a.slice/crio-0179b53cc309d613ada4e43cbdeee4ad78dc997d1e91b66a0f1877a1de16bdbd WatchSource:0}: Error finding container 0179b53cc309d613ada4e43cbdeee4ad78dc997d1e91b66a0f1877a1de16bdbd: Status 404 returned error can't find the container with id 0179b53cc309d613ada4e43cbdeee4ad78dc997d1e91b66a0f1877a1de16bdbd
Jul 29 06:40:54.136408 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:40:54.136388    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 06:40:54.981908 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:40:54.981801    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-677c488c9b-9lkwd" event={"ID":"8a458285-736b-42bf-8364-658ec80ab42a","Type":"ContainerStarted","Data":"1c1db28ebf5c482fda965099a262bfc9c9b9bc6e86f6a126d2f04682c68993ae"}
Jul 29 06:40:54.981908 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:40:54.981876    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-677c488c9b-9lkwd" event={"ID":"8a458285-736b-42bf-8364-658ec80ab42a","Type":"ContainerStarted","Data":"0179b53cc309d613ada4e43cbdeee4ad78dc997d1e91b66a0f1877a1de16bdbd"}
Jul 29 06:40:54.982299 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:40:54.981933    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-677c488c9b-9lkwd"
Jul 29 06:40:54.999591 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:40:54.999542    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-677c488c9b-9lkwd" podStartSLOduration=1.446570423 podStartE2EDuration="1.999524583s" totalImagesPullingTime="552.95416ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:40:54.997683245 +0000 UTC m=+1098.075315010" watchObservedRunningTime="2026-07-29 06:40:54.999524583 +0000 UTC m=+1098.077156351"
Jul 29 06:41:05.988415 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:05.988381    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-677c488c9b-9lkwd"
Jul 29 06:41:10.594996 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:10.594962    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-677c488c9b-9lkwd"]
Jul 29 06:41:10.595575 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:10.595149    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-677c488c9b-9lkwd" podUID="8a458285-736b-42bf-8364-658ec80ab42a" containerName="manager" containerID="cri-o://1c1db28ebf5c482fda965099a262bfc9c9b9bc6e86f6a126d2f04682c68993ae" gracePeriod=30
Jul 29 06:41:10.841188 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:10.841164    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-677c488c9b-9lkwd"
Jul 29 06:41:10.905036 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:10.904954    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98kb9\" (UniqueName: \"kubernetes.io/projected/8a458285-736b-42bf-8364-658ec80ab42a-kube-api-access-98kb9\") pod \"8a458285-736b-42bf-8364-658ec80ab42a\" (UID: \"8a458285-736b-42bf-8364-658ec80ab42a\") "
Jul 29 06:41:10.905167 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:10.905056    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a458285-736b-42bf-8364-658ec80ab42a-webhook-cert\") pod \"8a458285-736b-42bf-8364-658ec80ab42a\" (UID: \"8a458285-736b-42bf-8364-658ec80ab42a\") "
Jul 29 06:41:10.907070 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:10.907037    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a458285-736b-42bf-8364-658ec80ab42a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "8a458285-736b-42bf-8364-658ec80ab42a" (UID: "8a458285-736b-42bf-8364-658ec80ab42a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 06:41:10.907181 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:10.907110    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a458285-736b-42bf-8364-658ec80ab42a-kube-api-access-98kb9" (OuterVolumeSpecName: "kube-api-access-98kb9") pod "8a458285-736b-42bf-8364-658ec80ab42a" (UID: "8a458285-736b-42bf-8364-658ec80ab42a"). InnerVolumeSpecName "kube-api-access-98kb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 06:41:11.006011 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:11.005982    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a458285-736b-42bf-8364-658ec80ab42a-webhook-cert\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:41:11.006011 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:11.006007    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98kb9\" (UniqueName: \"kubernetes.io/projected/8a458285-736b-42bf-8364-658ec80ab42a-kube-api-access-98kb9\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:41:11.059559 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:11.059531    2563 generic.go:356] "Generic (PLEG): container finished" podID="8a458285-736b-42bf-8364-658ec80ab42a" containerID="1c1db28ebf5c482fda965099a262bfc9c9b9bc6e86f6a126d2f04682c68993ae" exitCode=0
Jul 29 06:41:11.059706 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:11.059572    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-677c488c9b-9lkwd"
Jul 29 06:41:11.059706 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:11.059625    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-677c488c9b-9lkwd" event={"ID":"8a458285-736b-42bf-8364-658ec80ab42a","Type":"ContainerDied","Data":"1c1db28ebf5c482fda965099a262bfc9c9b9bc6e86f6a126d2f04682c68993ae"}
Jul 29 06:41:11.059706 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:11.059674    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-677c488c9b-9lkwd" event={"ID":"8a458285-736b-42bf-8364-658ec80ab42a","Type":"ContainerDied","Data":"0179b53cc309d613ada4e43cbdeee4ad78dc997d1e91b66a0f1877a1de16bdbd"}
Jul 29 06:41:11.059819 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:11.059707    2563 scope.go:117] "RemoveContainer" containerID="1c1db28ebf5c482fda965099a262bfc9c9b9bc6e86f6a126d2f04682c68993ae"
Jul 29 06:41:11.068806 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:11.068790    2563 scope.go:117] "RemoveContainer" containerID="1c1db28ebf5c482fda965099a262bfc9c9b9bc6e86f6a126d2f04682c68993ae"
Jul 29 06:41:11.069097 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:41:11.069079    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c1db28ebf5c482fda965099a262bfc9c9b9bc6e86f6a126d2f04682c68993ae\": container with ID starting with 1c1db28ebf5c482fda965099a262bfc9c9b9bc6e86f6a126d2f04682c68993ae not found: ID does not exist" containerID="1c1db28ebf5c482fda965099a262bfc9c9b9bc6e86f6a126d2f04682c68993ae"
Jul 29 06:41:11.069176 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:11.069108    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c1db28ebf5c482fda965099a262bfc9c9b9bc6e86f6a126d2f04682c68993ae"} err="failed to get container status \"1c1db28ebf5c482fda965099a262bfc9c9b9bc6e86f6a126d2f04682c68993ae\": rpc error: code = NotFound desc = could not find container \"1c1db28ebf5c482fda965099a262bfc9c9b9bc6e86f6a126d2f04682c68993ae\": container with ID starting with 1c1db28ebf5c482fda965099a262bfc9c9b9bc6e86f6a126d2f04682c68993ae not found: ID does not exist"
Jul 29 06:41:11.082419 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:11.082397    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-677c488c9b-9lkwd"]
Jul 29 06:41:11.086040 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:11.086021    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-677c488c9b-9lkwd"]
Jul 29 06:41:11.398050 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:11.398018    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a458285-736b-42bf-8364-658ec80ab42a" path="/var/lib/kubelet/pods/8a458285-736b-42bf-8364-658ec80ab42a/volumes"
Jul 29 06:41:11.837347 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:11.837312    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-677c488c9b-8j2w7"]
Jul 29 06:41:11.837713 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:11.837692    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a458285-736b-42bf-8364-658ec80ab42a" containerName="manager"
Jul 29 06:41:11.837713 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:11.837702    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a458285-736b-42bf-8364-658ec80ab42a" containerName="manager"
Jul 29 06:41:11.837804 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:11.837773    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a458285-736b-42bf-8364-658ec80ab42a" containerName="manager"
Jul 29 06:41:11.842241 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:11.842220    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-677c488c9b-8j2w7"
Jul 29 06:41:11.844714 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:11.844696    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kkv5x\""
Jul 29 06:41:11.844828 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:11.844716    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 29 06:41:11.844828 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:11.844774    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 29 06:41:11.850956 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:11.850932    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-677c488c9b-8j2w7"]
Jul 29 06:41:11.913981 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:11.913945    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/014fe138-bc84-49bb-903e-dde1a4ddd7a6-webhook-cert\") pod \"maas-controller-677c488c9b-8j2w7\" (UID: \"014fe138-bc84-49bb-903e-dde1a4ddd7a6\") " pod="opendatahub/maas-controller-677c488c9b-8j2w7"
Jul 29 06:41:11.914139 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:11.914058    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p2tr\" (UniqueName: \"kubernetes.io/projected/014fe138-bc84-49bb-903e-dde1a4ddd7a6-kube-api-access-6p2tr\") pod \"maas-controller-677c488c9b-8j2w7\" (UID: \"014fe138-bc84-49bb-903e-dde1a4ddd7a6\") " pod="opendatahub/maas-controller-677c488c9b-8j2w7"
Jul 29 06:41:12.014523 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:12.014490    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p2tr\" (UniqueName: \"kubernetes.io/projected/014fe138-bc84-49bb-903e-dde1a4ddd7a6-kube-api-access-6p2tr\") pod \"maas-controller-677c488c9b-8j2w7\" (UID: \"014fe138-bc84-49bb-903e-dde1a4ddd7a6\") " pod="opendatahub/maas-controller-677c488c9b-8j2w7"
Jul 29 06:41:12.014523 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:12.014525    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/014fe138-bc84-49bb-903e-dde1a4ddd7a6-webhook-cert\") pod \"maas-controller-677c488c9b-8j2w7\" (UID: \"014fe138-bc84-49bb-903e-dde1a4ddd7a6\") " pod="opendatahub/maas-controller-677c488c9b-8j2w7"
Jul 29 06:41:12.017129 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:12.017101    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/014fe138-bc84-49bb-903e-dde1a4ddd7a6-webhook-cert\") pod \"maas-controller-677c488c9b-8j2w7\" (UID: \"014fe138-bc84-49bb-903e-dde1a4ddd7a6\") " pod="opendatahub/maas-controller-677c488c9b-8j2w7"
Jul 29 06:41:12.021400 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:12.021379    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p2tr\" (UniqueName: \"kubernetes.io/projected/014fe138-bc84-49bb-903e-dde1a4ddd7a6-kube-api-access-6p2tr\") pod \"maas-controller-677c488c9b-8j2w7\" (UID: \"014fe138-bc84-49bb-903e-dde1a4ddd7a6\") " pod="opendatahub/maas-controller-677c488c9b-8j2w7"
Jul 29 06:41:12.152799 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:12.152708    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-677c488c9b-8j2w7"
Jul 29 06:41:12.283442 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:12.283420    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-677c488c9b-8j2w7"]
Jul 29 06:41:12.285930 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:41:12.285904    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod014fe138_bc84_49bb_903e_dde1a4ddd7a6.slice/crio-731a6527ad578f182cf85d09fbb5e109d555629f90759cc2e406d328ffaa4606 WatchSource:0}: Error finding container 731a6527ad578f182cf85d09fbb5e109d555629f90759cc2e406d328ffaa4606: Status 404 returned error can't find the container with id 731a6527ad578f182cf85d09fbb5e109d555629f90759cc2e406d328ffaa4606
Jul 29 06:41:13.071503 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:13.071467    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-677c488c9b-8j2w7" event={"ID":"014fe138-bc84-49bb-903e-dde1a4ddd7a6","Type":"ContainerStarted","Data":"209efc9bbad3529c44ff8f6338e74b5ad4ea2f17afa3e5b5cde69e1e577494f6"}
Jul 29 06:41:13.072008 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:13.071535    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-677c488c9b-8j2w7" event={"ID":"014fe138-bc84-49bb-903e-dde1a4ddd7a6","Type":"ContainerStarted","Data":"731a6527ad578f182cf85d09fbb5e109d555629f90759cc2e406d328ffaa4606"}
Jul 29 06:41:13.072008 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:13.071564    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-677c488c9b-8j2w7"
Jul 29 06:41:13.087441 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:13.087389    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-677c488c9b-8j2w7" podStartSLOduration=1.638152454 podStartE2EDuration="2.087373308s" totalImagesPullingTime="449.220854ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:41:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:41:13.085517922 +0000 UTC m=+1116.163149690" watchObservedRunningTime="2026-07-29 06:41:13.087373308 +0000 UTC m=+1116.165005074"
Jul 29 06:41:24.079806 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:41:24.079769    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-677c488c9b-8j2w7"
Jul 29 06:45:00.141337 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:45:00.141255    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755125-8jwlj"]
Jul 29 06:45:00.145079 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:45:00.145062    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755125-8jwlj"
Jul 29 06:45:00.146957 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:45:00.146936    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bj47g\""
Jul 29 06:45:00.155460 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:45:00.155441    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755125-8jwlj"
Jul 29 06:45:00.157244 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:45:00.157219    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755125-8jwlj"]
Jul 29 06:45:00.294826 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:45:00.294801    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755125-8jwlj"]
Jul 29 06:45:00.297495 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:45:00.297454    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c89cce_87dc_4b43_b025_99153116c311.slice/crio-2fe682bb691660d9f33d21d96146f98873d4544a220d927e869d60fde8668ea0 WatchSource:0}: Error finding container 2fe682bb691660d9f33d21d96146f98873d4544a220d927e869d60fde8668ea0: Status 404 returned error can't find the container with id 2fe682bb691660d9f33d21d96146f98873d4544a220d927e869d60fde8668ea0
Jul 29 06:45:01.204534 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:45:01.204495    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755125-8jwlj" event={"ID":"05c89cce-87dc-4b43-b025-99153116c311","Type":"ContainerStarted","Data":"2fe682bb691660d9f33d21d96146f98873d4544a220d927e869d60fde8668ea0"}
Jul 29 06:45:04.223292 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:45:04.223256    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755125-8jwlj" event={"ID":"05c89cce-87dc-4b43-b025-99153116c311","Type":"ContainerStarted","Data":"2f993667c15cba52a8c311b6e3d465794135b9a81e5a93c7ed8b648060baa86b"}
Jul 29 06:45:04.236379 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:45:04.236329    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755125-8jwlj" podStartSLOduration=1.057786516 podStartE2EDuration="4.23631144s" totalImagesPullingTime="3.178524924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:45:04.235820204 +0000 UTC m=+1347.313451971" watchObservedRunningTime="2026-07-29 06:45:04.23631144 +0000 UTC m=+1347.313943208"
Jul 29 06:45:05.230291 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:45:05.230257    2563 generic.go:356] "Generic (PLEG): container finished" podID="05c89cce-87dc-4b43-b025-99153116c311" containerID="2f993667c15cba52a8c311b6e3d465794135b9a81e5a93c7ed8b648060baa86b" exitCode=0
Jul 29 06:45:05.230660 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:45:05.230339    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755125-8jwlj" event={"ID":"05c89cce-87dc-4b43-b025-99153116c311","Type":"ContainerDied","Data":"2f993667c15cba52a8c311b6e3d465794135b9a81e5a93c7ed8b648060baa86b"}
Jul 29 06:45:06.373271 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:45:06.373243    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755125-8jwlj"
Jul 29 06:45:07.244451 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:45:07.244415    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755125-8jwlj"
Jul 29 06:45:07.244623 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:45:07.244455    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755125-8jwlj" event={"ID":"05c89cce-87dc-4b43-b025-99153116c311","Type":"ContainerDied","Data":"2fe682bb691660d9f33d21d96146f98873d4544a220d927e869d60fde8668ea0"}
Jul 29 06:45:07.244623 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:45:07.244493    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fe682bb691660d9f33d21d96146f98873d4544a220d927e869d60fde8668ea0"
Jul 29 06:50:00.600596 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:00.600564    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x"]
Jul 29 06:50:00.601829 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:00.601065    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05c89cce-87dc-4b43-b025-99153116c311" containerName="cleanup"
Jul 29 06:50:00.601829 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:00.601079    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="05c89cce-87dc-4b43-b025-99153116c311" containerName="cleanup"
Jul 29 06:50:00.601829 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:00.601156    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="05c89cce-87dc-4b43-b025-99153116c311" containerName="cleanup"
Jul 29 06:50:00.602876 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:00.602839    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x"
Jul 29 06:50:00.604985 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:00.604963    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-697d22b0-kserve-self-signed-certs\""
Jul 29 06:50:00.613149 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:00.613132    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x"]
Jul 29 06:50:00.714464 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:00.714434    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-dshm\") pod \"e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x\" (UID: \"0f43b59f-3318-41bd-b76f-0f5061bd7cc7\") " pod="llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x"
Jul 29 06:50:00.714646 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:00.714472    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-home\") pod \"e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x\" (UID: \"0f43b59f-3318-41bd-b76f-0f5061bd7cc7\") " pod="llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x"
Jul 29 06:50:00.714646 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:00.714491    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-tmp-dir\") pod \"e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x\" (UID: \"0f43b59f-3318-41bd-b76f-0f5061bd7cc7\") " pod="llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x"
Jul 29 06:50:00.714646 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:00.714552    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-model-cache\") pod \"e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x\" (UID: \"0f43b59f-3318-41bd-b76f-0f5061bd7cc7\") " pod="llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x"
Jul 29 06:50:00.714646 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:00.714587    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-tls-certs\") pod \"e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x\" (UID: \"0f43b59f-3318-41bd-b76f-0f5061bd7cc7\") " pod="llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x"
Jul 29 06:50:00.714824 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:00.714666    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d8lk\" (UniqueName: \"kubernetes.io/projected/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-kube-api-access-8d8lk\") pod \"e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x\" (UID: \"0f43b59f-3318-41bd-b76f-0f5061bd7cc7\") " pod="llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x"
Jul 29 06:50:00.816068 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:00.816034    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-model-cache\") pod \"e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x\" (UID: \"0f43b59f-3318-41bd-b76f-0f5061bd7cc7\") " pod="llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x"
Jul 29 06:50:00.816255 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:00.816083    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-tls-certs\") pod \"e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x\" (UID: \"0f43b59f-3318-41bd-b76f-0f5061bd7cc7\") " pod="llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x"
Jul 29 06:50:00.816255 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:00.816114    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d8lk\" (UniqueName: \"kubernetes.io/projected/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-kube-api-access-8d8lk\") pod \"e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x\" (UID: \"0f43b59f-3318-41bd-b76f-0f5061bd7cc7\") " pod="llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x"
Jul 29 06:50:00.816255 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:00.816142    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-dshm\") pod \"e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x\" (UID: \"0f43b59f-3318-41bd-b76f-0f5061bd7cc7\") " pod="llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x"
Jul 29 06:50:00.816255 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:00.816161    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-home\") pod \"e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x\" (UID: \"0f43b59f-3318-41bd-b76f-0f5061bd7cc7\") " pod="llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x"
Jul 29 06:50:00.816255 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:00.816178    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-tmp-dir\") pod \"e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x\" (UID: \"0f43b59f-3318-41bd-b76f-0f5061bd7cc7\") " pod="llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x"
Jul 29 06:50:00.816533 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:00.816515    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-model-cache\") pod \"e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x\" (UID: \"0f43b59f-3318-41bd-b76f-0f5061bd7cc7\") " pod="llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x"
Jul 29 06:50:00.816533 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:00.816525    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-home\") pod \"e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x\" (UID: \"0f43b59f-3318-41bd-b76f-0f5061bd7cc7\") " pod="llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x"
Jul 29 06:50:00.816610 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:00.816580    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-tmp-dir\") pod \"e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x\" (UID: \"0f43b59f-3318-41bd-b76f-0f5061bd7cc7\") " pod="llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x"
Jul 29 06:50:00.818515 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:00.818488    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-dshm\") pod \"e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x\" (UID: \"0f43b59f-3318-41bd-b76f-0f5061bd7cc7\") " pod="llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x"
Jul 29 06:50:00.818803 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:00.818785    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-tls-certs\") pod \"e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x\" (UID: \"0f43b59f-3318-41bd-b76f-0f5061bd7cc7\") " pod="llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x"
Jul 29 06:50:00.823093 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:00.823073    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d8lk\" (UniqueName: \"kubernetes.io/projected/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-kube-api-access-8d8lk\") pod \"e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x\" (UID: \"0f43b59f-3318-41bd-b76f-0f5061bd7cc7\") " pod="llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x"
Jul 29 06:50:00.912988 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:00.912904    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x"
Jul 29 06:50:01.044359 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:01.044333    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x"]
Jul 29 06:50:01.046155 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:50:01.046122    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f43b59f_3318_41bd_b76f_0f5061bd7cc7.slice/crio-e373d2606bba3068e3f61962b757ed6132dc61471c3e9f7a74da77fad17e09d1 WatchSource:0}: Error finding container e373d2606bba3068e3f61962b757ed6132dc61471c3e9f7a74da77fad17e09d1: Status 404 returned error can't find the container with id e373d2606bba3068e3f61962b757ed6132dc61471c3e9f7a74da77fad17e09d1
Jul 29 06:50:01.684424 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:01.684389    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x" event={"ID":"0f43b59f-3318-41bd-b76f-0f5061bd7cc7","Type":"ContainerStarted","Data":"350a2f8bc7ceb8cb09de2b5d388fa7731c4287a1a4ad232cfdc4222d3f5b6893"}
Jul 29 06:50:01.684424 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:01.684426    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x" event={"ID":"0f43b59f-3318-41bd-b76f-0f5061bd7cc7","Type":"ContainerStarted","Data":"e373d2606bba3068e3f61962b757ed6132dc61471c3e9f7a74da77fad17e09d1"}
Jul 29 06:50:01.699862 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:01.699815    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x" podStartSLOduration=1.699800675 podStartE2EDuration="1.699800675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:50:01.697748965 +0000 UTC m=+1644.775380732" watchObservedRunningTime="2026-07-29 06:50:01.699800675 +0000 UTC m=+1644.777432441"
Jul 29 06:50:10.913807 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:10.913767    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x"
Jul 29 06:50:10.914260 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:10.913823    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x"
Jul 29 06:50:10.926422 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:10.926396    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x"
Jul 29 06:50:11.750232 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:11.750206    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x"
Jul 29 06:50:19.848754 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:19.848718    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x"]
Jul 29 06:50:34.869337 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:34.869293    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x" podUID="0f43b59f-3318-41bd-b76f-0f5061bd7cc7" containerName="main" containerID="cri-o://350a2f8bc7ceb8cb09de2b5d388fa7731c4287a1a4ad232cfdc4222d3f5b6893" gracePeriod=45
Jul 29 06:50:35.110890 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:35.110867    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x"
Jul 29 06:50:35.237801 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:35.237719    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-home\") pod \"0f43b59f-3318-41bd-b76f-0f5061bd7cc7\" (UID: \"0f43b59f-3318-41bd-b76f-0f5061bd7cc7\") "
Jul 29 06:50:35.237801 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:35.237753    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-tls-certs\") pod \"0f43b59f-3318-41bd-b76f-0f5061bd7cc7\" (UID: \"0f43b59f-3318-41bd-b76f-0f5061bd7cc7\") "
Jul 29 06:50:35.237801 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:35.237791    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-model-cache\") pod \"0f43b59f-3318-41bd-b76f-0f5061bd7cc7\" (UID: \"0f43b59f-3318-41bd-b76f-0f5061bd7cc7\") "
Jul 29 06:50:35.238103 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:35.237877    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-tmp-dir\") pod \"0f43b59f-3318-41bd-b76f-0f5061bd7cc7\" (UID: \"0f43b59f-3318-41bd-b76f-0f5061bd7cc7\") "
Jul 29 06:50:35.238103 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:35.237948    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-dshm\") pod \"0f43b59f-3318-41bd-b76f-0f5061bd7cc7\" (UID: \"0f43b59f-3318-41bd-b76f-0f5061bd7cc7\") "
Jul 29 06:50:35.238103 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:35.237973    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8d8lk\" (UniqueName: \"kubernetes.io/projected/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-kube-api-access-8d8lk\") pod \"0f43b59f-3318-41bd-b76f-0f5061bd7cc7\" (UID: \"0f43b59f-3318-41bd-b76f-0f5061bd7cc7\") "
Jul 29 06:50:35.238103 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:35.238067    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-home" (OuterVolumeSpecName: "home") pod "0f43b59f-3318-41bd-b76f-0f5061bd7cc7" (UID: "0f43b59f-3318-41bd-b76f-0f5061bd7cc7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 06:50:35.238307 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:35.238098    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-model-cache" (OuterVolumeSpecName: "model-cache") pod "0f43b59f-3318-41bd-b76f-0f5061bd7cc7" (UID: "0f43b59f-3318-41bd-b76f-0f5061bd7cc7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 06:50:35.238307 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:35.238207    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0f43b59f-3318-41bd-b76f-0f5061bd7cc7" (UID: "0f43b59f-3318-41bd-b76f-0f5061bd7cc7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 06:50:35.238413 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:35.238363    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-model-cache\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:50:35.238413 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:35.238407    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-tmp-dir\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:50:35.238493 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:35.238422    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-home\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:50:35.240177 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:35.240143    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0f43b59f-3318-41bd-b76f-0f5061bd7cc7" (UID: "0f43b59f-3318-41bd-b76f-0f5061bd7cc7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 06:50:35.240273 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:35.240230    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-kube-api-access-8d8lk" (OuterVolumeSpecName: "kube-api-access-8d8lk") pod "0f43b59f-3318-41bd-b76f-0f5061bd7cc7" (UID: "0f43b59f-3318-41bd-b76f-0f5061bd7cc7"). InnerVolumeSpecName "kube-api-access-8d8lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 06:50:35.240273 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:35.240243    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-dshm" (OuterVolumeSpecName: "dshm") pod "0f43b59f-3318-41bd-b76f-0f5061bd7cc7" (UID: "0f43b59f-3318-41bd-b76f-0f5061bd7cc7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 06:50:35.339209 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:35.339185    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-dshm\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:50:35.339209 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:35.339206    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d8lk\" (UniqueName: \"kubernetes.io/projected/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-kube-api-access-8d8lk\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:50:35.339367 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:35.339217    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f43b59f-3318-41bd-b76f-0f5061bd7cc7-tls-certs\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:50:35.858722 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:35.858692    2563 generic.go:356] "Generic (PLEG): container finished" podID="0f43b59f-3318-41bd-b76f-0f5061bd7cc7" containerID="350a2f8bc7ceb8cb09de2b5d388fa7731c4287a1a4ad232cfdc4222d3f5b6893" exitCode=0
Jul 29 06:50:35.858929 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:35.858738    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x"
Jul 29 06:50:35.858929 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:35.858779    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x" event={"ID":"0f43b59f-3318-41bd-b76f-0f5061bd7cc7","Type":"ContainerDied","Data":"350a2f8bc7ceb8cb09de2b5d388fa7731c4287a1a4ad232cfdc4222d3f5b6893"}
Jul 29 06:50:35.858929 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:35.858834    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x" event={"ID":"0f43b59f-3318-41bd-b76f-0f5061bd7cc7","Type":"ContainerDied","Data":"e373d2606bba3068e3f61962b757ed6132dc61471c3e9f7a74da77fad17e09d1"}
Jul 29 06:50:35.858929 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:35.858878    2563 scope.go:117] "RemoveContainer" containerID="350a2f8bc7ceb8cb09de2b5d388fa7731c4287a1a4ad232cfdc4222d3f5b6893"
Jul 29 06:50:35.867897 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:35.867876    2563 scope.go:117] "RemoveContainer" containerID="350a2f8bc7ceb8cb09de2b5d388fa7731c4287a1a4ad232cfdc4222d3f5b6893"
Jul 29 06:50:35.868151 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:50:35.868132    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"350a2f8bc7ceb8cb09de2b5d388fa7731c4287a1a4ad232cfdc4222d3f5b6893\": container with ID starting with 350a2f8bc7ceb8cb09de2b5d388fa7731c4287a1a4ad232cfdc4222d3f5b6893 not found: ID does not exist" containerID="350a2f8bc7ceb8cb09de2b5d388fa7731c4287a1a4ad232cfdc4222d3f5b6893"
Jul 29 06:50:35.868205 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:35.868161    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"350a2f8bc7ceb8cb09de2b5d388fa7731c4287a1a4ad232cfdc4222d3f5b6893"} err="failed to get container status \"350a2f8bc7ceb8cb09de2b5d388fa7731c4287a1a4ad232cfdc4222d3f5b6893\": rpc error: code = NotFound desc = could not find container \"350a2f8bc7ceb8cb09de2b5d388fa7731c4287a1a4ad232cfdc4222d3f5b6893\": container with ID starting with 350a2f8bc7ceb8cb09de2b5d388fa7731c4287a1a4ad232cfdc4222d3f5b6893 not found: ID does not exist"
Jul 29 06:50:35.872307 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:35.872286    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x"]
Jul 29 06:50:35.875637 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:35.875616    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-697d22b0-kserve-cd57d8775-4s42x"]
Jul 29 06:50:37.397386 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:50:37.397351    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f43b59f-3318-41bd-b76f-0f5061bd7cc7" path="/var/lib/kubelet/pods/0f43b59f-3318-41bd-b76f-0f5061bd7cc7/volumes"
Jul 29 06:53:59.799683 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:53:59.799594    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d"]
Jul 29 06:53:59.800162 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:53:59.800051    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f43b59f-3318-41bd-b76f-0f5061bd7cc7" containerName="main"
Jul 29 06:53:59.800162 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:53:59.800063    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f43b59f-3318-41bd-b76f-0f5061bd7cc7" containerName="main"
Jul 29 06:53:59.800162 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:53:59.800144    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f43b59f-3318-41bd-b76f-0f5061bd7cc7" containerName="main"
Jul 29 06:53:59.803389 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:53:59.803374    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d"
Jul 29 06:53:59.805391 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:53:59.805363    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-93562a48-serving-cert\""
Jul 29 06:53:59.811958 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:53:59.811463    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d"]
Jul 29 06:53:59.983275 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:53:59.983246    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3e0dc2b-36ae-4dda-83e9-657c59912627-openshift-service-ca\") pod \"maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d\" (UID: \"c3e0dc2b-36ae-4dda-83e9-657c59912627\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d"
Jul 29 06:53:59.983464 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:53:59.983290    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m8ts\" (UniqueName: \"kubernetes.io/projected/c3e0dc2b-36ae-4dda-83e9-657c59912627-kube-api-access-4m8ts\") pod \"maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d\" (UID: \"c3e0dc2b-36ae-4dda-83e9-657c59912627\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d"
Jul 29 06:53:59.983464 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:53:59.983310    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3e0dc2b-36ae-4dda-83e9-657c59912627-maas-api-tls\") pod \"maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d\" (UID: \"c3e0dc2b-36ae-4dda-83e9-657c59912627\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d"
Jul 29 06:54:00.084259 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:00.084228    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3e0dc2b-36ae-4dda-83e9-657c59912627-openshift-service-ca\") pod \"maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d\" (UID: \"c3e0dc2b-36ae-4dda-83e9-657c59912627\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d"
Jul 29 06:54:00.084423 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:00.084265    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m8ts\" (UniqueName: \"kubernetes.io/projected/c3e0dc2b-36ae-4dda-83e9-657c59912627-kube-api-access-4m8ts\") pod \"maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d\" (UID: \"c3e0dc2b-36ae-4dda-83e9-657c59912627\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d"
Jul 29 06:54:00.084423 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:00.084281    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3e0dc2b-36ae-4dda-83e9-657c59912627-maas-api-tls\") pod \"maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d\" (UID: \"c3e0dc2b-36ae-4dda-83e9-657c59912627\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d"
Jul 29 06:54:00.084423 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:54:00.084388    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-93562a48-serving-cert: secret "maas-api-e2e-ait-93562a48-serving-cert" not found
Jul 29 06:54:00.084568 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:54:00.084450    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3e0dc2b-36ae-4dda-83e9-657c59912627-maas-api-tls podName:c3e0dc2b-36ae-4dda-83e9-657c59912627 nodeName:}" failed. No retries permitted until 2026-07-29 06:54:00.584433375 +0000 UTC m=+1883.662065119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c3e0dc2b-36ae-4dda-83e9-657c59912627-maas-api-tls") pod "maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d" (UID: "c3e0dc2b-36ae-4dda-83e9-657c59912627") : secret "maas-api-e2e-ait-93562a48-serving-cert" not found
Jul 29 06:54:00.084899 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:00.084844    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3e0dc2b-36ae-4dda-83e9-657c59912627-openshift-service-ca\") pod \"maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d\" (UID: \"c3e0dc2b-36ae-4dda-83e9-657c59912627\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d"
Jul 29 06:54:00.102015 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:00.101994    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m8ts\" (UniqueName: \"kubernetes.io/projected/c3e0dc2b-36ae-4dda-83e9-657c59912627-kube-api-access-4m8ts\") pod \"maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d\" (UID: \"c3e0dc2b-36ae-4dda-83e9-657c59912627\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d"
Jul 29 06:54:00.589624 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:00.589584    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3e0dc2b-36ae-4dda-83e9-657c59912627-maas-api-tls\") pod \"maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d\" (UID: \"c3e0dc2b-36ae-4dda-83e9-657c59912627\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d"
Jul 29 06:54:00.592390 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:00.592362    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3e0dc2b-36ae-4dda-83e9-657c59912627-maas-api-tls\") pod \"maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d\" (UID: \"c3e0dc2b-36ae-4dda-83e9-657c59912627\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d"
Jul 29 06:54:00.716395 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:00.716360    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d"
Jul 29 06:54:00.851822 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:00.851799    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d"]
Jul 29 06:54:00.854564 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:54:00.854535    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e0dc2b_36ae_4dda_83e9_657c59912627.slice/crio-9b7878000bc443237202ecb871775a75626e423ed799c4b202a352620a415bc9 WatchSource:0}: Error finding container 9b7878000bc443237202ecb871775a75626e423ed799c4b202a352620a415bc9: Status 404 returned error can't find the container with id 9b7878000bc443237202ecb871775a75626e423ed799c4b202a352620a415bc9
Jul 29 06:54:00.855976 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:00.855956    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 06:54:00.885425 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:00.885397    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d" event={"ID":"c3e0dc2b-36ae-4dda-83e9-657c59912627","Type":"ContainerStarted","Data":"9b7878000bc443237202ecb871775a75626e423ed799c4b202a352620a415bc9"}
Jul 29 06:54:01.892769 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:01.892723    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d" event={"ID":"c3e0dc2b-36ae-4dda-83e9-657c59912627","Type":"ContainerStarted","Data":"18e1b9fceb33b3b77916556c5f9ba080793fa079e18fe68bfd4dd752751bcf88"}
Jul 29 06:54:01.893188 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:01.892883    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d"
Jul 29 06:54:01.913885 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:01.911313    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d" podStartSLOduration=2.414932377 podStartE2EDuration="2.911291472s" totalImagesPullingTime="496.359095ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:54:01.905958788 +0000 UTC m=+1884.983590568" watchObservedRunningTime="2026-07-29 06:54:01.911291472 +0000 UTC m=+1884.988923242"
Jul 29 06:54:07.903758 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:07.903731    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d"
Jul 29 06:54:40.435633 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:40.435598    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d"]
Jul 29 06:54:40.436249 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:40.435871    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d" podUID="c3e0dc2b-36ae-4dda-83e9-657c59912627" containerName="maas-api" containerID="cri-o://18e1b9fceb33b3b77916556c5f9ba080793fa079e18fe68bfd4dd752751bcf88" gracePeriod=30
Jul 29 06:54:40.692483 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:40.692430    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d"
Jul 29 06:54:40.823974 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:40.823944    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3e0dc2b-36ae-4dda-83e9-657c59912627-maas-api-tls\") pod \"c3e0dc2b-36ae-4dda-83e9-657c59912627\" (UID: \"c3e0dc2b-36ae-4dda-83e9-657c59912627\") "
Jul 29 06:54:40.824135 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:40.823993    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4m8ts\" (UniqueName: \"kubernetes.io/projected/c3e0dc2b-36ae-4dda-83e9-657c59912627-kube-api-access-4m8ts\") pod \"c3e0dc2b-36ae-4dda-83e9-657c59912627\" (UID: \"c3e0dc2b-36ae-4dda-83e9-657c59912627\") "
Jul 29 06:54:40.824135 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:40.824055    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3e0dc2b-36ae-4dda-83e9-657c59912627-openshift-service-ca\") pod \"c3e0dc2b-36ae-4dda-83e9-657c59912627\" (UID: \"c3e0dc2b-36ae-4dda-83e9-657c59912627\") "
Jul 29 06:54:40.824433 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:40.824410    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3e0dc2b-36ae-4dda-83e9-657c59912627-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c3e0dc2b-36ae-4dda-83e9-657c59912627" (UID: "c3e0dc2b-36ae-4dda-83e9-657c59912627"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 06:54:40.826418 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:40.826396    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3e0dc2b-36ae-4dda-83e9-657c59912627-kube-api-access-4m8ts" (OuterVolumeSpecName: "kube-api-access-4m8ts") pod "c3e0dc2b-36ae-4dda-83e9-657c59912627" (UID: "c3e0dc2b-36ae-4dda-83e9-657c59912627"). InnerVolumeSpecName "kube-api-access-4m8ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 06:54:40.826554 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:40.826442    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3e0dc2b-36ae-4dda-83e9-657c59912627-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c3e0dc2b-36ae-4dda-83e9-657c59912627" (UID: "c3e0dc2b-36ae-4dda-83e9-657c59912627"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 06:54:40.925009 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:40.924975    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3e0dc2b-36ae-4dda-83e9-657c59912627-openshift-service-ca\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:54:40.925009 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:40.925005    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3e0dc2b-36ae-4dda-83e9-657c59912627-maas-api-tls\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:54:40.925009 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:40.925017    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m8ts\" (UniqueName: \"kubernetes.io/projected/c3e0dc2b-36ae-4dda-83e9-657c59912627-kube-api-access-4m8ts\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:54:41.089086 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:41.089055    2563 generic.go:356] "Generic (PLEG): container finished" podID="c3e0dc2b-36ae-4dda-83e9-657c59912627" containerID="18e1b9fceb33b3b77916556c5f9ba080793fa079e18fe68bfd4dd752751bcf88" exitCode=0
Jul 29 06:54:41.089258 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:41.089095    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d"
Jul 29 06:54:41.089258 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:41.089140    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d" event={"ID":"c3e0dc2b-36ae-4dda-83e9-657c59912627","Type":"ContainerDied","Data":"18e1b9fceb33b3b77916556c5f9ba080793fa079e18fe68bfd4dd752751bcf88"}
Jul 29 06:54:41.089258 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:41.089208    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d" event={"ID":"c3e0dc2b-36ae-4dda-83e9-657c59912627","Type":"ContainerDied","Data":"9b7878000bc443237202ecb871775a75626e423ed799c4b202a352620a415bc9"}
Jul 29 06:54:41.089258 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:41.089233    2563 scope.go:117] "RemoveContainer" containerID="18e1b9fceb33b3b77916556c5f9ba080793fa079e18fe68bfd4dd752751bcf88"
Jul 29 06:54:41.099206 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:41.099190    2563 scope.go:117] "RemoveContainer" containerID="18e1b9fceb33b3b77916556c5f9ba080793fa079e18fe68bfd4dd752751bcf88"
Jul 29 06:54:41.099473 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:54:41.099452    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18e1b9fceb33b3b77916556c5f9ba080793fa079e18fe68bfd4dd752751bcf88\": container with ID starting with 18e1b9fceb33b3b77916556c5f9ba080793fa079e18fe68bfd4dd752751bcf88 not found: ID does not exist" containerID="18e1b9fceb33b3b77916556c5f9ba080793fa079e18fe68bfd4dd752751bcf88"
Jul 29 06:54:41.099527 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:41.099481    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18e1b9fceb33b3b77916556c5f9ba080793fa079e18fe68bfd4dd752751bcf88"} err="failed to get container status \"18e1b9fceb33b3b77916556c5f9ba080793fa079e18fe68bfd4dd752751bcf88\": rpc error: code = NotFound desc = could not find container \"18e1b9fceb33b3b77916556c5f9ba080793fa079e18fe68bfd4dd752751bcf88\": container with ID starting with 18e1b9fceb33b3b77916556c5f9ba080793fa079e18fe68bfd4dd752751bcf88 not found: ID does not exist"
Jul 29 06:54:41.116572 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:41.116551    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d"]
Jul 29 06:54:41.120115 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:41.120096    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-93562a48-549bdb5bc4-kb85d"]
Jul 29 06:54:41.397454 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:41.397374    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3e0dc2b-36ae-4dda-83e9-657c59912627" path="/var/lib/kubelet/pods/c3e0dc2b-36ae-4dda-83e9-657c59912627/volumes"
Jul 29 06:54:44.116157 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:44.116125    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c"]
Jul 29 06:54:44.116834 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:44.116808    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3e0dc2b-36ae-4dda-83e9-657c59912627" containerName="maas-api"
Jul 29 06:54:44.116834 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:44.116833    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3e0dc2b-36ae-4dda-83e9-657c59912627" containerName="maas-api"
Jul 29 06:54:44.117007 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:44.116963    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3e0dc2b-36ae-4dda-83e9-657c59912627" containerName="maas-api"
Jul 29 06:54:44.121818 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:44.121800    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c"
Jul 29 06:54:44.124470 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:44.124441    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-89063c4d-serving-cert\""
Jul 29 06:54:44.126421 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:44.126395    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c"]
Jul 29 06:54:44.256242 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:44.256209    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lpl7\" (UniqueName: \"kubernetes.io/projected/1777a955-409e-46e8-a83d-a635743030c7-kube-api-access-8lpl7\") pod \"maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c\" (UID: \"1777a955-409e-46e8-a83d-a635743030c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c"
Jul 29 06:54:44.256399 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:44.256274    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1777a955-409e-46e8-a83d-a635743030c7-openshift-service-ca\") pod \"maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c\" (UID: \"1777a955-409e-46e8-a83d-a635743030c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c"
Jul 29 06:54:44.256399 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:44.256345    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1777a955-409e-46e8-a83d-a635743030c7-maas-api-tls\") pod \"maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c\" (UID: \"1777a955-409e-46e8-a83d-a635743030c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c"
Jul 29 06:54:44.357813 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:44.357779    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lpl7\" (UniqueName: \"kubernetes.io/projected/1777a955-409e-46e8-a83d-a635743030c7-kube-api-access-8lpl7\") pod \"maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c\" (UID: \"1777a955-409e-46e8-a83d-a635743030c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c"
Jul 29 06:54:44.357999 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:44.357880    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1777a955-409e-46e8-a83d-a635743030c7-openshift-service-ca\") pod \"maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c\" (UID: \"1777a955-409e-46e8-a83d-a635743030c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c"
Jul 29 06:54:44.357999 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:44.357919    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1777a955-409e-46e8-a83d-a635743030c7-maas-api-tls\") pod \"maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c\" (UID: \"1777a955-409e-46e8-a83d-a635743030c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c"
Jul 29 06:54:44.358517 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:44.358494    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1777a955-409e-46e8-a83d-a635743030c7-openshift-service-ca\") pod \"maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c\" (UID: \"1777a955-409e-46e8-a83d-a635743030c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c"
Jul 29 06:54:44.360569 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:44.360547    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1777a955-409e-46e8-a83d-a635743030c7-maas-api-tls\") pod \"maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c\" (UID: \"1777a955-409e-46e8-a83d-a635743030c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c"
Jul 29 06:54:44.364920 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:44.364899    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lpl7\" (UniqueName: \"kubernetes.io/projected/1777a955-409e-46e8-a83d-a635743030c7-kube-api-access-8lpl7\") pod \"maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c\" (UID: \"1777a955-409e-46e8-a83d-a635743030c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c"
Jul 29 06:54:44.435040 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:44.434976    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c"
Jul 29 06:54:44.574695 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:44.574664    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c"]
Jul 29 06:54:44.577931 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:54:44.577831    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1777a955_409e_46e8_a83d_a635743030c7.slice/crio-40b2719e68f226dc7adbae35f0847e8b1a55dfc99eb10f9dfc21977c7fc34473 WatchSource:0}: Error finding container 40b2719e68f226dc7adbae35f0847e8b1a55dfc99eb10f9dfc21977c7fc34473: Status 404 returned error can't find the container with id 40b2719e68f226dc7adbae35f0847e8b1a55dfc99eb10f9dfc21977c7fc34473
Jul 29 06:54:45.113374 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:45.113344    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c" event={"ID":"1777a955-409e-46e8-a83d-a635743030c7","Type":"ContainerStarted","Data":"40b2719e68f226dc7adbae35f0847e8b1a55dfc99eb10f9dfc21977c7fc34473"}
Jul 29 06:54:46.120582 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:46.120545    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c" event={"ID":"1777a955-409e-46e8-a83d-a635743030c7","Type":"ContainerStarted","Data":"74764b047fc8369584540e4c24bfb4b0593b85d15c850f6f905bcfb2d017dbbd"}
Jul 29 06:54:46.121082 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:46.120726    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c"
Jul 29 06:54:46.138210 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:46.138152    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c" podStartSLOduration=1.651695911 podStartE2EDuration="2.138139191s" totalImagesPullingTime="486.44328ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:54:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:54:46.136525576 +0000 UTC m=+1929.214157343" watchObservedRunningTime="2026-07-29 06:54:46.138139191 +0000 UTC m=+1929.215770957"
Jul 29 06:54:52.132272 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:54:52.132244    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c"
Jul 29 06:55:14.620104 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:14.620073    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c"]
Jul 29 06:55:14.620706 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:14.620272    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c" podUID="1777a955-409e-46e8-a83d-a635743030c7" containerName="maas-api" containerID="cri-o://74764b047fc8369584540e4c24bfb4b0593b85d15c850f6f905bcfb2d017dbbd" gracePeriod=30
Jul 29 06:55:14.877779 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:14.877720    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c"
Jul 29 06:55:14.937441 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:14.937417    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lpl7\" (UniqueName: \"kubernetes.io/projected/1777a955-409e-46e8-a83d-a635743030c7-kube-api-access-8lpl7\") pod \"1777a955-409e-46e8-a83d-a635743030c7\" (UID: \"1777a955-409e-46e8-a83d-a635743030c7\") "
Jul 29 06:55:14.937607 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:14.937548    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1777a955-409e-46e8-a83d-a635743030c7-openshift-service-ca\") pod \"1777a955-409e-46e8-a83d-a635743030c7\" (UID: \"1777a955-409e-46e8-a83d-a635743030c7\") "
Jul 29 06:55:14.937607 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:14.937602    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1777a955-409e-46e8-a83d-a635743030c7-maas-api-tls\") pod \"1777a955-409e-46e8-a83d-a635743030c7\" (UID: \"1777a955-409e-46e8-a83d-a635743030c7\") "
Jul 29 06:55:14.937924 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:14.937903    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1777a955-409e-46e8-a83d-a635743030c7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1777a955-409e-46e8-a83d-a635743030c7" (UID: "1777a955-409e-46e8-a83d-a635743030c7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 06:55:14.939708 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:14.939686    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1777a955-409e-46e8-a83d-a635743030c7-kube-api-access-8lpl7" (OuterVolumeSpecName: "kube-api-access-8lpl7") pod "1777a955-409e-46e8-a83d-a635743030c7" (UID: "1777a955-409e-46e8-a83d-a635743030c7"). InnerVolumeSpecName "kube-api-access-8lpl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 06:55:14.939708 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:14.939685    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1777a955-409e-46e8-a83d-a635743030c7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1777a955-409e-46e8-a83d-a635743030c7" (UID: "1777a955-409e-46e8-a83d-a635743030c7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 06:55:15.039298 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:15.039267    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1777a955-409e-46e8-a83d-a635743030c7-openshift-service-ca\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:55:15.039298 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:15.039295    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1777a955-409e-46e8-a83d-a635743030c7-maas-api-tls\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:55:15.039528 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:15.039310    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lpl7\" (UniqueName: \"kubernetes.io/projected/1777a955-409e-46e8-a83d-a635743030c7-kube-api-access-8lpl7\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:55:15.275063 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:15.274977    2563 generic.go:356] "Generic (PLEG): container finished" podID="1777a955-409e-46e8-a83d-a635743030c7" containerID="74764b047fc8369584540e4c24bfb4b0593b85d15c850f6f905bcfb2d017dbbd" exitCode=0
Jul 29 06:55:15.275063 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:15.275020    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c"
Jul 29 06:55:15.275259 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:15.275060    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c" event={"ID":"1777a955-409e-46e8-a83d-a635743030c7","Type":"ContainerDied","Data":"74764b047fc8369584540e4c24bfb4b0593b85d15c850f6f905bcfb2d017dbbd"}
Jul 29 06:55:15.275259 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:15.275105    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c" event={"ID":"1777a955-409e-46e8-a83d-a635743030c7","Type":"ContainerDied","Data":"40b2719e68f226dc7adbae35f0847e8b1a55dfc99eb10f9dfc21977c7fc34473"}
Jul 29 06:55:15.275259 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:15.275128    2563 scope.go:117] "RemoveContainer" containerID="74764b047fc8369584540e4c24bfb4b0593b85d15c850f6f905bcfb2d017dbbd"
Jul 29 06:55:15.285168 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:15.285146    2563 scope.go:117] "RemoveContainer" containerID="74764b047fc8369584540e4c24bfb4b0593b85d15c850f6f905bcfb2d017dbbd"
Jul 29 06:55:15.285468 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:55:15.285443    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74764b047fc8369584540e4c24bfb4b0593b85d15c850f6f905bcfb2d017dbbd\": container with ID starting with 74764b047fc8369584540e4c24bfb4b0593b85d15c850f6f905bcfb2d017dbbd not found: ID does not exist" containerID="74764b047fc8369584540e4c24bfb4b0593b85d15c850f6f905bcfb2d017dbbd"
Jul 29 06:55:15.285586 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:15.285479    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74764b047fc8369584540e4c24bfb4b0593b85d15c850f6f905bcfb2d017dbbd"} err="failed to get container status \"74764b047fc8369584540e4c24bfb4b0593b85d15c850f6f905bcfb2d017dbbd\": rpc error: code = NotFound desc = could not find container \"74764b047fc8369584540e4c24bfb4b0593b85d15c850f6f905bcfb2d017dbbd\": container with ID starting with 74764b047fc8369584540e4c24bfb4b0593b85d15c850f6f905bcfb2d017dbbd not found: ID does not exist"
Jul 29 06:55:15.295270 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:15.295246    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c"]
Jul 29 06:55:15.298958 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:15.298936    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-89063c4d-6795f6f5bd-wp29c"]
Jul 29 06:55:15.398175 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:15.398142    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1777a955-409e-46e8-a83d-a635743030c7" path="/var/lib/kubelet/pods/1777a955-409e-46e8-a83d-a635743030c7/volumes"
Jul 29 06:55:18.180107 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:18.180071    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb"]
Jul 29 06:55:18.180726 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:18.180706    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1777a955-409e-46e8-a83d-a635743030c7" containerName="maas-api"
Jul 29 06:55:18.180726 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:18.180726    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1777a955-409e-46e8-a83d-a635743030c7" containerName="maas-api"
Jul 29 06:55:18.180878 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:18.180834    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1777a955-409e-46e8-a83d-a635743030c7" containerName="maas-api"
Jul 29 06:55:18.185972 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:18.185948    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb"
Jul 29 06:55:18.188035 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:18.188014    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-88aa9015-serving-cert\""
Jul 29 06:55:18.192635 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:18.192577    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb"]
Jul 29 06:55:18.267690 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:18.267655    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6mt6\" (UniqueName: \"kubernetes.io/projected/22b74a68-acc6-4df3-a06f-63bda4d071c1-kube-api-access-z6mt6\") pod \"maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb\" (UID: \"22b74a68-acc6-4df3-a06f-63bda4d071c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb"
Jul 29 06:55:18.267905 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:18.267734    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22b74a68-acc6-4df3-a06f-63bda4d071c1-maas-api-tls\") pod \"maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb\" (UID: \"22b74a68-acc6-4df3-a06f-63bda4d071c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb"
Jul 29 06:55:18.267905 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:18.267755    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22b74a68-acc6-4df3-a06f-63bda4d071c1-openshift-service-ca\") pod \"maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb\" (UID: \"22b74a68-acc6-4df3-a06f-63bda4d071c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb"
Jul 29 06:55:18.368915 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:18.368837    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6mt6\" (UniqueName: \"kubernetes.io/projected/22b74a68-acc6-4df3-a06f-63bda4d071c1-kube-api-access-z6mt6\") pod \"maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb\" (UID: \"22b74a68-acc6-4df3-a06f-63bda4d071c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb"
Jul 29 06:55:18.369126 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:18.368968    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22b74a68-acc6-4df3-a06f-63bda4d071c1-maas-api-tls\") pod \"maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb\" (UID: \"22b74a68-acc6-4df3-a06f-63bda4d071c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb"
Jul 29 06:55:18.369126 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:18.368996    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22b74a68-acc6-4df3-a06f-63bda4d071c1-openshift-service-ca\") pod \"maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb\" (UID: \"22b74a68-acc6-4df3-a06f-63bda4d071c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb"
Jul 29 06:55:18.369126 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:55:18.369110    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-88aa9015-serving-cert: secret "maas-api-e2e-ait-88aa9015-serving-cert" not found
Jul 29 06:55:18.369272 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:55:18.369177    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22b74a68-acc6-4df3-a06f-63bda4d071c1-maas-api-tls podName:22b74a68-acc6-4df3-a06f-63bda4d071c1 nodeName:}" failed. No retries permitted until 2026-07-29 06:55:18.869157116 +0000 UTC m=+1961.946788880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/22b74a68-acc6-4df3-a06f-63bda4d071c1-maas-api-tls") pod "maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb" (UID: "22b74a68-acc6-4df3-a06f-63bda4d071c1") : secret "maas-api-e2e-ait-88aa9015-serving-cert" not found
Jul 29 06:55:18.369795 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:18.369773    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22b74a68-acc6-4df3-a06f-63bda4d071c1-openshift-service-ca\") pod \"maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb\" (UID: \"22b74a68-acc6-4df3-a06f-63bda4d071c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb"
Jul 29 06:55:18.387010 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:18.386985    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6mt6\" (UniqueName: \"kubernetes.io/projected/22b74a68-acc6-4df3-a06f-63bda4d071c1-kube-api-access-z6mt6\") pod \"maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb\" (UID: \"22b74a68-acc6-4df3-a06f-63bda4d071c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb"
Jul 29 06:55:18.873489 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:18.873451    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22b74a68-acc6-4df3-a06f-63bda4d071c1-maas-api-tls\") pod \"maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb\" (UID: \"22b74a68-acc6-4df3-a06f-63bda4d071c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb"
Jul 29 06:55:18.876000 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:18.875982    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22b74a68-acc6-4df3-a06f-63bda4d071c1-maas-api-tls\") pod \"maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb\" (UID: \"22b74a68-acc6-4df3-a06f-63bda4d071c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb"
Jul 29 06:55:19.100381 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:19.100344    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb"
Jul 29 06:55:19.239822 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:19.239795    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb"]
Jul 29 06:55:19.242418 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:55:19.242396    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22b74a68_acc6_4df3_a06f_63bda4d071c1.slice/crio-eef63802aafe6946e482a568abe6cbbfe46f15f4ca6e141469f2f792362014a2 WatchSource:0}: Error finding container eef63802aafe6946e482a568abe6cbbfe46f15f4ca6e141469f2f792362014a2: Status 404 returned error can't find the container with id eef63802aafe6946e482a568abe6cbbfe46f15f4ca6e141469f2f792362014a2
Jul 29 06:55:19.297211 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:19.297182    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb" event={"ID":"22b74a68-acc6-4df3-a06f-63bda4d071c1","Type":"ContainerStarted","Data":"eef63802aafe6946e482a568abe6cbbfe46f15f4ca6e141469f2f792362014a2"}
Jul 29 06:55:20.304357 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:20.304322    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb" event={"ID":"22b74a68-acc6-4df3-a06f-63bda4d071c1","Type":"ContainerStarted","Data":"4ff0d1d0511e75f53f7cb4af1587d676a7e5216ed0c91295dfa97b1b93c45b21"}
Jul 29 06:55:20.304798 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:20.304419    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb"
Jul 29 06:55:20.325247 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:20.325195    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb" podStartSLOduration=1.916035765 podStartE2EDuration="2.325177847s" totalImagesPullingTime="409.142082ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:55:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:55:20.32265579 +0000 UTC m=+1963.400287557" watchObservedRunningTime="2026-07-29 06:55:20.325177847 +0000 UTC m=+1963.402809614"
Jul 29 06:55:26.314185 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:26.314159    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb"
Jul 29 06:55:43.933267 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:43.933237    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88aa9015-r5lj9"]
Jul 29 06:55:43.936998 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:43.936981    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88aa9015-r5lj9"
Jul 29 06:55:43.939109 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:43.939090    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bj47g\""
Jul 29 06:55:43.944240 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:43.944218    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88aa9015-r5lj9"]
Jul 29 06:55:44.107193 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:44.107156    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0eade35-5235-4a20-ab98-25cb8437a755-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-88aa9015-r5lj9\" (UID: \"a0eade35-5235-4a20-ab98-25cb8437a755\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88aa9015-r5lj9"
Jul 29 06:55:44.208341 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:44.208262    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0eade35-5235-4a20-ab98-25cb8437a755-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-88aa9015-r5lj9\" (UID: \"a0eade35-5235-4a20-ab98-25cb8437a755\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88aa9015-r5lj9"
Jul 29 06:55:44.208922 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:44.208893    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0eade35-5235-4a20-ab98-25cb8437a755-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-88aa9015-r5lj9\" (UID: \"a0eade35-5235-4a20-ab98-25cb8437a755\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88aa9015-r5lj9"
Jul 29 06:55:44.252390 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:44.252369    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88aa9015-r5lj9"
Jul 29 06:55:44.393879 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:44.393838    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88aa9015-r5lj9"]
Jul 29 06:55:44.396723 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:55:44.396691    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0eade35_5235_4a20_ab98_25cb8437a755.slice/crio-bae319d12bbee632891448687ffdff5266a382fda6c46c6b7cfe9b0c324a934a WatchSource:0}: Error finding container bae319d12bbee632891448687ffdff5266a382fda6c46c6b7cfe9b0c324a934a: Status 404 returned error can't find the container with id bae319d12bbee632891448687ffdff5266a382fda6c46c6b7cfe9b0c324a934a
Jul 29 06:55:44.423568 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:44.423547    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88aa9015-r5lj9" event={"ID":"a0eade35-5235-4a20-ab98-25cb8437a755","Type":"ContainerStarted","Data":"bae319d12bbee632891448687ffdff5266a382fda6c46c6b7cfe9b0c324a934a"}
Jul 29 06:55:45.431129 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:45.431046    2563 generic.go:356] "Generic (PLEG): container finished" podID="a0eade35-5235-4a20-ab98-25cb8437a755" containerID="bee36473bc0aa308c391eaef9db9f0527143b5d557d7c104073ecb7237dea397" exitCode=0
Jul 29 06:55:45.431129 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:45.431116    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88aa9015-r5lj9" event={"ID":"a0eade35-5235-4a20-ab98-25cb8437a755","Type":"ContainerDied","Data":"bee36473bc0aa308c391eaef9db9f0527143b5d557d7c104073ecb7237dea397"}
Jul 29 06:55:46.573302 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:46.573280    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88aa9015-r5lj9"
Jul 29 06:55:46.733071 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:46.732991    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0eade35-5235-4a20-ab98-25cb8437a755-maas-api-service-ca\") pod \"a0eade35-5235-4a20-ab98-25cb8437a755\" (UID: \"a0eade35-5235-4a20-ab98-25cb8437a755\") "
Jul 29 06:55:46.733333 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:46.733313    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0eade35-5235-4a20-ab98-25cb8437a755-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a0eade35-5235-4a20-ab98-25cb8437a755" (UID: "a0eade35-5235-4a20-ab98-25cb8437a755"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 06:55:46.834411 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:46.834379    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0eade35-5235-4a20-ab98-25cb8437a755-maas-api-service-ca\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:55:47.444404 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:47.444375    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88aa9015-r5lj9"
Jul 29 06:55:47.444404 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:47.444395    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88aa9015-r5lj9" event={"ID":"a0eade35-5235-4a20-ab98-25cb8437a755","Type":"ContainerDied","Data":"bae319d12bbee632891448687ffdff5266a382fda6c46c6b7cfe9b0c324a934a"}
Jul 29 06:55:47.444606 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:47.444439    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bae319d12bbee632891448687ffdff5266a382fda6c46c6b7cfe9b0c324a934a"
Jul 29 06:55:48.721531 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:48.721496    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb"]
Jul 29 06:55:48.721959 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:48.721710    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb" podUID="22b74a68-acc6-4df3-a06f-63bda4d071c1" containerName="maas-api" containerID="cri-o://4ff0d1d0511e75f53f7cb4af1587d676a7e5216ed0c91295dfa97b1b93c45b21" gracePeriod=30
Jul 29 06:55:48.914573 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:48.914544    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88aa9015-r5lj9"]
Jul 29 06:55:48.919046 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:48.919019    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88aa9015-r5lj9"]
Jul 29 06:55:48.978084 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:48.978019    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb"
Jul 29 06:55:49.155417 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:49.155388    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22b74a68-acc6-4df3-a06f-63bda4d071c1-maas-api-tls\") pod \"22b74a68-acc6-4df3-a06f-63bda4d071c1\" (UID: \"22b74a68-acc6-4df3-a06f-63bda4d071c1\") "
Jul 29 06:55:49.155654 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:49.155547    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6mt6\" (UniqueName: \"kubernetes.io/projected/22b74a68-acc6-4df3-a06f-63bda4d071c1-kube-api-access-z6mt6\") pod \"22b74a68-acc6-4df3-a06f-63bda4d071c1\" (UID: \"22b74a68-acc6-4df3-a06f-63bda4d071c1\") "
Jul 29 06:55:49.155654 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:49.155581    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22b74a68-acc6-4df3-a06f-63bda4d071c1-openshift-service-ca\") pod \"22b74a68-acc6-4df3-a06f-63bda4d071c1\" (UID: \"22b74a68-acc6-4df3-a06f-63bda4d071c1\") "
Jul 29 06:55:49.156019 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:49.155983    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22b74a68-acc6-4df3-a06f-63bda4d071c1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "22b74a68-acc6-4df3-a06f-63bda4d071c1" (UID: "22b74a68-acc6-4df3-a06f-63bda4d071c1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 06:55:49.157746 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:49.157716    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22b74a68-acc6-4df3-a06f-63bda4d071c1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "22b74a68-acc6-4df3-a06f-63bda4d071c1" (UID: "22b74a68-acc6-4df3-a06f-63bda4d071c1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 06:55:49.157924 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:49.157798    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22b74a68-acc6-4df3-a06f-63bda4d071c1-kube-api-access-z6mt6" (OuterVolumeSpecName: "kube-api-access-z6mt6") pod "22b74a68-acc6-4df3-a06f-63bda4d071c1" (UID: "22b74a68-acc6-4df3-a06f-63bda4d071c1"). InnerVolumeSpecName "kube-api-access-z6mt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 06:55:49.256596 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:49.256539    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6mt6\" (UniqueName: \"kubernetes.io/projected/22b74a68-acc6-4df3-a06f-63bda4d071c1-kube-api-access-z6mt6\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:55:49.256596 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:49.256562    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22b74a68-acc6-4df3-a06f-63bda4d071c1-openshift-service-ca\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:55:49.256596 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:49.256571    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22b74a68-acc6-4df3-a06f-63bda4d071c1-maas-api-tls\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:55:49.398492 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:49.398455    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0eade35-5235-4a20-ab98-25cb8437a755" path="/var/lib/kubelet/pods/a0eade35-5235-4a20-ab98-25cb8437a755/volumes"
Jul 29 06:55:49.456730 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:49.456701    2563 generic.go:356] "Generic (PLEG): container finished" podID="22b74a68-acc6-4df3-a06f-63bda4d071c1" containerID="4ff0d1d0511e75f53f7cb4af1587d676a7e5216ed0c91295dfa97b1b93c45b21" exitCode=0
Jul 29 06:55:49.456873 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:49.456747    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb"
Jul 29 06:55:49.456873 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:49.456789    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb" event={"ID":"22b74a68-acc6-4df3-a06f-63bda4d071c1","Type":"ContainerDied","Data":"4ff0d1d0511e75f53f7cb4af1587d676a7e5216ed0c91295dfa97b1b93c45b21"}
Jul 29 06:55:49.456873 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:49.456844    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb" event={"ID":"22b74a68-acc6-4df3-a06f-63bda4d071c1","Type":"ContainerDied","Data":"eef63802aafe6946e482a568abe6cbbfe46f15f4ca6e141469f2f792362014a2"}
Jul 29 06:55:49.456989 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:49.456875    2563 scope.go:117] "RemoveContainer" containerID="4ff0d1d0511e75f53f7cb4af1587d676a7e5216ed0c91295dfa97b1b93c45b21"
Jul 29 06:55:49.465575 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:49.465558    2563 scope.go:117] "RemoveContainer" containerID="4ff0d1d0511e75f53f7cb4af1587d676a7e5216ed0c91295dfa97b1b93c45b21"
Jul 29 06:55:49.465831 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:55:49.465810    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ff0d1d0511e75f53f7cb4af1587d676a7e5216ed0c91295dfa97b1b93c45b21\": container with ID starting with 4ff0d1d0511e75f53f7cb4af1587d676a7e5216ed0c91295dfa97b1b93c45b21 not found: ID does not exist" containerID="4ff0d1d0511e75f53f7cb4af1587d676a7e5216ed0c91295dfa97b1b93c45b21"
Jul 29 06:55:49.465901 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:49.465840    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ff0d1d0511e75f53f7cb4af1587d676a7e5216ed0c91295dfa97b1b93c45b21"} err="failed to get container status \"4ff0d1d0511e75f53f7cb4af1587d676a7e5216ed0c91295dfa97b1b93c45b21\": rpc error: code = NotFound desc = could not find container \"4ff0d1d0511e75f53f7cb4af1587d676a7e5216ed0c91295dfa97b1b93c45b21\": container with ID starting with 4ff0d1d0511e75f53f7cb4af1587d676a7e5216ed0c91295dfa97b1b93c45b21 not found: ID does not exist"
Jul 29 06:55:49.471464 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:49.471441    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb"]
Jul 29 06:55:49.476708 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:49.476680    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-88aa9015-77cdcfd8fc-7kddb"]
Jul 29 06:55:51.397002 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:51.396971    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22b74a68-acc6-4df3-a06f-63bda4d071c1" path="/var/lib/kubelet/pods/22b74a68-acc6-4df3-a06f-63bda4d071c1/volumes"
Jul 29 06:55:53.427451 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:53.427409    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm"]
Jul 29 06:55:53.427840 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:53.427812    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22b74a68-acc6-4df3-a06f-63bda4d071c1" containerName="maas-api"
Jul 29 06:55:53.427840 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:53.427824    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="22b74a68-acc6-4df3-a06f-63bda4d071c1" containerName="maas-api"
Jul 29 06:55:53.427840 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:53.427832    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0eade35-5235-4a20-ab98-25cb8437a755" containerName="revoke-keys"
Jul 29 06:55:53.427840 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:53.427838    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0eade35-5235-4a20-ab98-25cb8437a755" containerName="revoke-keys"
Jul 29 06:55:53.428023 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:53.427927    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0eade35-5235-4a20-ab98-25cb8437a755" containerName="revoke-keys"
Jul 29 06:55:53.428023 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:53.427936    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="22b74a68-acc6-4df3-a06f-63bda4d071c1" containerName="maas-api"
Jul 29 06:55:53.432405 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:53.432381    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm"
Jul 29 06:55:53.434386 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:53.434367    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-0f99172f-serving-cert\""
Jul 29 06:55:53.442424 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:53.442400    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm"]
Jul 29 06:55:53.497348 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:53.497312    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10d626ec-dba3-4a7e-ad3b-77b0caca98ff-maas-api-tls\") pod \"maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm\" (UID: \"10d626ec-dba3-4a7e-ad3b-77b0caca98ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm"
Jul 29 06:55:53.497524 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:53.497431    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5rsh\" (UniqueName: \"kubernetes.io/projected/10d626ec-dba3-4a7e-ad3b-77b0caca98ff-kube-api-access-p5rsh\") pod \"maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm\" (UID: \"10d626ec-dba3-4a7e-ad3b-77b0caca98ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm"
Jul 29 06:55:53.497524 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:53.497455    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10d626ec-dba3-4a7e-ad3b-77b0caca98ff-openshift-service-ca\") pod \"maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm\" (UID: \"10d626ec-dba3-4a7e-ad3b-77b0caca98ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm"
Jul 29 06:55:53.598089 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:53.598050    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10d626ec-dba3-4a7e-ad3b-77b0caca98ff-maas-api-tls\") pod \"maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm\" (UID: \"10d626ec-dba3-4a7e-ad3b-77b0caca98ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm"
Jul 29 06:55:53.598261 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:53.598145    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5rsh\" (UniqueName: \"kubernetes.io/projected/10d626ec-dba3-4a7e-ad3b-77b0caca98ff-kube-api-access-p5rsh\") pod \"maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm\" (UID: \"10d626ec-dba3-4a7e-ad3b-77b0caca98ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm"
Jul 29 06:55:53.598261 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:53.598165    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10d626ec-dba3-4a7e-ad3b-77b0caca98ff-openshift-service-ca\") pod \"maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm\" (UID: \"10d626ec-dba3-4a7e-ad3b-77b0caca98ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm"
Jul 29 06:55:53.598774 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:53.598757    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10d626ec-dba3-4a7e-ad3b-77b0caca98ff-openshift-service-ca\") pod \"maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm\" (UID: \"10d626ec-dba3-4a7e-ad3b-77b0caca98ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm"
Jul 29 06:55:53.600807 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:53.600789    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10d626ec-dba3-4a7e-ad3b-77b0caca98ff-maas-api-tls\") pod \"maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm\" (UID: \"10d626ec-dba3-4a7e-ad3b-77b0caca98ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm"
Jul 29 06:55:53.605202 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:53.605183    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5rsh\" (UniqueName: \"kubernetes.io/projected/10d626ec-dba3-4a7e-ad3b-77b0caca98ff-kube-api-access-p5rsh\") pod \"maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm\" (UID: \"10d626ec-dba3-4a7e-ad3b-77b0caca98ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm"
Jul 29 06:55:53.746231 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:53.746145    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm"
Jul 29 06:55:54.025073 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:54.024997    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0f99172f-685d677d8d-rg5mr"]
Jul 29 06:55:54.029678 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:54.029659    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0f99172f-685d677d8d-rg5mr"
Jul 29 06:55:54.031789 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:54.031761    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-0f99172f-dockercfg-776q2\""
Jul 29 06:55:54.031928 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:54.031799    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-0f99172f\""
Jul 29 06:55:54.038641 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:54.038617    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0f99172f-685d677d8d-rg5mr"]
Jul 29 06:55:54.088454 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:54.088428    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm"]
Jul 29 06:55:54.091324 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:55:54.091300    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10d626ec_dba3_4a7e_ad3b_77b0caca98ff.slice/crio-3a9ae8e43d90a598f0729f26c71a035aca0839b06cb005eb46f00f27102fdc1e WatchSource:0}: Error finding container 3a9ae8e43d90a598f0729f26c71a035aca0839b06cb005eb46f00f27102fdc1e: Status 404 returned error can't find the container with id 3a9ae8e43d90a598f0729f26c71a035aca0839b06cb005eb46f00f27102fdc1e
Jul 29 06:55:54.103523 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:54.103499    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8a3bb66-f1a7-469b-9151-59de1b247ecf-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-0f99172f-685d677d8d-rg5mr\" (UID: \"e8a3bb66-f1a7-469b-9151-59de1b247ecf\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0f99172f-685d677d8d-rg5mr"
Jul 29 06:55:54.103627 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:54.103554    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fdjp\" (UniqueName: \"kubernetes.io/projected/e8a3bb66-f1a7-469b-9151-59de1b247ecf-kube-api-access-4fdjp\") pod \"payload-pre-processing-e2e-derive-0f99172f-685d677d8d-rg5mr\" (UID: \"e8a3bb66-f1a7-469b-9151-59de1b247ecf\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0f99172f-685d677d8d-rg5mr"
Jul 29 06:55:54.204428 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:54.204391    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8a3bb66-f1a7-469b-9151-59de1b247ecf-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-0f99172f-685d677d8d-rg5mr\" (UID: \"e8a3bb66-f1a7-469b-9151-59de1b247ecf\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0f99172f-685d677d8d-rg5mr"
Jul 29 06:55:54.204602 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:54.204474    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fdjp\" (UniqueName: \"kubernetes.io/projected/e8a3bb66-f1a7-469b-9151-59de1b247ecf-kube-api-access-4fdjp\") pod \"payload-pre-processing-e2e-derive-0f99172f-685d677d8d-rg5mr\" (UID: \"e8a3bb66-f1a7-469b-9151-59de1b247ecf\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0f99172f-685d677d8d-rg5mr"
Jul 29 06:55:54.205091 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:54.205073    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8a3bb66-f1a7-469b-9151-59de1b247ecf-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-0f99172f-685d677d8d-rg5mr\" (UID: \"e8a3bb66-f1a7-469b-9151-59de1b247ecf\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0f99172f-685d677d8d-rg5mr"
Jul 29 06:55:54.211570 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:54.211549    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fdjp\" (UniqueName: \"kubernetes.io/projected/e8a3bb66-f1a7-469b-9151-59de1b247ecf-kube-api-access-4fdjp\") pod \"payload-pre-processing-e2e-derive-0f99172f-685d677d8d-rg5mr\" (UID: \"e8a3bb66-f1a7-469b-9151-59de1b247ecf\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0f99172f-685d677d8d-rg5mr"
Jul 29 06:55:54.342326 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:54.342293    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0f99172f-685d677d8d-rg5mr"
Jul 29 06:55:54.483185 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:54.483158    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0f99172f-685d677d8d-rg5mr"]
Jul 29 06:55:54.485077 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:55:54.485051    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a3bb66_f1a7_469b_9151_59de1b247ecf.slice/crio-f5db31363ba42bf84381b1e60c9babc90ff5a392d01b3ce636b882956b404878 WatchSource:0}: Error finding container f5db31363ba42bf84381b1e60c9babc90ff5a392d01b3ce636b882956b404878: Status 404 returned error can't find the container with id f5db31363ba42bf84381b1e60c9babc90ff5a392d01b3ce636b882956b404878
Jul 29 06:55:54.485879 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:54.485777    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm" event={"ID":"10d626ec-dba3-4a7e-ad3b-77b0caca98ff","Type":"ContainerStarted","Data":"3a9ae8e43d90a598f0729f26c71a035aca0839b06cb005eb46f00f27102fdc1e"}
Jul 29 06:55:55.492162 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:55.492125    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0f99172f-685d677d8d-rg5mr" event={"ID":"e8a3bb66-f1a7-469b-9151-59de1b247ecf","Type":"ContainerStarted","Data":"1238af3699302615d47eaa9a6c5e210018350c56128ff7b487ef64fbdad0e4aa"}
Jul 29 06:55:55.492610 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:55.492190    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-0f99172f-685d677d8d-rg5mr"
Jul 29 06:55:55.492610 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:55.492216    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0f99172f-685d677d8d-rg5mr" event={"ID":"e8a3bb66-f1a7-469b-9151-59de1b247ecf","Type":"ContainerStarted","Data":"f5db31363ba42bf84381b1e60c9babc90ff5a392d01b3ce636b882956b404878"}
Jul 29 06:55:55.493705 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:55.493676    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm" event={"ID":"10d626ec-dba3-4a7e-ad3b-77b0caca98ff","Type":"ContainerStarted","Data":"9c2e16f460be97f010ff2d6dafb65d659428372889a8b8a6ab0de4a8ef9c08be"}
Jul 29 06:55:55.493823 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:55.493792    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm"
Jul 29 06:55:55.506387 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:55.506346    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-0f99172f-685d677d8d-rg5mr" podStartSLOduration=1.506332832 podStartE2EDuration="1.506332832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:55:55.505650191 +0000 UTC m=+1998.583281958" watchObservedRunningTime="2026-07-29 06:55:55.506332832 +0000 UTC m=+1998.583964598"
Jul 29 06:55:55.520207 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:55:55.520169    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm" podStartSLOduration=2.120786458 podStartE2EDuration="2.520154758s" totalImagesPullingTime="399.3683ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:55:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:55:55.519349999 +0000 UTC m=+1998.596981767" watchObservedRunningTime="2026-07-29 06:55:55.520154758 +0000 UTC m=+1998.597786524"
Jul 29 06:56:01.504926 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:01.504892    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm"
Jul 29 06:56:06.500837 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:06.500808    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-0f99172f-685d677d8d-rg5mr"
Jul 29 06:56:18.168457 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:18.168419    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f99172f-2gqbx"]
Jul 29 06:56:18.172359 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:18.172337    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f99172f-2gqbx"
Jul 29 06:56:18.174399 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:18.174375    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bj47g\""
Jul 29 06:56:18.178331 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:18.178307    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f99172f-2gqbx"]
Jul 29 06:56:18.325559 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:18.325525    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b4455c-2a59-42d8-b43d-68028e37a5a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0f99172f-2gqbx\" (UID: \"51b4455c-2a59-42d8-b43d-68028e37a5a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f99172f-2gqbx"
Jul 29 06:56:18.426145 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:18.426059    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b4455c-2a59-42d8-b43d-68028e37a5a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0f99172f-2gqbx\" (UID: \"51b4455c-2a59-42d8-b43d-68028e37a5a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f99172f-2gqbx"
Jul 29 06:56:18.426602 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:18.426582    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b4455c-2a59-42d8-b43d-68028e37a5a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0f99172f-2gqbx\" (UID: \"51b4455c-2a59-42d8-b43d-68028e37a5a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f99172f-2gqbx"
Jul 29 06:56:18.486694 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:18.486664    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f99172f-2gqbx"
Jul 29 06:56:18.623643 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:18.623409    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f99172f-2gqbx"]
Jul 29 06:56:18.627093 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:56:18.626137    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b4455c_2a59_42d8_b43d_68028e37a5a1.slice/crio-abff97e781268160e287dd37474467baa7171526c5acda8d3e90f56731cc0f65 WatchSource:0}: Error finding container abff97e781268160e287dd37474467baa7171526c5acda8d3e90f56731cc0f65: Status 404 returned error can't find the container with id abff97e781268160e287dd37474467baa7171526c5acda8d3e90f56731cc0f65
Jul 29 06:56:19.620094 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:19.620063    2563 generic.go:356] "Generic (PLEG): container finished" podID="51b4455c-2a59-42d8-b43d-68028e37a5a1" containerID="dc2c050aa39e1713c847c039e15fb3e55724a2329aa37f5488724d3255b9b898" exitCode=0
Jul 29 06:56:19.620464 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:19.620148    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f99172f-2gqbx" event={"ID":"51b4455c-2a59-42d8-b43d-68028e37a5a1","Type":"ContainerDied","Data":"dc2c050aa39e1713c847c039e15fb3e55724a2329aa37f5488724d3255b9b898"}
Jul 29 06:56:19.620464 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:19.620198    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f99172f-2gqbx" event={"ID":"51b4455c-2a59-42d8-b43d-68028e37a5a1","Type":"ContainerStarted","Data":"abff97e781268160e287dd37474467baa7171526c5acda8d3e90f56731cc0f65"}
Jul 29 06:56:20.773370 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:20.773345    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f99172f-2gqbx"
Jul 29 06:56:20.949633 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:20.949545    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b4455c-2a59-42d8-b43d-68028e37a5a1-maas-api-service-ca\") pod \"51b4455c-2a59-42d8-b43d-68028e37a5a1\" (UID: \"51b4455c-2a59-42d8-b43d-68028e37a5a1\") "
Jul 29 06:56:20.949984 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:20.949958    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51b4455c-2a59-42d8-b43d-68028e37a5a1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "51b4455c-2a59-42d8-b43d-68028e37a5a1" (UID: "51b4455c-2a59-42d8-b43d-68028e37a5a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 06:56:21.050397 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:21.050364    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b4455c-2a59-42d8-b43d-68028e37a5a1-maas-api-service-ca\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:56:21.630930 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:21.630895    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f99172f-2gqbx" event={"ID":"51b4455c-2a59-42d8-b43d-68028e37a5a1","Type":"ContainerDied","Data":"abff97e781268160e287dd37474467baa7171526c5acda8d3e90f56731cc0f65"}
Jul 29 06:56:21.631117 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:21.630941    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abff97e781268160e287dd37474467baa7171526c5acda8d3e90f56731cc0f65"
Jul 29 06:56:21.631117 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:21.630986    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f99172f-2gqbx"
Jul 29 06:56:23.951972 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:23.951935    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm"]
Jul 29 06:56:23.952386 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:23.952191    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm" podUID="10d626ec-dba3-4a7e-ad3b-77b0caca98ff" containerName="maas-api" containerID="cri-o://9c2e16f460be97f010ff2d6dafb65d659428372889a8b8a6ab0de4a8ef9c08be" gracePeriod=30
Jul 29 06:56:24.016687 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.016651    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0f99172f-685d677d8d-rg5mr"]
Jul 29 06:56:24.017293 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.017041    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-0f99172f-685d677d8d-rg5mr" podUID="e8a3bb66-f1a7-469b-9151-59de1b247ecf" containerName="payload-pre-processing" containerID="cri-o://1238af3699302615d47eaa9a6c5e210018350c56128ff7b487ef64fbdad0e4aa" gracePeriod=30
Jul 29 06:56:24.139436 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.139371    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f99172f-2gqbx"]
Jul 29 06:56:24.141678 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.141655    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0f99172f-2gqbx"]
Jul 29 06:56:24.296388 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.296117    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm"
Jul 29 06:56:24.301206 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.301183    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0f99172f-685d677d8d-rg5mr"
Jul 29 06:56:24.481087 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.481002    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5rsh\" (UniqueName: \"kubernetes.io/projected/10d626ec-dba3-4a7e-ad3b-77b0caca98ff-kube-api-access-p5rsh\") pod \"10d626ec-dba3-4a7e-ad3b-77b0caca98ff\" (UID: \"10d626ec-dba3-4a7e-ad3b-77b0caca98ff\") "
Jul 29 06:56:24.481087 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.481067    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8a3bb66-f1a7-469b-9151-59de1b247ecf-plugins-config-volume\") pod \"e8a3bb66-f1a7-469b-9151-59de1b247ecf\" (UID: \"e8a3bb66-f1a7-469b-9151-59de1b247ecf\") "
Jul 29 06:56:24.481087 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.481089    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fdjp\" (UniqueName: \"kubernetes.io/projected/e8a3bb66-f1a7-469b-9151-59de1b247ecf-kube-api-access-4fdjp\") pod \"e8a3bb66-f1a7-469b-9151-59de1b247ecf\" (UID: \"e8a3bb66-f1a7-469b-9151-59de1b247ecf\") "
Jul 29 06:56:24.481381 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.481135    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10d626ec-dba3-4a7e-ad3b-77b0caca98ff-maas-api-tls\") pod \"10d626ec-dba3-4a7e-ad3b-77b0caca98ff\" (UID: \"10d626ec-dba3-4a7e-ad3b-77b0caca98ff\") "
Jul 29 06:56:24.481381 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.481158    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10d626ec-dba3-4a7e-ad3b-77b0caca98ff-openshift-service-ca\") pod \"10d626ec-dba3-4a7e-ad3b-77b0caca98ff\" (UID: \"10d626ec-dba3-4a7e-ad3b-77b0caca98ff\") "
Jul 29 06:56:24.481568 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.481525    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a3bb66-f1a7-469b-9151-59de1b247ecf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e8a3bb66-f1a7-469b-9151-59de1b247ecf" (UID: "e8a3bb66-f1a7-469b-9151-59de1b247ecf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 06:56:24.481568 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.481553    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10d626ec-dba3-4a7e-ad3b-77b0caca98ff-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "10d626ec-dba3-4a7e-ad3b-77b0caca98ff" (UID: "10d626ec-dba3-4a7e-ad3b-77b0caca98ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 06:56:24.483358 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.483326    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10d626ec-dba3-4a7e-ad3b-77b0caca98ff-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "10d626ec-dba3-4a7e-ad3b-77b0caca98ff" (UID: "10d626ec-dba3-4a7e-ad3b-77b0caca98ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 06:56:24.483446 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.483363    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10d626ec-dba3-4a7e-ad3b-77b0caca98ff-kube-api-access-p5rsh" (OuterVolumeSpecName: "kube-api-access-p5rsh") pod "10d626ec-dba3-4a7e-ad3b-77b0caca98ff" (UID: "10d626ec-dba3-4a7e-ad3b-77b0caca98ff"). InnerVolumeSpecName "kube-api-access-p5rsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 06:56:24.483446 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.483373    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a3bb66-f1a7-469b-9151-59de1b247ecf-kube-api-access-4fdjp" (OuterVolumeSpecName: "kube-api-access-4fdjp") pod "e8a3bb66-f1a7-469b-9151-59de1b247ecf" (UID: "e8a3bb66-f1a7-469b-9151-59de1b247ecf"). InnerVolumeSpecName "kube-api-access-4fdjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 06:56:24.582693 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.582666    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5rsh\" (UniqueName: \"kubernetes.io/projected/10d626ec-dba3-4a7e-ad3b-77b0caca98ff-kube-api-access-p5rsh\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:56:24.582693 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.582690    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8a3bb66-f1a7-469b-9151-59de1b247ecf-plugins-config-volume\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:56:24.582890 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.582700    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fdjp\" (UniqueName: \"kubernetes.io/projected/e8a3bb66-f1a7-469b-9151-59de1b247ecf-kube-api-access-4fdjp\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:56:24.582890 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.582711    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10d626ec-dba3-4a7e-ad3b-77b0caca98ff-maas-api-tls\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:56:24.582890 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.582720    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10d626ec-dba3-4a7e-ad3b-77b0caca98ff-openshift-service-ca\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:56:24.649255 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.649223    2563 generic.go:356] "Generic (PLEG): container finished" podID="10d626ec-dba3-4a7e-ad3b-77b0caca98ff" containerID="9c2e16f460be97f010ff2d6dafb65d659428372889a8b8a6ab0de4a8ef9c08be" exitCode=0
Jul 29 06:56:24.649411 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.649274    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm"
Jul 29 06:56:24.649411 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.649315    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm" event={"ID":"10d626ec-dba3-4a7e-ad3b-77b0caca98ff","Type":"ContainerDied","Data":"9c2e16f460be97f010ff2d6dafb65d659428372889a8b8a6ab0de4a8ef9c08be"}
Jul 29 06:56:24.649411 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.649370    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm" event={"ID":"10d626ec-dba3-4a7e-ad3b-77b0caca98ff","Type":"ContainerDied","Data":"3a9ae8e43d90a598f0729f26c71a035aca0839b06cb005eb46f00f27102fdc1e"}
Jul 29 06:56:24.649411 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.649398    2563 scope.go:117] "RemoveContainer" containerID="9c2e16f460be97f010ff2d6dafb65d659428372889a8b8a6ab0de4a8ef9c08be"
Jul 29 06:56:24.650707 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.650685    2563 generic.go:356] "Generic (PLEG): container finished" podID="e8a3bb66-f1a7-469b-9151-59de1b247ecf" containerID="1238af3699302615d47eaa9a6c5e210018350c56128ff7b487ef64fbdad0e4aa" exitCode=0
Jul 29 06:56:24.650827 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.650727    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0f99172f-685d677d8d-rg5mr" event={"ID":"e8a3bb66-f1a7-469b-9151-59de1b247ecf","Type":"ContainerDied","Data":"1238af3699302615d47eaa9a6c5e210018350c56128ff7b487ef64fbdad0e4aa"}
Jul 29 06:56:24.650827 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.650729    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0f99172f-685d677d8d-rg5mr"
Jul 29 06:56:24.650827 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.650757    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0f99172f-685d677d8d-rg5mr" event={"ID":"e8a3bb66-f1a7-469b-9151-59de1b247ecf","Type":"ContainerDied","Data":"f5db31363ba42bf84381b1e60c9babc90ff5a392d01b3ce636b882956b404878"}
Jul 29 06:56:24.658598 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.658576    2563 scope.go:117] "RemoveContainer" containerID="9c2e16f460be97f010ff2d6dafb65d659428372889a8b8a6ab0de4a8ef9c08be"
Jul 29 06:56:24.658898 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:56:24.658873    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c2e16f460be97f010ff2d6dafb65d659428372889a8b8a6ab0de4a8ef9c08be\": container with ID starting with 9c2e16f460be97f010ff2d6dafb65d659428372889a8b8a6ab0de4a8ef9c08be not found: ID does not exist" containerID="9c2e16f460be97f010ff2d6dafb65d659428372889a8b8a6ab0de4a8ef9c08be"
Jul 29 06:56:24.658998 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.658908    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c2e16f460be97f010ff2d6dafb65d659428372889a8b8a6ab0de4a8ef9c08be"} err="failed to get container status \"9c2e16f460be97f010ff2d6dafb65d659428372889a8b8a6ab0de4a8ef9c08be\": rpc error: code = NotFound desc = could not find container \"9c2e16f460be97f010ff2d6dafb65d659428372889a8b8a6ab0de4a8ef9c08be\": container with ID starting with 9c2e16f460be97f010ff2d6dafb65d659428372889a8b8a6ab0de4a8ef9c08be not found: ID does not exist"
Jul 29 06:56:24.658998 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.658932    2563 scope.go:117] "RemoveContainer" containerID="1238af3699302615d47eaa9a6c5e210018350c56128ff7b487ef64fbdad0e4aa"
Jul 29 06:56:24.667516 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.667500    2563 scope.go:117] "RemoveContainer" containerID="1238af3699302615d47eaa9a6c5e210018350c56128ff7b487ef64fbdad0e4aa"
Jul 29 06:56:24.667801 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:56:24.667784    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1238af3699302615d47eaa9a6c5e210018350c56128ff7b487ef64fbdad0e4aa\": container with ID starting with 1238af3699302615d47eaa9a6c5e210018350c56128ff7b487ef64fbdad0e4aa not found: ID does not exist" containerID="1238af3699302615d47eaa9a6c5e210018350c56128ff7b487ef64fbdad0e4aa"
Jul 29 06:56:24.667878 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.667808    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1238af3699302615d47eaa9a6c5e210018350c56128ff7b487ef64fbdad0e4aa"} err="failed to get container status \"1238af3699302615d47eaa9a6c5e210018350c56128ff7b487ef64fbdad0e4aa\": rpc error: code = NotFound desc = could not find container \"1238af3699302615d47eaa9a6c5e210018350c56128ff7b487ef64fbdad0e4aa\": container with ID starting with 1238af3699302615d47eaa9a6c5e210018350c56128ff7b487ef64fbdad0e4aa not found: ID does not exist"
Jul 29 06:56:24.673334 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.673315    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0f99172f-685d677d8d-rg5mr"]
Jul 29 06:56:24.678493 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.678475    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0f99172f-685d677d8d-rg5mr"]
Jul 29 06:56:24.687681 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.687651    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm"]
Jul 29 06:56:24.691050 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:24.691025    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0f99172f-84df6dcfc7-52nbm"]
Jul 29 06:56:25.398137 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:25.398094    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10d626ec-dba3-4a7e-ad3b-77b0caca98ff" path="/var/lib/kubelet/pods/10d626ec-dba3-4a7e-ad3b-77b0caca98ff/volumes"
Jul 29 06:56:25.398602 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:25.398568    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51b4455c-2a59-42d8-b43d-68028e37a5a1" path="/var/lib/kubelet/pods/51b4455c-2a59-42d8-b43d-68028e37a5a1/volumes"
Jul 29 06:56:25.398978 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:56:25.398960    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8a3bb66-f1a7-469b-9151-59de1b247ecf" path="/var/lib/kubelet/pods/e8a3bb66-f1a7-469b-9151-59de1b247ecf/volumes"
Jul 29 06:58:16.616013 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:16.615971    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh"]
Jul 29 06:58:16.617216 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:16.617190    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10d626ec-dba3-4a7e-ad3b-77b0caca98ff" containerName="maas-api"
Jul 29 06:58:16.617369 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:16.617358    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="10d626ec-dba3-4a7e-ad3b-77b0caca98ff" containerName="maas-api"
Jul 29 06:58:16.617454 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:16.617445    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8a3bb66-f1a7-469b-9151-59de1b247ecf" containerName="payload-pre-processing"
Jul 29 06:58:16.617532 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:16.617524    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8a3bb66-f1a7-469b-9151-59de1b247ecf" containerName="payload-pre-processing"
Jul 29 06:58:16.617623 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:16.617614    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51b4455c-2a59-42d8-b43d-68028e37a5a1" containerName="revoke-keys"
Jul 29 06:58:16.617695 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:16.617687    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="51b4455c-2a59-42d8-b43d-68028e37a5a1" containerName="revoke-keys"
Jul 29 06:58:16.617898 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:16.617885    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="51b4455c-2a59-42d8-b43d-68028e37a5a1" containerName="revoke-keys"
Jul 29 06:58:16.618004 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:16.617995    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="10d626ec-dba3-4a7e-ad3b-77b0caca98ff" containerName="maas-api"
Jul 29 06:58:16.618084 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:16.618074    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8a3bb66-f1a7-469b-9151-59de1b247ecf" containerName="payload-pre-processing"
Jul 29 06:58:16.626715 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:16.626692    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh"]
Jul 29 06:58:16.626912 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:16.626801    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh"
Jul 29 06:58:16.629386 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:16.629363    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-bc10550f-serving-cert\""
Jul 29 06:58:16.686326 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:16.686294    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12bbbe7-07d2-4b7b-969b-da0d30d092a5-maas-api-tls\") pod \"maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh\" (UID: \"e12bbbe7-07d2-4b7b-969b-da0d30d092a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh"
Jul 29 06:58:16.686477 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:16.686346    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6bpz\" (UniqueName: \"kubernetes.io/projected/e12bbbe7-07d2-4b7b-969b-da0d30d092a5-kube-api-access-r6bpz\") pod \"maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh\" (UID: \"e12bbbe7-07d2-4b7b-969b-da0d30d092a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh"
Jul 29 06:58:16.686477 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:16.686368    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12bbbe7-07d2-4b7b-969b-da0d30d092a5-openshift-service-ca\") pod \"maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh\" (UID: \"e12bbbe7-07d2-4b7b-969b-da0d30d092a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh"
Jul 29 06:58:16.787787 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:16.787754    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12bbbe7-07d2-4b7b-969b-da0d30d092a5-maas-api-tls\") pod \"maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh\" (UID: \"e12bbbe7-07d2-4b7b-969b-da0d30d092a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh"
Jul 29 06:58:16.788008 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:16.787808    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6bpz\" (UniqueName: \"kubernetes.io/projected/e12bbbe7-07d2-4b7b-969b-da0d30d092a5-kube-api-access-r6bpz\") pod \"maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh\" (UID: \"e12bbbe7-07d2-4b7b-969b-da0d30d092a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh"
Jul 29 06:58:16.788008 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:58:16.787933    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-bc10550f-serving-cert: secret "maas-api-e2e-mt-bc10550f-serving-cert" not found
Jul 29 06:58:16.788008 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:16.787980    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12bbbe7-07d2-4b7b-969b-da0d30d092a5-openshift-service-ca\") pod \"maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh\" (UID: \"e12bbbe7-07d2-4b7b-969b-da0d30d092a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh"
Jul 29 06:58:16.788200 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:58:16.788012    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e12bbbe7-07d2-4b7b-969b-da0d30d092a5-maas-api-tls podName:e12bbbe7-07d2-4b7b-969b-da0d30d092a5 nodeName:}" failed. No retries permitted until 2026-07-29 06:58:17.287991372 +0000 UTC m=+2140.365623117 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e12bbbe7-07d2-4b7b-969b-da0d30d092a5-maas-api-tls") pod "maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh" (UID: "e12bbbe7-07d2-4b7b-969b-da0d30d092a5") : secret "maas-api-e2e-mt-bc10550f-serving-cert" not found
Jul 29 06:58:16.788570 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:16.788552    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12bbbe7-07d2-4b7b-969b-da0d30d092a5-openshift-service-ca\") pod \"maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh\" (UID: \"e12bbbe7-07d2-4b7b-969b-da0d30d092a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh"
Jul 29 06:58:16.796080 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:16.796055    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6bpz\" (UniqueName: \"kubernetes.io/projected/e12bbbe7-07d2-4b7b-969b-da0d30d092a5-kube-api-access-r6bpz\") pod \"maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh\" (UID: \"e12bbbe7-07d2-4b7b-969b-da0d30d092a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh"
Jul 29 06:58:17.293645 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:17.293604    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12bbbe7-07d2-4b7b-969b-da0d30d092a5-maas-api-tls\") pod \"maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh\" (UID: \"e12bbbe7-07d2-4b7b-969b-da0d30d092a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh"
Jul 29 06:58:17.296257 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:17.296227    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12bbbe7-07d2-4b7b-969b-da0d30d092a5-maas-api-tls\") pod \"maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh\" (UID: \"e12bbbe7-07d2-4b7b-969b-da0d30d092a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh"
Jul 29 06:58:17.383766 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:17.383726    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bc10550f-7b5d6689cb-gm89x"]
Jul 29 06:58:17.388812 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:17.388108    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc10550f-7b5d6689cb-gm89x"
Jul 29 06:58:17.390179 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:17.390156    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-bc10550f-dockercfg-8hzbn\""
Jul 29 06:58:17.390420 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:17.390404    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-bc10550f\""
Jul 29 06:58:17.401079 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:17.401058    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bc10550f-7b5d6689cb-gm89x"]
Jul 29 06:58:17.496534 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:17.496491    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twmcd\" (UniqueName: \"kubernetes.io/projected/2266f00e-1ba5-455a-b3f0-daccb2f25569-kube-api-access-twmcd\") pod \"payload-pre-processing-e2e-mt-bc10550f-7b5d6689cb-gm89x\" (UID: \"2266f00e-1ba5-455a-b3f0-daccb2f25569\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bc10550f-7b5d6689cb-gm89x"
Jul 29 06:58:17.496716 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:17.496577    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2266f00e-1ba5-455a-b3f0-daccb2f25569-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bc10550f-7b5d6689cb-gm89x\" (UID: \"2266f00e-1ba5-455a-b3f0-daccb2f25569\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bc10550f-7b5d6689cb-gm89x"
Jul 29 06:58:17.539027 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:17.538990    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh"
Jul 29 06:58:17.597315 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:17.597271    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twmcd\" (UniqueName: \"kubernetes.io/projected/2266f00e-1ba5-455a-b3f0-daccb2f25569-kube-api-access-twmcd\") pod \"payload-pre-processing-e2e-mt-bc10550f-7b5d6689cb-gm89x\" (UID: \"2266f00e-1ba5-455a-b3f0-daccb2f25569\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bc10550f-7b5d6689cb-gm89x"
Jul 29 06:58:17.597506 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:17.597350    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2266f00e-1ba5-455a-b3f0-daccb2f25569-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bc10550f-7b5d6689cb-gm89x\" (UID: \"2266f00e-1ba5-455a-b3f0-daccb2f25569\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bc10550f-7b5d6689cb-gm89x"
Jul 29 06:58:17.598268 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:17.598222    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2266f00e-1ba5-455a-b3f0-daccb2f25569-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bc10550f-7b5d6689cb-gm89x\" (UID: \"2266f00e-1ba5-455a-b3f0-daccb2f25569\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bc10550f-7b5d6689cb-gm89x"
Jul 29 06:58:17.611943 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:17.611877    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twmcd\" (UniqueName: \"kubernetes.io/projected/2266f00e-1ba5-455a-b3f0-daccb2f25569-kube-api-access-twmcd\") pod \"payload-pre-processing-e2e-mt-bc10550f-7b5d6689cb-gm89x\" (UID: \"2266f00e-1ba5-455a-b3f0-daccb2f25569\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bc10550f-7b5d6689cb-gm89x"
Jul 29 06:58:17.707494 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:17.707445    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc10550f-7b5d6689cb-gm89x"
Jul 29 06:58:17.855199 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:17.855156    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bc10550f-7b5d6689cb-gm89x"]
Jul 29 06:58:17.858210 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:58:17.858178    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2266f00e_1ba5_455a_b3f0_daccb2f25569.slice/crio-ede0fd79d5e851a9be312a867cab08c261e0ee14023295986c4a788c22de1cd0 WatchSource:0}: Error finding container ede0fd79d5e851a9be312a867cab08c261e0ee14023295986c4a788c22de1cd0: Status 404 returned error can't find the container with id ede0fd79d5e851a9be312a867cab08c261e0ee14023295986c4a788c22de1cd0
Jul 29 06:58:17.907431 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:17.907343    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh"]
Jul 29 06:58:17.910740 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:58:17.910706    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode12bbbe7_07d2_4b7b_969b_da0d30d092a5.slice/crio-20cabc6735b47e7ade206c055e0279711a47181cfe3895eb338e3a72be6b443f WatchSource:0}: Error finding container 20cabc6735b47e7ade206c055e0279711a47181cfe3895eb338e3a72be6b443f: Status 404 returned error can't find the container with id 20cabc6735b47e7ade206c055e0279711a47181cfe3895eb338e3a72be6b443f
Jul 29 06:58:18.232812 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:18.232719    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc10550f-7b5d6689cb-gm89x" event={"ID":"2266f00e-1ba5-455a-b3f0-daccb2f25569","Type":"ContainerStarted","Data":"fc7c7d8a2ea209ce44dea4d896e87088117d659ed6b557fbae1c5d4a6c2a475c"}
Jul 29 06:58:18.232812 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:18.232772    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc10550f-7b5d6689cb-gm89x" event={"ID":"2266f00e-1ba5-455a-b3f0-daccb2f25569","Type":"ContainerStarted","Data":"ede0fd79d5e851a9be312a867cab08c261e0ee14023295986c4a788c22de1cd0"}
Jul 29 06:58:18.233077 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:18.232834    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc10550f-7b5d6689cb-gm89x"
Jul 29 06:58:18.234294 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:18.234260    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh" event={"ID":"e12bbbe7-07d2-4b7b-969b-da0d30d092a5","Type":"ContainerStarted","Data":"20cabc6735b47e7ade206c055e0279711a47181cfe3895eb338e3a72be6b443f"}
Jul 29 06:58:18.248892 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:18.248821    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc10550f-7b5d6689cb-gm89x" podStartSLOduration=1.24880598 podStartE2EDuration="1.24880598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:58:18.247074821 +0000 UTC m=+2141.324706610" watchObservedRunningTime="2026-07-29 06:58:18.24880598 +0000 UTC m=+2141.326437746"
Jul 29 06:58:19.242595 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:19.242563    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh" event={"ID":"e12bbbe7-07d2-4b7b-969b-da0d30d092a5","Type":"ContainerStarted","Data":"66c110e3e8c6b7ee387c207867ecff7dd7fd0beb25265f37801f0c3f8ed2ebe5"}
Jul 29 06:58:19.256755 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:19.256708    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh" podStartSLOduration=2.40980092 podStartE2EDuration="3.256693112s" totalImagesPullingTime="846.892192ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:58:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:58:19.256639627 +0000 UTC m=+2142.334271394" watchObservedRunningTime="2026-07-29 06:58:19.256693112 +0000 UTC m=+2142.334324869"
Jul 29 06:58:20.248143 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:20.248109    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh"
Jul 29 06:58:26.258296 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:26.258262    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh"
Jul 29 06:58:29.244604 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:29.244526    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc10550f-7b5d6689cb-gm89x"
Jul 29 06:58:58.933677 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:58.933643    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh"]
Jul 29 06:58:58.934263 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:58.933983    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh" podUID="e12bbbe7-07d2-4b7b-969b-da0d30d092a5" containerName="maas-api" containerID="cri-o://66c110e3e8c6b7ee387c207867ecff7dd7fd0beb25265f37801f0c3f8ed2ebe5" gracePeriod=30
Jul 29 06:58:58.989828 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:58.989797    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bc10550f-7b5d6689cb-gm89x"]
Jul 29 06:58:58.990065 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:58.990039    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc10550f-7b5d6689cb-gm89x" podUID="2266f00e-1ba5-455a-b3f0-daccb2f25569" containerName="payload-pre-processing" containerID="cri-o://fc7c7d8a2ea209ce44dea4d896e87088117d659ed6b557fbae1c5d4a6c2a475c" gracePeriod=30
Jul 29 06:58:59.241505 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:59.241482    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh"
Jul 29 06:58:59.262114 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:59.262089    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc10550f-7b5d6689cb-gm89x"
Jul 29 06:58:59.286658 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:59.286633    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12bbbe7-07d2-4b7b-969b-da0d30d092a5-openshift-service-ca\") pod \"e12bbbe7-07d2-4b7b-969b-da0d30d092a5\" (UID: \"e12bbbe7-07d2-4b7b-969b-da0d30d092a5\") "
Jul 29 06:58:59.286868 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:59.286670    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2266f00e-1ba5-455a-b3f0-daccb2f25569-plugins-config-volume\") pod \"2266f00e-1ba5-455a-b3f0-daccb2f25569\" (UID: \"2266f00e-1ba5-455a-b3f0-daccb2f25569\") "
Jul 29 06:58:59.286868 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:59.286698    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12bbbe7-07d2-4b7b-969b-da0d30d092a5-maas-api-tls\") pod \"e12bbbe7-07d2-4b7b-969b-da0d30d092a5\" (UID: \"e12bbbe7-07d2-4b7b-969b-da0d30d092a5\") "
Jul 29 06:58:59.286868 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:59.286753    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twmcd\" (UniqueName: \"kubernetes.io/projected/2266f00e-1ba5-455a-b3f0-daccb2f25569-kube-api-access-twmcd\") pod \"2266f00e-1ba5-455a-b3f0-daccb2f25569\" (UID: \"2266f00e-1ba5-455a-b3f0-daccb2f25569\") "
Jul 29 06:58:59.286868 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:59.286805    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6bpz\" (UniqueName: \"kubernetes.io/projected/e12bbbe7-07d2-4b7b-969b-da0d30d092a5-kube-api-access-r6bpz\") pod \"e12bbbe7-07d2-4b7b-969b-da0d30d092a5\" (UID: \"e12bbbe7-07d2-4b7b-969b-da0d30d092a5\") "
Jul 29 06:58:59.287103 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:59.287082    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e12bbbe7-07d2-4b7b-969b-da0d30d092a5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e12bbbe7-07d2-4b7b-969b-da0d30d092a5" (UID: "e12bbbe7-07d2-4b7b-969b-da0d30d092a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 06:58:59.287167 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:59.287088    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2266f00e-1ba5-455a-b3f0-daccb2f25569-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2266f00e-1ba5-455a-b3f0-daccb2f25569" (UID: "2266f00e-1ba5-455a-b3f0-daccb2f25569"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 06:58:59.289187 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:59.289153    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e12bbbe7-07d2-4b7b-969b-da0d30d092a5-kube-api-access-r6bpz" (OuterVolumeSpecName: "kube-api-access-r6bpz") pod "e12bbbe7-07d2-4b7b-969b-da0d30d092a5" (UID: "e12bbbe7-07d2-4b7b-969b-da0d30d092a5"). InnerVolumeSpecName "kube-api-access-r6bpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 06:58:59.289312 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:59.289202    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2266f00e-1ba5-455a-b3f0-daccb2f25569-kube-api-access-twmcd" (OuterVolumeSpecName: "kube-api-access-twmcd") pod "2266f00e-1ba5-455a-b3f0-daccb2f25569" (UID: "2266f00e-1ba5-455a-b3f0-daccb2f25569"). InnerVolumeSpecName "kube-api-access-twmcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 06:58:59.289489 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:59.289460    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e12bbbe7-07d2-4b7b-969b-da0d30d092a5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e12bbbe7-07d2-4b7b-969b-da0d30d092a5" (UID: "e12bbbe7-07d2-4b7b-969b-da0d30d092a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 06:58:59.388449 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:59.388425    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12bbbe7-07d2-4b7b-969b-da0d30d092a5-openshift-service-ca\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:58:59.388449 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:59.388448    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2266f00e-1ba5-455a-b3f0-daccb2f25569-plugins-config-volume\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:58:59.388610 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:59.388459    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12bbbe7-07d2-4b7b-969b-da0d30d092a5-maas-api-tls\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:58:59.388610 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:59.388467    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twmcd\" (UniqueName: \"kubernetes.io/projected/2266f00e-1ba5-455a-b3f0-daccb2f25569-kube-api-access-twmcd\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:58:59.388610 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:59.388475    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6bpz\" (UniqueName: \"kubernetes.io/projected/e12bbbe7-07d2-4b7b-969b-da0d30d092a5-kube-api-access-r6bpz\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 06:58:59.460140 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:59.460107    2563 generic.go:356] "Generic (PLEG): container finished" podID="2266f00e-1ba5-455a-b3f0-daccb2f25569" containerID="fc7c7d8a2ea209ce44dea4d896e87088117d659ed6b557fbae1c5d4a6c2a475c" exitCode=0
Jul 29 06:58:59.460258 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:59.460152    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc10550f-7b5d6689cb-gm89x"
Jul 29 06:58:59.460258 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:59.460176    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc10550f-7b5d6689cb-gm89x" event={"ID":"2266f00e-1ba5-455a-b3f0-daccb2f25569","Type":"ContainerDied","Data":"fc7c7d8a2ea209ce44dea4d896e87088117d659ed6b557fbae1c5d4a6c2a475c"}
Jul 29 06:58:59.460258 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:59.460219    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc10550f-7b5d6689cb-gm89x" event={"ID":"2266f00e-1ba5-455a-b3f0-daccb2f25569","Type":"ContainerDied","Data":"ede0fd79d5e851a9be312a867cab08c261e0ee14023295986c4a788c22de1cd0"}
Jul 29 06:58:59.460258 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:59.460244    2563 scope.go:117] "RemoveContainer" containerID="fc7c7d8a2ea209ce44dea4d896e87088117d659ed6b557fbae1c5d4a6c2a475c"
Jul 29 06:58:59.461660 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:59.461634    2563 generic.go:356] "Generic (PLEG): container finished" podID="e12bbbe7-07d2-4b7b-969b-da0d30d092a5" containerID="66c110e3e8c6b7ee387c207867ecff7dd7fd0beb25265f37801f0c3f8ed2ebe5" exitCode=0
Jul 29 06:58:59.461767 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:59.461677    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh" event={"ID":"e12bbbe7-07d2-4b7b-969b-da0d30d092a5","Type":"ContainerDied","Data":"66c110e3e8c6b7ee387c207867ecff7dd7fd0beb25265f37801f0c3f8ed2ebe5"}
Jul 29 06:58:59.461767 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:59.461708    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh" event={"ID":"e12bbbe7-07d2-4b7b-969b-da0d30d092a5","Type":"ContainerDied","Data":"20cabc6735b47e7ade206c055e0279711a47181cfe3895eb338e3a72be6b443f"}
Jul 29 06:58:59.461767 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:59.461680    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh"
Jul 29 06:58:59.471722 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:59.471677    2563 scope.go:117] "RemoveContainer" containerID="fc7c7d8a2ea209ce44dea4d896e87088117d659ed6b557fbae1c5d4a6c2a475c"
Jul 29 06:58:59.472217 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:58:59.472194    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc7c7d8a2ea209ce44dea4d896e87088117d659ed6b557fbae1c5d4a6c2a475c\": container with ID starting with fc7c7d8a2ea209ce44dea4d896e87088117d659ed6b557fbae1c5d4a6c2a475c not found: ID does not exist" containerID="fc7c7d8a2ea209ce44dea4d896e87088117d659ed6b557fbae1c5d4a6c2a475c"
Jul 29 06:58:59.472308 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:59.472227    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc7c7d8a2ea209ce44dea4d896e87088117d659ed6b557fbae1c5d4a6c2a475c"} err="failed to get container status \"fc7c7d8a2ea209ce44dea4d896e87088117d659ed6b557fbae1c5d4a6c2a475c\": rpc error: code = NotFound desc = could not find container \"fc7c7d8a2ea209ce44dea4d896e87088117d659ed6b557fbae1c5d4a6c2a475c\": container with ID starting with fc7c7d8a2ea209ce44dea4d896e87088117d659ed6b557fbae1c5d4a6c2a475c not found: ID does not exist"
Jul 29 06:58:59.472308 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:59.472252    2563 scope.go:117] "RemoveContainer" containerID="66c110e3e8c6b7ee387c207867ecff7dd7fd0beb25265f37801f0c3f8ed2ebe5"
Jul 29 06:58:59.481400 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:59.481384    2563 scope.go:117] "RemoveContainer" containerID="66c110e3e8c6b7ee387c207867ecff7dd7fd0beb25265f37801f0c3f8ed2ebe5"
Jul 29 06:58:59.481655 ip-10-0-129-223 kubenswrapper[2563]: E0729 06:58:59.481636    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66c110e3e8c6b7ee387c207867ecff7dd7fd0beb25265f37801f0c3f8ed2ebe5\": container with ID starting with 66c110e3e8c6b7ee387c207867ecff7dd7fd0beb25265f37801f0c3f8ed2ebe5 not found: ID does not exist" containerID="66c110e3e8c6b7ee387c207867ecff7dd7fd0beb25265f37801f0c3f8ed2ebe5"
Jul 29 06:58:59.481710 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:59.481664    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66c110e3e8c6b7ee387c207867ecff7dd7fd0beb25265f37801f0c3f8ed2ebe5"} err="failed to get container status \"66c110e3e8c6b7ee387c207867ecff7dd7fd0beb25265f37801f0c3f8ed2ebe5\": rpc error: code = NotFound desc = could not find container \"66c110e3e8c6b7ee387c207867ecff7dd7fd0beb25265f37801f0c3f8ed2ebe5\": container with ID starting with 66c110e3e8c6b7ee387c207867ecff7dd7fd0beb25265f37801f0c3f8ed2ebe5 not found: ID does not exist"
Jul 29 06:58:59.481784 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:59.481773    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh"]
Jul 29 06:58:59.483613 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:59.483591    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bc10550f-5d97798f45-v4rsh"]
Jul 29 06:58:59.491235 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:59.491216    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bc10550f-7b5d6689cb-gm89x"]
Jul 29 06:58:59.494601 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:58:59.494584    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bc10550f-7b5d6689cb-gm89x"]
Jul 29 06:59:00.243334 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:59:00.243243    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc10550f-7b5d6689cb-gm89x" podUID="2266f00e-1ba5-455a-b3f0-daccb2f25569" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.53:9004: i/o timeout"
Jul 29 06:59:01.398100 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:59:01.398066    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2266f00e-1ba5-455a-b3f0-daccb2f25569" path="/var/lib/kubelet/pods/2266f00e-1ba5-455a-b3f0-daccb2f25569/volumes"
Jul 29 06:59:01.398650 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:59:01.398634    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e12bbbe7-07d2-4b7b-969b-da0d30d092a5" path="/var/lib/kubelet/pods/e12bbbe7-07d2-4b7b-969b-da0d30d092a5/volumes"
Jul 29 06:59:27.964605 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:59:27.964571    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-11cbe871-8684569c7-nwsk9"]
Jul 29 06:59:27.965217 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:59:27.965200    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2266f00e-1ba5-455a-b3f0-daccb2f25569" containerName="payload-pre-processing"
Jul 29 06:59:27.965267 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:59:27.965221    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2266f00e-1ba5-455a-b3f0-daccb2f25569" containerName="payload-pre-processing"
Jul 29 06:59:27.965267 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:59:27.965241    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e12bbbe7-07d2-4b7b-969b-da0d30d092a5" containerName="maas-api"
Jul 29 06:59:27.965267 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:59:27.965250    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e12bbbe7-07d2-4b7b-969b-da0d30d092a5" containerName="maas-api"
Jul 29 06:59:27.965369 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:59:27.965347    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e12bbbe7-07d2-4b7b-969b-da0d30d092a5" containerName="maas-api"
Jul 29 06:59:27.965369 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:59:27.965358    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2266f00e-1ba5-455a-b3f0-daccb2f25569" containerName="payload-pre-processing"
Jul 29 06:59:27.968838 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:59:27.968821    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11cbe871-8684569c7-nwsk9"
Jul 29 06:59:27.970543 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:59:27.970523    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-11cbe871-serving-cert\""
Jul 29 06:59:27.978195 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:59:27.978171    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-11cbe871-8684569c7-nwsk9"]
Jul 29 06:59:28.145343 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:59:28.145307    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vpwh\" (UniqueName: \"kubernetes.io/projected/9aa4c325-bdab-42e2-a207-c1415a298afd-kube-api-access-5vpwh\") pod \"maas-api-e2e-mt-11cbe871-8684569c7-nwsk9\" (UID: \"9aa4c325-bdab-42e2-a207-c1415a298afd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11cbe871-8684569c7-nwsk9"
Jul 29 06:59:28.145343 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:59:28.145344    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa4c325-bdab-42e2-a207-c1415a298afd-openshift-service-ca\") pod \"maas-api-e2e-mt-11cbe871-8684569c7-nwsk9\" (UID: \"9aa4c325-bdab-42e2-a207-c1415a298afd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11cbe871-8684569c7-nwsk9"
Jul 29 06:59:28.145549 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:59:28.145371    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aa4c325-bdab-42e2-a207-c1415a298afd-maas-api-tls\") pod \"maas-api-e2e-mt-11cbe871-8684569c7-nwsk9\" (UID: \"9aa4c325-bdab-42e2-a207-c1415a298afd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11cbe871-8684569c7-nwsk9"
Jul 29 06:59:28.246186 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:59:28.246092    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vpwh\" (UniqueName: \"kubernetes.io/projected/9aa4c325-bdab-42e2-a207-c1415a298afd-kube-api-access-5vpwh\") pod \"maas-api-e2e-mt-11cbe871-8684569c7-nwsk9\" (UID: \"9aa4c325-bdab-42e2-a207-c1415a298afd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11cbe871-8684569c7-nwsk9"
Jul 29 06:59:28.246186 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:59:28.246134    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa4c325-bdab-42e2-a207-c1415a298afd-openshift-service-ca\") pod \"maas-api-e2e-mt-11cbe871-8684569c7-nwsk9\" (UID: \"9aa4c325-bdab-42e2-a207-c1415a298afd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11cbe871-8684569c7-nwsk9"
Jul 29 06:59:28.246186 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:59:28.246161    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aa4c325-bdab-42e2-a207-c1415a298afd-maas-api-tls\") pod \"maas-api-e2e-mt-11cbe871-8684569c7-nwsk9\" (UID: \"9aa4c325-bdab-42e2-a207-c1415a298afd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11cbe871-8684569c7-nwsk9"
Jul 29 06:59:28.246768 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:59:28.246736    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa4c325-bdab-42e2-a207-c1415a298afd-openshift-service-ca\") pod \"maas-api-e2e-mt-11cbe871-8684569c7-nwsk9\" (UID: \"9aa4c325-bdab-42e2-a207-c1415a298afd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11cbe871-8684569c7-nwsk9"
Jul 29 06:59:28.248828 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:59:28.248796    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aa4c325-bdab-42e2-a207-c1415a298afd-maas-api-tls\") pod \"maas-api-e2e-mt-11cbe871-8684569c7-nwsk9\" (UID: \"9aa4c325-bdab-42e2-a207-c1415a298afd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11cbe871-8684569c7-nwsk9"
Jul 29 06:59:28.256489 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:59:28.256464    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vpwh\" (UniqueName: \"kubernetes.io/projected/9aa4c325-bdab-42e2-a207-c1415a298afd-kube-api-access-5vpwh\") pod \"maas-api-e2e-mt-11cbe871-8684569c7-nwsk9\" (UID: \"9aa4c325-bdab-42e2-a207-c1415a298afd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-11cbe871-8684569c7-nwsk9"
Jul 29 06:59:28.282283 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:59:28.282253    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11cbe871-8684569c7-nwsk9"
Jul 29 06:59:28.426190 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:59:28.426161    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-11cbe871-8684569c7-nwsk9"]
Jul 29 06:59:28.429514 ip-10-0-129-223 kubenswrapper[2563]: W0729 06:59:28.429489    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aa4c325_bdab_42e2_a207_c1415a298afd.slice/crio-7ce34152e30e3293839cd12797283feb5efc6aed9b5ab9f7d527e414764db597 WatchSource:0}: Error finding container 7ce34152e30e3293839cd12797283feb5efc6aed9b5ab9f7d527e414764db597: Status 404 returned error can't find the container with id 7ce34152e30e3293839cd12797283feb5efc6aed9b5ab9f7d527e414764db597
Jul 29 06:59:28.431198 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:59:28.431178    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 06:59:28.610759 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:59:28.610727    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11cbe871-8684569c7-nwsk9" event={"ID":"9aa4c325-bdab-42e2-a207-c1415a298afd","Type":"ContainerStarted","Data":"7ce34152e30e3293839cd12797283feb5efc6aed9b5ab9f7d527e414764db597"}
Jul 29 06:59:29.617932 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:59:29.617895    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11cbe871-8684569c7-nwsk9" event={"ID":"9aa4c325-bdab-42e2-a207-c1415a298afd","Type":"ContainerStarted","Data":"7cbe9c9321904af8d6f3f98924f2035517a4ad3578f9f7d346dfc289812c81ce"}
Jul 29 06:59:29.618325 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:59:29.618006    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11cbe871-8684569c7-nwsk9"
Jul 29 06:59:29.633048 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:59:29.632968    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11cbe871-8684569c7-nwsk9" podStartSLOduration=2.162502991 podStartE2EDuration="2.632947815s" totalImagesPullingTime="470.444824ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 06:59:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 06:59:29.63251262 +0000 UTC m=+2212.710144397" watchObservedRunningTime="2026-07-29 06:59:29.632947815 +0000 UTC m=+2212.710579583"
Jul 29 06:59:35.628641 ip-10-0-129-223 kubenswrapper[2563]: I0729 06:59:35.628613    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11cbe871-8684569c7-nwsk9"
Jul 29 07:00:00.131841 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:00.131757    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755140-7687w"]
Jul 29 07:00:00.135566 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:00.135548    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11cbe871-29755140-bp88b"]
Jul 29 07:00:00.135702 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:00.135683    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755140-7687w"
Jul 29 07:00:00.137666 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:00.137642    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bj47g\""
Jul 29 07:00:00.139707 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:00.139685    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11cbe871-29755140-bp88b"
Jul 29 07:00:00.141623 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:00.141600    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755140-7687w"]
Jul 29 07:00:00.149676 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:00.149656    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755140-7687w"
Jul 29 07:00:00.154608 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:00.154587    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11cbe871-29755140-bp88b"]
Jul 29 07:00:00.159115 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:00.159098    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11cbe871-29755140-bp88b"
Jul 29 07:00:00.297166 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:00.297136    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29755140-7687w"]
Jul 29 07:00:00.299053 ip-10-0-129-223 kubenswrapper[2563]: W0729 07:00:00.299024    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca26ab7d_4c33_4a9b_a510_d9f477cf199c.slice/crio-db2a3995aec5276f3aa293bd1bb38597c2e6481e9e700015978263b8cfc1d162 WatchSource:0}: Error finding container db2a3995aec5276f3aa293bd1bb38597c2e6481e9e700015978263b8cfc1d162: Status 404 returned error can't find the container with id db2a3995aec5276f3aa293bd1bb38597c2e6481e9e700015978263b8cfc1d162
Jul 29 07:00:00.320215 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:00.320192    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11cbe871-29755140-bp88b"]
Jul 29 07:00:00.322364 ip-10-0-129-223 kubenswrapper[2563]: W0729 07:00:00.322342    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d60d98c_505b_4da4_bad1_6de91235caca.slice/crio-0dd34618c7a0fbff71b8fbc5a72f886e453aff64f5283844b0a54b2599334dd0 WatchSource:0}: Error finding container 0dd34618c7a0fbff71b8fbc5a72f886e453aff64f5283844b0a54b2599334dd0: Status 404 returned error can't find the container with id 0dd34618c7a0fbff71b8fbc5a72f886e453aff64f5283844b0a54b2599334dd0
Jul 29 07:00:00.772934 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:00.772895    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11cbe871-29755140-bp88b" event={"ID":"5d60d98c-505b-4da4-bad1-6de91235caca","Type":"ContainerStarted","Data":"256c4d16c63217be818eeba641e5c77acf1b99dd3232351d08c471d906d59526"}
Jul 29 07:00:00.773140 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:00.772953    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11cbe871-29755140-bp88b" event={"ID":"5d60d98c-505b-4da4-bad1-6de91235caca","Type":"ContainerStarted","Data":"0dd34618c7a0fbff71b8fbc5a72f886e453aff64f5283844b0a54b2599334dd0"}
Jul 29 07:00:00.774418 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:00.774396    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755140-7687w" event={"ID":"ca26ab7d-4c33-4a9b-a510-d9f477cf199c","Type":"ContainerStarted","Data":"efb4f07c4b2888eda67881c8590495f2e930fff4fcb96d27712512de22f403d3"}
Jul 29 07:00:00.774527 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:00.774436    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755140-7687w" event={"ID":"ca26ab7d-4c33-4a9b-a510-d9f477cf199c","Type":"ContainerStarted","Data":"db2a3995aec5276f3aa293bd1bb38597c2e6481e9e700015978263b8cfc1d162"}
Jul 29 07:00:00.787826 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:00.787777    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11cbe871-29755140-bp88b" podStartSLOduration=0.787760224 podStartE2EDuration="787.760224ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:00:00.787371811 +0000 UTC m=+2243.865003577" watchObservedRunningTime="2026-07-29 07:00:00.787760224 +0000 UTC m=+2243.865391992"
Jul 29 07:00:00.800564 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:00.800480    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755140-7687w" podStartSLOduration=0.800468201 podStartE2EDuration="800.468201ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:00:00.799563997 +0000 UTC m=+2243.877195761" watchObservedRunningTime="2026-07-29 07:00:00.800468201 +0000 UTC m=+2243.878099967"
Jul 29 07:00:01.780818 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:01.780772    2563 generic.go:356] "Generic (PLEG): container finished" podID="5d60d98c-505b-4da4-bad1-6de91235caca" containerID="256c4d16c63217be818eeba641e5c77acf1b99dd3232351d08c471d906d59526" exitCode=0
Jul 29 07:00:01.781255 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:01.780890    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11cbe871-29755140-bp88b" event={"ID":"5d60d98c-505b-4da4-bad1-6de91235caca","Type":"ContainerDied","Data":"256c4d16c63217be818eeba641e5c77acf1b99dd3232351d08c471d906d59526"}
Jul 29 07:00:01.782445 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:01.782419    2563 generic.go:356] "Generic (PLEG): container finished" podID="ca26ab7d-4c33-4a9b-a510-d9f477cf199c" containerID="efb4f07c4b2888eda67881c8590495f2e930fff4fcb96d27712512de22f403d3" exitCode=0
Jul 29 07:00:01.782566 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:01.782504    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755140-7687w" event={"ID":"ca26ab7d-4c33-4a9b-a510-d9f477cf199c","Type":"ContainerDied","Data":"efb4f07c4b2888eda67881c8590495f2e930fff4fcb96d27712512de22f403d3"}
Jul 29 07:00:02.918221 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:02.918198    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11cbe871-29755140-bp88b"
Jul 29 07:00:02.949812 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:02.949790    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755140-7687w"
Jul 29 07:00:03.793515 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:03.793422    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11cbe871-29755140-bp88b"
Jul 29 07:00:03.793515 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:03.793423    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11cbe871-29755140-bp88b" event={"ID":"5d60d98c-505b-4da4-bad1-6de91235caca","Type":"ContainerDied","Data":"0dd34618c7a0fbff71b8fbc5a72f886e453aff64f5283844b0a54b2599334dd0"}
Jul 29 07:00:03.793729 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:03.793546    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0dd34618c7a0fbff71b8fbc5a72f886e453aff64f5283844b0a54b2599334dd0"
Jul 29 07:00:03.794748 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:03.794726    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755140-7687w"
Jul 29 07:00:03.794904 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:03.794778    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29755140-7687w" event={"ID":"ca26ab7d-4c33-4a9b-a510-d9f477cf199c","Type":"ContainerDied","Data":"db2a3995aec5276f3aa293bd1bb38597c2e6481e9e700015978263b8cfc1d162"}
Jul 29 07:00:03.794904 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:03.794826    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db2a3995aec5276f3aa293bd1bb38597c2e6481e9e700015978263b8cfc1d162"
Jul 29 07:00:42.751457 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:42.751425    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-11cbe871-8684569c7-nwsk9"]
Jul 29 07:00:42.752140 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:42.751697    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11cbe871-8684569c7-nwsk9" podUID="9aa4c325-bdab-42e2-a207-c1415a298afd" containerName="maas-api" containerID="cri-o://7cbe9c9321904af8d6f3f98924f2035517a4ad3578f9f7d346dfc289812c81ce" gracePeriod=30
Jul 29 07:00:42.782992 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:42.782962    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11cbe871-29755140-bp88b"]
Jul 29 07:00:42.785662 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:42.785640    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-11cbe871-29755140-bp88b"]
Jul 29 07:00:42.999613 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:42.999535    2563 generic.go:356] "Generic (PLEG): container finished" podID="9aa4c325-bdab-42e2-a207-c1415a298afd" containerID="7cbe9c9321904af8d6f3f98924f2035517a4ad3578f9f7d346dfc289812c81ce" exitCode=0
Jul 29 07:00:42.999613 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:42.999577    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11cbe871-8684569c7-nwsk9" event={"ID":"9aa4c325-bdab-42e2-a207-c1415a298afd","Type":"ContainerDied","Data":"7cbe9c9321904af8d6f3f98924f2035517a4ad3578f9f7d346dfc289812c81ce"}
Jul 29 07:00:43.015266 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:43.015216    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11cbe871-8684569c7-nwsk9"
Jul 29 07:00:43.117525 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:43.117490    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa4c325-bdab-42e2-a207-c1415a298afd-openshift-service-ca\") pod \"9aa4c325-bdab-42e2-a207-c1415a298afd\" (UID: \"9aa4c325-bdab-42e2-a207-c1415a298afd\") "
Jul 29 07:00:43.117743 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:43.117536    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aa4c325-bdab-42e2-a207-c1415a298afd-maas-api-tls\") pod \"9aa4c325-bdab-42e2-a207-c1415a298afd\" (UID: \"9aa4c325-bdab-42e2-a207-c1415a298afd\") "
Jul 29 07:00:43.117743 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:43.117564    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vpwh\" (UniqueName: \"kubernetes.io/projected/9aa4c325-bdab-42e2-a207-c1415a298afd-kube-api-access-5vpwh\") pod \"9aa4c325-bdab-42e2-a207-c1415a298afd\" (UID: \"9aa4c325-bdab-42e2-a207-c1415a298afd\") "
Jul 29 07:00:43.117986 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:43.117953    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aa4c325-bdab-42e2-a207-c1415a298afd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9aa4c325-bdab-42e2-a207-c1415a298afd" (UID: "9aa4c325-bdab-42e2-a207-c1415a298afd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 07:00:43.120005 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:43.119982    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aa4c325-bdab-42e2-a207-c1415a298afd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9aa4c325-bdab-42e2-a207-c1415a298afd" (UID: "9aa4c325-bdab-42e2-a207-c1415a298afd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 07:00:43.120106 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:43.120046    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aa4c325-bdab-42e2-a207-c1415a298afd-kube-api-access-5vpwh" (OuterVolumeSpecName: "kube-api-access-5vpwh") pod "9aa4c325-bdab-42e2-a207-c1415a298afd" (UID: "9aa4c325-bdab-42e2-a207-c1415a298afd"). InnerVolumeSpecName "kube-api-access-5vpwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 07:00:43.218675 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:43.218645    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa4c325-bdab-42e2-a207-c1415a298afd-openshift-service-ca\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:00:43.218675 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:43.218672    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aa4c325-bdab-42e2-a207-c1415a298afd-maas-api-tls\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:00:43.218675 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:43.218681    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vpwh\" (UniqueName: \"kubernetes.io/projected/9aa4c325-bdab-42e2-a207-c1415a298afd-kube-api-access-5vpwh\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:00:43.397612 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:43.397581    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d60d98c-505b-4da4-bad1-6de91235caca" path="/var/lib/kubelet/pods/5d60d98c-505b-4da4-bad1-6de91235caca/volumes"
Jul 29 07:00:44.005769 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:44.005740    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11cbe871-8684569c7-nwsk9"
Jul 29 07:00:44.006232 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:44.005739    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-11cbe871-8684569c7-nwsk9" event={"ID":"9aa4c325-bdab-42e2-a207-c1415a298afd","Type":"ContainerDied","Data":"7ce34152e30e3293839cd12797283feb5efc6aed9b5ab9f7d527e414764db597"}
Jul 29 07:00:44.006232 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:44.005912    2563 scope.go:117] "RemoveContainer" containerID="7cbe9c9321904af8d6f3f98924f2035517a4ad3578f9f7d346dfc289812c81ce"
Jul 29 07:00:44.020171 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:44.020148    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-11cbe871-8684569c7-nwsk9"]
Jul 29 07:00:44.023650 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:44.023628    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-11cbe871-8684569c7-nwsk9"]
Jul 29 07:00:45.398517 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:00:45.398484    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9aa4c325-bdab-42e2-a207-c1415a298afd" path="/var/lib/kubelet/pods/9aa4c325-bdab-42e2-a207-c1415a298afd/volumes"
Jul 29 07:02:31.675737 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:31.675694    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4"]
Jul 29 07:02:31.676275 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:31.676105    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d60d98c-505b-4da4-bad1-6de91235caca" containerName="cleanup"
Jul 29 07:02:31.676275 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:31.676117    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d60d98c-505b-4da4-bad1-6de91235caca" containerName="cleanup"
Jul 29 07:02:31.676275 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:31.676143    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca26ab7d-4c33-4a9b-a510-d9f477cf199c" containerName="cleanup"
Jul 29 07:02:31.676275 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:31.676190    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca26ab7d-4c33-4a9b-a510-d9f477cf199c" containerName="cleanup"
Jul 29 07:02:31.676275 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:31.676217    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aa4c325-bdab-42e2-a207-c1415a298afd" containerName="maas-api"
Jul 29 07:02:31.676275 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:31.676227    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aa4c325-bdab-42e2-a207-c1415a298afd" containerName="maas-api"
Jul 29 07:02:31.676612 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:31.676346    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aa4c325-bdab-42e2-a207-c1415a298afd" containerName="maas-api"
Jul 29 07:02:31.676612 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:31.676362    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d60d98c-505b-4da4-bad1-6de91235caca" containerName="cleanup"
Jul 29 07:02:31.676612 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:31.676374    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca26ab7d-4c33-4a9b-a510-d9f477cf199c" containerName="cleanup"
Jul 29 07:02:31.680258 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:31.680236    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4"
Jul 29 07:02:31.682219 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:31.682201    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-dd1393-serving-cert\""
Jul 29 07:02:31.690529 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:31.690504    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4"]
Jul 29 07:02:31.722544 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:31.722516    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7nqk\" (UniqueName: \"kubernetes.io/projected/7cba4bfb-f9f7-4d76-ba82-91458918abb2-kube-api-access-k7nqk\") pod \"maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4\" (UID: \"7cba4bfb-f9f7-4d76-ba82-91458918abb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4"
Jul 29 07:02:31.722680 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:31.722660    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cba4bfb-f9f7-4d76-ba82-91458918abb2-maas-api-tls\") pod \"maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4\" (UID: \"7cba4bfb-f9f7-4d76-ba82-91458918abb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4"
Jul 29 07:02:31.722761 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:31.722743    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cba4bfb-f9f7-4d76-ba82-91458918abb2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4\" (UID: \"7cba4bfb-f9f7-4d76-ba82-91458918abb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4"
Jul 29 07:02:31.823354 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:31.823327    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cba4bfb-f9f7-4d76-ba82-91458918abb2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4\" (UID: \"7cba4bfb-f9f7-4d76-ba82-91458918abb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4"
Jul 29 07:02:31.823524 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:31.823380    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7nqk\" (UniqueName: \"kubernetes.io/projected/7cba4bfb-f9f7-4d76-ba82-91458918abb2-kube-api-access-k7nqk\") pod \"maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4\" (UID: \"7cba4bfb-f9f7-4d76-ba82-91458918abb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4"
Jul 29 07:02:31.823524 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:31.823429    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cba4bfb-f9f7-4d76-ba82-91458918abb2-maas-api-tls\") pod \"maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4\" (UID: \"7cba4bfb-f9f7-4d76-ba82-91458918abb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4"
Jul 29 07:02:31.823609 ip-10-0-129-223 kubenswrapper[2563]: E0729 07:02:31.823527    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-dd1393-serving-cert: secret "maas-api-e2e-inf-a-dd1393-serving-cert" not found
Jul 29 07:02:31.823609 ip-10-0-129-223 kubenswrapper[2563]: E0729 07:02:31.823590    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cba4bfb-f9f7-4d76-ba82-91458918abb2-maas-api-tls podName:7cba4bfb-f9f7-4d76-ba82-91458918abb2 nodeName:}" failed. No retries permitted until 2026-07-29 07:02:32.323573925 +0000 UTC m=+2395.401205669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7cba4bfb-f9f7-4d76-ba82-91458918abb2-maas-api-tls") pod "maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4" (UID: "7cba4bfb-f9f7-4d76-ba82-91458918abb2") : secret "maas-api-e2e-inf-a-dd1393-serving-cert" not found
Jul 29 07:02:31.823978 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:31.823960    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cba4bfb-f9f7-4d76-ba82-91458918abb2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4\" (UID: \"7cba4bfb-f9f7-4d76-ba82-91458918abb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4"
Jul 29 07:02:31.831380 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:31.831355    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7nqk\" (UniqueName: \"kubernetes.io/projected/7cba4bfb-f9f7-4d76-ba82-91458918abb2-kube-api-access-k7nqk\") pod \"maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4\" (UID: \"7cba4bfb-f9f7-4d76-ba82-91458918abb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4"
Jul 29 07:02:32.328126 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:32.328088    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cba4bfb-f9f7-4d76-ba82-91458918abb2-maas-api-tls\") pod \"maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4\" (UID: \"7cba4bfb-f9f7-4d76-ba82-91458918abb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4"
Jul 29 07:02:32.330625 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:32.330601    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cba4bfb-f9f7-4d76-ba82-91458918abb2-maas-api-tls\") pod \"maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4\" (UID: \"7cba4bfb-f9f7-4d76-ba82-91458918abb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4"
Jul 29 07:02:32.593668 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:32.593589    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4"
Jul 29 07:02:32.730914 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:32.730889    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4"]
Jul 29 07:02:32.736750 ip-10-0-129-223 kubenswrapper[2563]: W0729 07:02:32.733670    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cba4bfb_f9f7_4d76_ba82_91458918abb2.slice/crio-6aa1716be784f8286ebcaa2c09c68d700b82ee8dd4b9f66358babb5425a88175 WatchSource:0}: Error finding container 6aa1716be784f8286ebcaa2c09c68d700b82ee8dd4b9f66358babb5425a88175: Status 404 returned error can't find the container with id 6aa1716be784f8286ebcaa2c09c68d700b82ee8dd4b9f66358babb5425a88175
Jul 29 07:02:33.565076 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:33.565028    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4" event={"ID":"7cba4bfb-f9f7-4d76-ba82-91458918abb2","Type":"ContainerStarted","Data":"72ea8da183d360790ec66e73420e56842e1b99b508288c64fcfba5c10a9c0035"}
Jul 29 07:02:33.565245 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:33.565098    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4" event={"ID":"7cba4bfb-f9f7-4d76-ba82-91458918abb2","Type":"ContainerStarted","Data":"6aa1716be784f8286ebcaa2c09c68d700b82ee8dd4b9f66358babb5425a88175"}
Jul 29 07:02:33.565245 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:33.565155    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4"
Jul 29 07:02:33.581288 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:33.581237    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4" podStartSLOduration=1.9852434730000001 podStartE2EDuration="2.581221145s" totalImagesPullingTime="595.977672ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:02:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:02:33.579425595 +0000 UTC m=+2396.657057363" watchObservedRunningTime="2026-07-29 07:02:33.581221145 +0000 UTC m=+2396.658852912"
Jul 29 07:02:37.570736 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:37.570707    2563 scope.go:117] "RemoveContainer" containerID="dc2c050aa39e1713c847c039e15fb3e55724a2329aa37f5488724d3255b9b898"
Jul 29 07:02:37.579171 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:37.579152    2563 scope.go:117] "RemoveContainer" containerID="bee36473bc0aa308c391eaef9db9f0527143b5d557d7c104073ecb7237dea397"
Jul 29 07:02:39.575416 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:39.575389    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4"
Jul 29 07:02:59.833101 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:59.833021    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h"]
Jul 29 07:02:59.836757 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:59.836736    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h"
Jul 29 07:02:59.838744 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:59.838715    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-9aa496-serving-cert\""
Jul 29 07:02:59.846834 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:59.846318    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h"]
Jul 29 07:02:59.981076 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:59.981032    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8s5w\" (UniqueName: \"kubernetes.io/projected/50a1eba3-c5cd-4c97-ae5e-c359e8bd6622-kube-api-access-j8s5w\") pod \"maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h\" (UID: \"50a1eba3-c5cd-4c97-ae5e-c359e8bd6622\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h"
Jul 29 07:02:59.981246 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:59.981085    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50a1eba3-c5cd-4c97-ae5e-c359e8bd6622-maas-api-tls\") pod \"maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h\" (UID: \"50a1eba3-c5cd-4c97-ae5e-c359e8bd6622\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h"
Jul 29 07:02:59.981246 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:02:59.981173    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a1eba3-c5cd-4c97-ae5e-c359e8bd6622-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h\" (UID: \"50a1eba3-c5cd-4c97-ae5e-c359e8bd6622\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h"
Jul 29 07:03:00.082158 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:00.082125    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8s5w\" (UniqueName: \"kubernetes.io/projected/50a1eba3-c5cd-4c97-ae5e-c359e8bd6622-kube-api-access-j8s5w\") pod \"maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h\" (UID: \"50a1eba3-c5cd-4c97-ae5e-c359e8bd6622\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h"
Jul 29 07:03:00.082158 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:00.082159    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50a1eba3-c5cd-4c97-ae5e-c359e8bd6622-maas-api-tls\") pod \"maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h\" (UID: \"50a1eba3-c5cd-4c97-ae5e-c359e8bd6622\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h"
Jul 29 07:03:00.082372 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:00.082201    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a1eba3-c5cd-4c97-ae5e-c359e8bd6622-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h\" (UID: \"50a1eba3-c5cd-4c97-ae5e-c359e8bd6622\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h"
Jul 29 07:03:00.082372 ip-10-0-129-223 kubenswrapper[2563]: E0729 07:03:00.082336    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-9aa496-serving-cert: secret "maas-api-e2e-inf-b-9aa496-serving-cert" not found
Jul 29 07:03:00.082449 ip-10-0-129-223 kubenswrapper[2563]: E0729 07:03:00.082405    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50a1eba3-c5cd-4c97-ae5e-c359e8bd6622-maas-api-tls podName:50a1eba3-c5cd-4c97-ae5e-c359e8bd6622 nodeName:}" failed. No retries permitted until 2026-07-29 07:03:00.582389336 +0000 UTC m=+2423.660021081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/50a1eba3-c5cd-4c97-ae5e-c359e8bd6622-maas-api-tls") pod "maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h" (UID: "50a1eba3-c5cd-4c97-ae5e-c359e8bd6622") : secret "maas-api-e2e-inf-b-9aa496-serving-cert" not found
Jul 29 07:03:00.082805 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:00.082789    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a1eba3-c5cd-4c97-ae5e-c359e8bd6622-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h\" (UID: \"50a1eba3-c5cd-4c97-ae5e-c359e8bd6622\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h"
Jul 29 07:03:00.089709 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:00.089647    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8s5w\" (UniqueName: \"kubernetes.io/projected/50a1eba3-c5cd-4c97-ae5e-c359e8bd6622-kube-api-access-j8s5w\") pod \"maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h\" (UID: \"50a1eba3-c5cd-4c97-ae5e-c359e8bd6622\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h"
Jul 29 07:03:00.587363 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:00.587329    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50a1eba3-c5cd-4c97-ae5e-c359e8bd6622-maas-api-tls\") pod \"maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h\" (UID: \"50a1eba3-c5cd-4c97-ae5e-c359e8bd6622\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h"
Jul 29 07:03:00.590147 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:00.590120    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50a1eba3-c5cd-4c97-ae5e-c359e8bd6622-maas-api-tls\") pod \"maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h\" (UID: \"50a1eba3-c5cd-4c97-ae5e-c359e8bd6622\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h"
Jul 29 07:03:00.752293 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:00.752260    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h"
Jul 29 07:03:00.889033 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:00.888919    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h"]
Jul 29 07:03:00.891351 ip-10-0-129-223 kubenswrapper[2563]: W0729 07:03:00.891325    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50a1eba3_c5cd_4c97_ae5e_c359e8bd6622.slice/crio-39d7580346b7cde7885fdda30f21bdc3cfa6160b9e5ba51ebd92c03cd819de76 WatchSource:0}: Error finding container 39d7580346b7cde7885fdda30f21bdc3cfa6160b9e5ba51ebd92c03cd819de76: Status 404 returned error can't find the container with id 39d7580346b7cde7885fdda30f21bdc3cfa6160b9e5ba51ebd92c03cd819de76
Jul 29 07:03:01.714212 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:01.714173    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h" event={"ID":"50a1eba3-c5cd-4c97-ae5e-c359e8bd6622","Type":"ContainerStarted","Data":"9b06d79cb69694ddde966b9589e7127e4cc8f02adb749dc1055cea3dc58afbda"}
Jul 29 07:03:01.714375 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:01.714232    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h" event={"ID":"50a1eba3-c5cd-4c97-ae5e-c359e8bd6622","Type":"ContainerStarted","Data":"39d7580346b7cde7885fdda30f21bdc3cfa6160b9e5ba51ebd92c03cd819de76"}
Jul 29 07:03:01.714375 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:01.714300    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h"
Jul 29 07:03:01.731694 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:01.731636    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h" podStartSLOduration=2.096062956 podStartE2EDuration="2.731620607s" totalImagesPullingTime="635.557651ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:02:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:03:01.729580118 +0000 UTC m=+2424.807211885" watchObservedRunningTime="2026-07-29 07:03:01.731620607 +0000 UTC m=+2424.809252391"
Jul 29 07:03:07.725702 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:07.725673    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h"
Jul 29 07:03:43.015916 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:43.015881    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf"]
Jul 29 07:03:43.019976 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:43.019958    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf"
Jul 29 07:03:43.023131 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:43.023112    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-dd1393\"/\"kube-root-ca.crt\""
Jul 29 07:03:43.023657 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:43.023633    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-dd1393\"/\"openshift-service-ca.crt\""
Jul 29 07:03:43.023767 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:43.023706    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-dd1393\"/\"test-model-dd1393-kserve-self-signed-certs\""
Jul 29 07:03:43.023829 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:43.023710    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-dd1393\"/\"default-dockercfg-fdqfc\""
Jul 29 07:03:43.042584 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:43.042553    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf"]
Jul 29 07:03:43.177788 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:43.177746    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2385ea2-646c-45e9-b8ab-53844944d549-model-cache\") pod \"test-model-dd1393-kserve-6cb4d97b45-hxfsf\" (UID: \"d2385ea2-646c-45e9-b8ab-53844944d549\") " pod="ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf"
Jul 29 07:03:43.177994 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:43.177820    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2385ea2-646c-45e9-b8ab-53844944d549-dshm\") pod \"test-model-dd1393-kserve-6cb4d97b45-hxfsf\" (UID: \"d2385ea2-646c-45e9-b8ab-53844944d549\") " pod="ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf"
Jul 29 07:03:43.177994 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:43.177926    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2385ea2-646c-45e9-b8ab-53844944d549-tmp-dir\") pod \"test-model-dd1393-kserve-6cb4d97b45-hxfsf\" (UID: \"d2385ea2-646c-45e9-b8ab-53844944d549\") " pod="ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf"
Jul 29 07:03:43.177994 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:43.177960    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2385ea2-646c-45e9-b8ab-53844944d549-home\") pod \"test-model-dd1393-kserve-6cb4d97b45-hxfsf\" (UID: \"d2385ea2-646c-45e9-b8ab-53844944d549\") " pod="ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf"
Jul 29 07:03:43.177994 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:43.177981    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2385ea2-646c-45e9-b8ab-53844944d549-tls-certs\") pod \"test-model-dd1393-kserve-6cb4d97b45-hxfsf\" (UID: \"d2385ea2-646c-45e9-b8ab-53844944d549\") " pod="ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf"
Jul 29 07:03:43.178132 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:43.178006    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmhzj\" (UniqueName: \"kubernetes.io/projected/d2385ea2-646c-45e9-b8ab-53844944d549-kube-api-access-zmhzj\") pod \"test-model-dd1393-kserve-6cb4d97b45-hxfsf\" (UID: \"d2385ea2-646c-45e9-b8ab-53844944d549\") " pod="ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf"
Jul 29 07:03:43.278690 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:43.278604    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2385ea2-646c-45e9-b8ab-53844944d549-dshm\") pod \"test-model-dd1393-kserve-6cb4d97b45-hxfsf\" (UID: \"d2385ea2-646c-45e9-b8ab-53844944d549\") " pod="ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf"
Jul 29 07:03:43.278690 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:43.278650    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2385ea2-646c-45e9-b8ab-53844944d549-tmp-dir\") pod \"test-model-dd1393-kserve-6cb4d97b45-hxfsf\" (UID: \"d2385ea2-646c-45e9-b8ab-53844944d549\") " pod="ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf"
Jul 29 07:03:43.278690 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:43.278668    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2385ea2-646c-45e9-b8ab-53844944d549-home\") pod \"test-model-dd1393-kserve-6cb4d97b45-hxfsf\" (UID: \"d2385ea2-646c-45e9-b8ab-53844944d549\") " pod="ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf"
Jul 29 07:03:43.278690 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:43.278685    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2385ea2-646c-45e9-b8ab-53844944d549-tls-certs\") pod \"test-model-dd1393-kserve-6cb4d97b45-hxfsf\" (UID: \"d2385ea2-646c-45e9-b8ab-53844944d549\") " pod="ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf"
Jul 29 07:03:43.279067 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:43.278704    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmhzj\" (UniqueName: \"kubernetes.io/projected/d2385ea2-646c-45e9-b8ab-53844944d549-kube-api-access-zmhzj\") pod \"test-model-dd1393-kserve-6cb4d97b45-hxfsf\" (UID: \"d2385ea2-646c-45e9-b8ab-53844944d549\") " pod="ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf"
Jul 29 07:03:43.279067 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:43.278745    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2385ea2-646c-45e9-b8ab-53844944d549-model-cache\") pod \"test-model-dd1393-kserve-6cb4d97b45-hxfsf\" (UID: \"d2385ea2-646c-45e9-b8ab-53844944d549\") " pod="ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf"
Jul 29 07:03:43.279176 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:43.279159    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2385ea2-646c-45e9-b8ab-53844944d549-tmp-dir\") pod \"test-model-dd1393-kserve-6cb4d97b45-hxfsf\" (UID: \"d2385ea2-646c-45e9-b8ab-53844944d549\") " pod="ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf"
Jul 29 07:03:43.279232 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:43.279167    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2385ea2-646c-45e9-b8ab-53844944d549-model-cache\") pod \"test-model-dd1393-kserve-6cb4d97b45-hxfsf\" (UID: \"d2385ea2-646c-45e9-b8ab-53844944d549\") " pod="ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf"
Jul 29 07:03:43.279276 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:43.279234    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2385ea2-646c-45e9-b8ab-53844944d549-home\") pod \"test-model-dd1393-kserve-6cb4d97b45-hxfsf\" (UID: \"d2385ea2-646c-45e9-b8ab-53844944d549\") " pod="ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf"
Jul 29 07:03:43.281043 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:43.281018    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2385ea2-646c-45e9-b8ab-53844944d549-dshm\") pod \"test-model-dd1393-kserve-6cb4d97b45-hxfsf\" (UID: \"d2385ea2-646c-45e9-b8ab-53844944d549\") " pod="ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf"
Jul 29 07:03:43.281333 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:43.281318    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2385ea2-646c-45e9-b8ab-53844944d549-tls-certs\") pod \"test-model-dd1393-kserve-6cb4d97b45-hxfsf\" (UID: \"d2385ea2-646c-45e9-b8ab-53844944d549\") " pod="ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf"
Jul 29 07:03:43.285473 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:43.285451    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmhzj\" (UniqueName: \"kubernetes.io/projected/d2385ea2-646c-45e9-b8ab-53844944d549-kube-api-access-zmhzj\") pod \"test-model-dd1393-kserve-6cb4d97b45-hxfsf\" (UID: \"d2385ea2-646c-45e9-b8ab-53844944d549\") " pod="ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf"
Jul 29 07:03:43.331731 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:43.331704    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf"
Jul 29 07:03:43.469355 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:43.469323    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf"]
Jul 29 07:03:43.473022 ip-10-0-129-223 kubenswrapper[2563]: W0729 07:03:43.472997    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2385ea2_646c_45e9_b8ab_53844944d549.slice/crio-d8094342c7ef9958664f08bec42a33ed54d367a47799775b71e940644c4167be WatchSource:0}: Error finding container d8094342c7ef9958664f08bec42a33ed54d367a47799775b71e940644c4167be: Status 404 returned error can't find the container with id d8094342c7ef9958664f08bec42a33ed54d367a47799775b71e940644c4167be
Jul 29 07:03:43.929336 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:43.929301    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf" event={"ID":"d2385ea2-646c-45e9-b8ab-53844944d549","Type":"ContainerStarted","Data":"dc11de59f1d1f0939df8aee6d6a304ead7b0ab162251045ca37718a5c5f68af5"}
Jul 29 07:03:43.929520 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:43.929344    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf" event={"ID":"d2385ea2-646c-45e9-b8ab-53844944d549","Type":"ContainerStarted","Data":"d8094342c7ef9958664f08bec42a33ed54d367a47799775b71e940644c4167be"}
Jul 29 07:03:43.944844 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:43.944798    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf" podStartSLOduration=1.944777932 podStartE2EDuration="1.944777932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:03:43.942986984 +0000 UTC m=+2467.020618777" watchObservedRunningTime="2026-07-29 07:03:43.944777932 +0000 UTC m=+2467.022409699"
Jul 29 07:03:53.332553 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:53.332522    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf"
Jul 29 07:03:53.333097 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:53.332569    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf"
Jul 29 07:03:53.345462 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:53.345436    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf"
Jul 29 07:03:53.990961 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:03:53.990931    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf"
Jul 29 07:05:09.896053 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:09.896018    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf"]
Jul 29 07:05:21.345262 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:21.345222    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4"]
Jul 29 07:05:21.345779 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:21.345493    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4" podUID="7cba4bfb-f9f7-4d76-ba82-91458918abb2" containerName="maas-api" containerID="cri-o://72ea8da183d360790ec66e73420e56842e1b99b508288c64fcfba5c10a9c0035" gracePeriod=30
Jul 29 07:05:21.594521 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:21.594499    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4"
Jul 29 07:05:21.605883 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:21.605808    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7nqk\" (UniqueName: \"kubernetes.io/projected/7cba4bfb-f9f7-4d76-ba82-91458918abb2-kube-api-access-k7nqk\") pod \"7cba4bfb-f9f7-4d76-ba82-91458918abb2\" (UID: \"7cba4bfb-f9f7-4d76-ba82-91458918abb2\") "
Jul 29 07:05:21.605994 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:21.605916    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cba4bfb-f9f7-4d76-ba82-91458918abb2-maas-api-tls\") pod \"7cba4bfb-f9f7-4d76-ba82-91458918abb2\" (UID: \"7cba4bfb-f9f7-4d76-ba82-91458918abb2\") "
Jul 29 07:05:21.605994 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:21.605966    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cba4bfb-f9f7-4d76-ba82-91458918abb2-openshift-service-ca\") pod \"7cba4bfb-f9f7-4d76-ba82-91458918abb2\" (UID: \"7cba4bfb-f9f7-4d76-ba82-91458918abb2\") "
Jul 29 07:05:21.606336 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:21.606298    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cba4bfb-f9f7-4d76-ba82-91458918abb2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7cba4bfb-f9f7-4d76-ba82-91458918abb2" (UID: "7cba4bfb-f9f7-4d76-ba82-91458918abb2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 07:05:21.607961 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:21.607935    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cba4bfb-f9f7-4d76-ba82-91458918abb2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7cba4bfb-f9f7-4d76-ba82-91458918abb2" (UID: "7cba4bfb-f9f7-4d76-ba82-91458918abb2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 07:05:21.608056 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:21.608021    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cba4bfb-f9f7-4d76-ba82-91458918abb2-kube-api-access-k7nqk" (OuterVolumeSpecName: "kube-api-access-k7nqk") pod "7cba4bfb-f9f7-4d76-ba82-91458918abb2" (UID: "7cba4bfb-f9f7-4d76-ba82-91458918abb2"). InnerVolumeSpecName "kube-api-access-k7nqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 07:05:21.707632 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:21.707600    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cba4bfb-f9f7-4d76-ba82-91458918abb2-maas-api-tls\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:05:21.707632 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:21.707626    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cba4bfb-f9f7-4d76-ba82-91458918abb2-openshift-service-ca\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:05:21.707632 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:21.707636    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7nqk\" (UniqueName: \"kubernetes.io/projected/7cba4bfb-f9f7-4d76-ba82-91458918abb2-kube-api-access-k7nqk\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:05:22.434378 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:22.434343    2563 generic.go:356] "Generic (PLEG): container finished" podID="7cba4bfb-f9f7-4d76-ba82-91458918abb2" containerID="72ea8da183d360790ec66e73420e56842e1b99b508288c64fcfba5c10a9c0035" exitCode=0
Jul 29 07:05:22.434848 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:22.434394    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4"
Jul 29 07:05:22.434848 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:22.434432    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4" event={"ID":"7cba4bfb-f9f7-4d76-ba82-91458918abb2","Type":"ContainerDied","Data":"72ea8da183d360790ec66e73420e56842e1b99b508288c64fcfba5c10a9c0035"}
Jul 29 07:05:22.434848 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:22.434525    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4" event={"ID":"7cba4bfb-f9f7-4d76-ba82-91458918abb2","Type":"ContainerDied","Data":"6aa1716be784f8286ebcaa2c09c68d700b82ee8dd4b9f66358babb5425a88175"}
Jul 29 07:05:22.434848 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:22.434556    2563 scope.go:117] "RemoveContainer" containerID="72ea8da183d360790ec66e73420e56842e1b99b508288c64fcfba5c10a9c0035"
Jul 29 07:05:22.446282 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:22.446267    2563 scope.go:117] "RemoveContainer" containerID="72ea8da183d360790ec66e73420e56842e1b99b508288c64fcfba5c10a9c0035"
Jul 29 07:05:22.446537 ip-10-0-129-223 kubenswrapper[2563]: E0729 07:05:22.446520    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72ea8da183d360790ec66e73420e56842e1b99b508288c64fcfba5c10a9c0035\": container with ID starting with 72ea8da183d360790ec66e73420e56842e1b99b508288c64fcfba5c10a9c0035 not found: ID does not exist" containerID="72ea8da183d360790ec66e73420e56842e1b99b508288c64fcfba5c10a9c0035"
Jul 29 07:05:22.446583 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:22.446545    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72ea8da183d360790ec66e73420e56842e1b99b508288c64fcfba5c10a9c0035"} err="failed to get container status \"72ea8da183d360790ec66e73420e56842e1b99b508288c64fcfba5c10a9c0035\": rpc error: code = NotFound desc = could not find container \"72ea8da183d360790ec66e73420e56842e1b99b508288c64fcfba5c10a9c0035\": container with ID starting with 72ea8da183d360790ec66e73420e56842e1b99b508288c64fcfba5c10a9c0035 not found: ID does not exist"
Jul 29 07:05:22.457473 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:22.457445    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4"]
Jul 29 07:05:22.461253 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:22.461233    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dd1393-66bf88d497-c4wz4"]
Jul 29 07:05:23.397870 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:23.397787    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cba4bfb-f9f7-4d76-ba82-91458918abb2" path="/var/lib/kubelet/pods/7cba4bfb-f9f7-4d76-ba82-91458918abb2/volumes"
Jul 29 07:05:24.915677 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:24.915638    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf" podUID="d2385ea2-646c-45e9-b8ab-53844944d549" containerName="main" containerID="cri-o://dc11de59f1d1f0939df8aee6d6a304ead7b0ab162251045ca37718a5c5f68af5" gracePeriod=45
Jul 29 07:05:25.170799 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:25.170737    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf"
Jul 29 07:05:25.238658 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:25.238629    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmhzj\" (UniqueName: \"kubernetes.io/projected/d2385ea2-646c-45e9-b8ab-53844944d549-kube-api-access-zmhzj\") pod \"d2385ea2-646c-45e9-b8ab-53844944d549\" (UID: \"d2385ea2-646c-45e9-b8ab-53844944d549\") "
Jul 29 07:05:25.238816 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:25.238684    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2385ea2-646c-45e9-b8ab-53844944d549-tls-certs\") pod \"d2385ea2-646c-45e9-b8ab-53844944d549\" (UID: \"d2385ea2-646c-45e9-b8ab-53844944d549\") "
Jul 29 07:05:25.238816 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:25.238713    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2385ea2-646c-45e9-b8ab-53844944d549-tmp-dir\") pod \"d2385ea2-646c-45e9-b8ab-53844944d549\" (UID: \"d2385ea2-646c-45e9-b8ab-53844944d549\") "
Jul 29 07:05:25.238816 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:25.238757    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2385ea2-646c-45e9-b8ab-53844944d549-dshm\") pod \"d2385ea2-646c-45e9-b8ab-53844944d549\" (UID: \"d2385ea2-646c-45e9-b8ab-53844944d549\") "
Jul 29 07:05:25.238816 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:25.238791    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2385ea2-646c-45e9-b8ab-53844944d549-home\") pod \"d2385ea2-646c-45e9-b8ab-53844944d549\" (UID: \"d2385ea2-646c-45e9-b8ab-53844944d549\") "
Jul 29 07:05:25.239070 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:25.238820    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2385ea2-646c-45e9-b8ab-53844944d549-model-cache\") pod \"d2385ea2-646c-45e9-b8ab-53844944d549\" (UID: \"d2385ea2-646c-45e9-b8ab-53844944d549\") "
Jul 29 07:05:25.239070 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:25.239033    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2385ea2-646c-45e9-b8ab-53844944d549-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d2385ea2-646c-45e9-b8ab-53844944d549" (UID: "d2385ea2-646c-45e9-b8ab-53844944d549"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 07:05:25.239178 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:25.239141    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2385ea2-646c-45e9-b8ab-53844944d549-model-cache" (OuterVolumeSpecName: "model-cache") pod "d2385ea2-646c-45e9-b8ab-53844944d549" (UID: "d2385ea2-646c-45e9-b8ab-53844944d549"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 07:05:25.239178 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:25.239149    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2385ea2-646c-45e9-b8ab-53844944d549-tmp-dir\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:05:25.239178 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:25.239154    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2385ea2-646c-45e9-b8ab-53844944d549-home" (OuterVolumeSpecName: "home") pod "d2385ea2-646c-45e9-b8ab-53844944d549" (UID: "d2385ea2-646c-45e9-b8ab-53844944d549"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 07:05:25.241174 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:25.241153    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2385ea2-646c-45e9-b8ab-53844944d549-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d2385ea2-646c-45e9-b8ab-53844944d549" (UID: "d2385ea2-646c-45e9-b8ab-53844944d549"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 07:05:25.241459 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:25.241441    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2385ea2-646c-45e9-b8ab-53844944d549-dshm" (OuterVolumeSpecName: "dshm") pod "d2385ea2-646c-45e9-b8ab-53844944d549" (UID: "d2385ea2-646c-45e9-b8ab-53844944d549"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 07:05:25.241526 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:25.241464    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2385ea2-646c-45e9-b8ab-53844944d549-kube-api-access-zmhzj" (OuterVolumeSpecName: "kube-api-access-zmhzj") pod "d2385ea2-646c-45e9-b8ab-53844944d549" (UID: "d2385ea2-646c-45e9-b8ab-53844944d549"). InnerVolumeSpecName "kube-api-access-zmhzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 07:05:25.340496 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:25.340465    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2385ea2-646c-45e9-b8ab-53844944d549-dshm\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:05:25.340496 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:25.340492    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2385ea2-646c-45e9-b8ab-53844944d549-home\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:05:25.340685 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:25.340504    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2385ea2-646c-45e9-b8ab-53844944d549-model-cache\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:05:25.340685 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:25.340520    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmhzj\" (UniqueName: \"kubernetes.io/projected/d2385ea2-646c-45e9-b8ab-53844944d549-kube-api-access-zmhzj\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:05:25.340685 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:25.340534    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2385ea2-646c-45e9-b8ab-53844944d549-tls-certs\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:05:25.451889 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:25.451799    2563 generic.go:356] "Generic (PLEG): container finished" podID="d2385ea2-646c-45e9-b8ab-53844944d549" containerID="dc11de59f1d1f0939df8aee6d6a304ead7b0ab162251045ca37718a5c5f68af5" exitCode=0
Jul 29 07:05:25.451889 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:25.451841    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf"
Jul 29 07:05:25.452076 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:25.451884    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf" event={"ID":"d2385ea2-646c-45e9-b8ab-53844944d549","Type":"ContainerDied","Data":"dc11de59f1d1f0939df8aee6d6a304ead7b0ab162251045ca37718a5c5f68af5"}
Jul 29 07:05:25.452076 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:25.451929    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf" event={"ID":"d2385ea2-646c-45e9-b8ab-53844944d549","Type":"ContainerDied","Data":"d8094342c7ef9958664f08bec42a33ed54d367a47799775b71e940644c4167be"}
Jul 29 07:05:25.452076 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:25.451951    2563 scope.go:117] "RemoveContainer" containerID="dc11de59f1d1f0939df8aee6d6a304ead7b0ab162251045ca37718a5c5f68af5"
Jul 29 07:05:25.460974 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:25.460945    2563 scope.go:117] "RemoveContainer" containerID="dc11de59f1d1f0939df8aee6d6a304ead7b0ab162251045ca37718a5c5f68af5"
Jul 29 07:05:25.461218 ip-10-0-129-223 kubenswrapper[2563]: E0729 07:05:25.461199    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc11de59f1d1f0939df8aee6d6a304ead7b0ab162251045ca37718a5c5f68af5\": container with ID starting with dc11de59f1d1f0939df8aee6d6a304ead7b0ab162251045ca37718a5c5f68af5 not found: ID does not exist" containerID="dc11de59f1d1f0939df8aee6d6a304ead7b0ab162251045ca37718a5c5f68af5"
Jul 29 07:05:25.461280 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:25.461225    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc11de59f1d1f0939df8aee6d6a304ead7b0ab162251045ca37718a5c5f68af5"} err="failed to get container status \"dc11de59f1d1f0939df8aee6d6a304ead7b0ab162251045ca37718a5c5f68af5\": rpc error: code = NotFound desc = could not find container \"dc11de59f1d1f0939df8aee6d6a304ead7b0ab162251045ca37718a5c5f68af5\": container with ID starting with dc11de59f1d1f0939df8aee6d6a304ead7b0ab162251045ca37718a5c5f68af5 not found: ID does not exist"
Jul 29 07:05:25.466413 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:25.466393    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf"]
Jul 29 07:05:25.470328 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:25.470310    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-dd1393/test-model-dd1393-kserve-6cb4d97b45-hxfsf"]
Jul 29 07:05:27.397322 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:27.397294    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2385ea2-646c-45e9-b8ab-53844944d549" path="/var/lib/kubelet/pods/d2385ea2-646c-45e9-b8ab-53844944d549/volumes"
Jul 29 07:05:28.057364 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:28.057330    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9aa496-cvwt2"]
Jul 29 07:05:28.057789 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:28.057776    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cba4bfb-f9f7-4d76-ba82-91458918abb2" containerName="maas-api"
Jul 29 07:05:28.057834 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:28.057791    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cba4bfb-f9f7-4d76-ba82-91458918abb2" containerName="maas-api"
Jul 29 07:05:28.057834 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:28.057806    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2385ea2-646c-45e9-b8ab-53844944d549" containerName="main"
Jul 29 07:05:28.057834 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:28.057816    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2385ea2-646c-45e9-b8ab-53844944d549" containerName="main"
Jul 29 07:05:28.057945 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:28.057905    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2385ea2-646c-45e9-b8ab-53844944d549" containerName="main"
Jul 29 07:05:28.057945 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:28.057915    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cba4bfb-f9f7-4d76-ba82-91458918abb2" containerName="maas-api"
Jul 29 07:05:28.062322 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:28.062306    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9aa496-cvwt2"
Jul 29 07:05:28.064978 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:28.064481    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bj47g\""
Jul 29 07:05:28.066939 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:28.066840    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9aa496-cvwt2"]
Jul 29 07:05:28.165804 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:28.165725    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64f4b837-e8b8-423f-9fc1-15463bb2814a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9aa496-cvwt2\" (UID: \"64f4b837-e8b8-423f-9fc1-15463bb2814a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9aa496-cvwt2"
Jul 29 07:05:28.267243 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:28.267206    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64f4b837-e8b8-423f-9fc1-15463bb2814a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9aa496-cvwt2\" (UID: \"64f4b837-e8b8-423f-9fc1-15463bb2814a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9aa496-cvwt2"
Jul 29 07:05:28.267913 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:28.267896    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64f4b837-e8b8-423f-9fc1-15463bb2814a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9aa496-cvwt2\" (UID: \"64f4b837-e8b8-423f-9fc1-15463bb2814a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9aa496-cvwt2"
Jul 29 07:05:28.374739 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:28.374704    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9aa496-cvwt2"
Jul 29 07:05:28.516037 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:28.516013    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9aa496-cvwt2"]
Jul 29 07:05:28.518840 ip-10-0-129-223 kubenswrapper[2563]: W0729 07:05:28.518809    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f4b837_e8b8_423f_9fc1_15463bb2814a.slice/crio-948bc9792904a5e743363fd9d3089ca0a27e4a6a44bfbfd9bbeb7564c2063256 WatchSource:0}: Error finding container 948bc9792904a5e743363fd9d3089ca0a27e4a6a44bfbfd9bbeb7564c2063256: Status 404 returned error can't find the container with id 948bc9792904a5e743363fd9d3089ca0a27e4a6a44bfbfd9bbeb7564c2063256
Jul 29 07:05:29.476551 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:29.476458    2563 generic.go:356] "Generic (PLEG): container finished" podID="64f4b837-e8b8-423f-9fc1-15463bb2814a" containerID="72b575d964078e19fd05c3d9190f7d72c061f25a611765c85f8ab08e86ecb1df" exitCode=0
Jul 29 07:05:29.476701 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:29.476563    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9aa496-cvwt2" event={"ID":"64f4b837-e8b8-423f-9fc1-15463bb2814a","Type":"ContainerDied","Data":"72b575d964078e19fd05c3d9190f7d72c061f25a611765c85f8ab08e86ecb1df"}
Jul 29 07:05:29.476701 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:29.476608    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9aa496-cvwt2" event={"ID":"64f4b837-e8b8-423f-9fc1-15463bb2814a","Type":"ContainerStarted","Data":"948bc9792904a5e743363fd9d3089ca0a27e4a6a44bfbfd9bbeb7564c2063256"}
Jul 29 07:05:30.613999 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:30.613974    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9aa496-cvwt2"
Jul 29 07:05:30.690619 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:30.690591    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64f4b837-e8b8-423f-9fc1-15463bb2814a-maas-api-service-ca\") pod \"64f4b837-e8b8-423f-9fc1-15463bb2814a\" (UID: \"64f4b837-e8b8-423f-9fc1-15463bb2814a\") "
Jul 29 07:05:30.690901 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:30.690845    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64f4b837-e8b8-423f-9fc1-15463bb2814a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "64f4b837-e8b8-423f-9fc1-15463bb2814a" (UID: "64f4b837-e8b8-423f-9fc1-15463bb2814a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 07:05:30.791387 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:30.791317    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64f4b837-e8b8-423f-9fc1-15463bb2814a-maas-api-service-ca\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:05:31.488047 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:31.488020    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9aa496-cvwt2"
Jul 29 07:05:31.488197 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:31.488066    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9aa496-cvwt2" event={"ID":"64f4b837-e8b8-423f-9fc1-15463bb2814a","Type":"ContainerDied","Data":"948bc9792904a5e743363fd9d3089ca0a27e4a6a44bfbfd9bbeb7564c2063256"}
Jul 29 07:05:31.488197 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:31.488114    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="948bc9792904a5e743363fd9d3089ca0a27e4a6a44bfbfd9bbeb7564c2063256"
Jul 29 07:05:38.460512 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:38.460479    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h"]
Jul 29 07:05:38.460980 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:38.460693    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h" podUID="50a1eba3-c5cd-4c97-ae5e-c359e8bd6622" containerName="maas-api" containerID="cri-o://9b06d79cb69694ddde966b9589e7127e4cc8f02adb749dc1055cea3dc58afbda" gracePeriod=30
Jul 29 07:05:38.646339 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:38.646303    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9aa496-cvwt2"]
Jul 29 07:05:38.649533 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:38.649499    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9aa496-cvwt2"]
Jul 29 07:05:38.706098 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:38.706077    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h"
Jul 29 07:05:38.760292 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:38.760207    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a1eba3-c5cd-4c97-ae5e-c359e8bd6622-openshift-service-ca\") pod \"50a1eba3-c5cd-4c97-ae5e-c359e8bd6622\" (UID: \"50a1eba3-c5cd-4c97-ae5e-c359e8bd6622\") "
Jul 29 07:05:38.760292 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:38.760278    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8s5w\" (UniqueName: \"kubernetes.io/projected/50a1eba3-c5cd-4c97-ae5e-c359e8bd6622-kube-api-access-j8s5w\") pod \"50a1eba3-c5cd-4c97-ae5e-c359e8bd6622\" (UID: \"50a1eba3-c5cd-4c97-ae5e-c359e8bd6622\") "
Jul 29 07:05:38.760501 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:38.760349    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50a1eba3-c5cd-4c97-ae5e-c359e8bd6622-maas-api-tls\") pod \"50a1eba3-c5cd-4c97-ae5e-c359e8bd6622\" (UID: \"50a1eba3-c5cd-4c97-ae5e-c359e8bd6622\") "
Jul 29 07:05:38.760567 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:38.760546    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50a1eba3-c5cd-4c97-ae5e-c359e8bd6622-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "50a1eba3-c5cd-4c97-ae5e-c359e8bd6622" (UID: "50a1eba3-c5cd-4c97-ae5e-c359e8bd6622"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 07:05:38.760641 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:38.760627    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a1eba3-c5cd-4c97-ae5e-c359e8bd6622-openshift-service-ca\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:05:38.762621 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:38.762601    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a1eba3-c5cd-4c97-ae5e-c359e8bd6622-kube-api-access-j8s5w" (OuterVolumeSpecName: "kube-api-access-j8s5w") pod "50a1eba3-c5cd-4c97-ae5e-c359e8bd6622" (UID: "50a1eba3-c5cd-4c97-ae5e-c359e8bd6622"). InnerVolumeSpecName "kube-api-access-j8s5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 07:05:38.762691 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:38.762661    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50a1eba3-c5cd-4c97-ae5e-c359e8bd6622-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "50a1eba3-c5cd-4c97-ae5e-c359e8bd6622" (UID: "50a1eba3-c5cd-4c97-ae5e-c359e8bd6622"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 07:05:38.861573 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:38.861543    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8s5w\" (UniqueName: \"kubernetes.io/projected/50a1eba3-c5cd-4c97-ae5e-c359e8bd6622-kube-api-access-j8s5w\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:05:38.861573 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:38.861571    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50a1eba3-c5cd-4c97-ae5e-c359e8bd6622-maas-api-tls\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:05:39.398697 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:39.398665    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64f4b837-e8b8-423f-9fc1-15463bb2814a" path="/var/lib/kubelet/pods/64f4b837-e8b8-423f-9fc1-15463bb2814a/volumes"
Jul 29 07:05:39.528149 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:39.528116    2563 generic.go:356] "Generic (PLEG): container finished" podID="50a1eba3-c5cd-4c97-ae5e-c359e8bd6622" containerID="9b06d79cb69694ddde966b9589e7127e4cc8f02adb749dc1055cea3dc58afbda" exitCode=0
Jul 29 07:05:39.528560 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:39.528153    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h" event={"ID":"50a1eba3-c5cd-4c97-ae5e-c359e8bd6622","Type":"ContainerDied","Data":"9b06d79cb69694ddde966b9589e7127e4cc8f02adb749dc1055cea3dc58afbda"}
Jul 29 07:05:39.528560 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:39.528163    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h"
Jul 29 07:05:39.528560 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:39.528212    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h" event={"ID":"50a1eba3-c5cd-4c97-ae5e-c359e8bd6622","Type":"ContainerDied","Data":"39d7580346b7cde7885fdda30f21bdc3cfa6160b9e5ba51ebd92c03cd819de76"}
Jul 29 07:05:39.528560 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:39.528236    2563 scope.go:117] "RemoveContainer" containerID="9b06d79cb69694ddde966b9589e7127e4cc8f02adb749dc1055cea3dc58afbda"
Jul 29 07:05:39.537612 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:39.537595    2563 scope.go:117] "RemoveContainer" containerID="9b06d79cb69694ddde966b9589e7127e4cc8f02adb749dc1055cea3dc58afbda"
Jul 29 07:05:39.537906 ip-10-0-129-223 kubenswrapper[2563]: E0729 07:05:39.537885    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b06d79cb69694ddde966b9589e7127e4cc8f02adb749dc1055cea3dc58afbda\": container with ID starting with 9b06d79cb69694ddde966b9589e7127e4cc8f02adb749dc1055cea3dc58afbda not found: ID does not exist" containerID="9b06d79cb69694ddde966b9589e7127e4cc8f02adb749dc1055cea3dc58afbda"
Jul 29 07:05:39.537979 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:39.537913    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b06d79cb69694ddde966b9589e7127e4cc8f02adb749dc1055cea3dc58afbda"} err="failed to get container status \"9b06d79cb69694ddde966b9589e7127e4cc8f02adb749dc1055cea3dc58afbda\": rpc error: code = NotFound desc = could not find container \"9b06d79cb69694ddde966b9589e7127e4cc8f02adb749dc1055cea3dc58afbda\": container with ID starting with 9b06d79cb69694ddde966b9589e7127e4cc8f02adb749dc1055cea3dc58afbda not found: ID does not exist"
Jul 29 07:05:39.543598 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:39.543578    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h"]
Jul 29 07:05:39.545437 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:39.545420    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9aa496-6b6747d5c8-nq87h"]
Jul 29 07:05:41.397947 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:41.397902    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50a1eba3-c5cd-4c97-ae5e-c359e8bd6622" path="/var/lib/kubelet/pods/50a1eba3-c5cd-4c97-ae5e-c359e8bd6622/volumes"
Jul 29 07:05:48.696779 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:48.696745    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2"]
Jul 29 07:05:48.697246 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:48.697231    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64f4b837-e8b8-423f-9fc1-15463bb2814a" containerName="revoke-keys"
Jul 29 07:05:48.697246 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:48.697247    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="64f4b837-e8b8-423f-9fc1-15463bb2814a" containerName="revoke-keys"
Jul 29 07:05:48.697333 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:48.697256    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50a1eba3-c5cd-4c97-ae5e-c359e8bd6622" containerName="maas-api"
Jul 29 07:05:48.697333 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:48.697261    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="50a1eba3-c5cd-4c97-ae5e-c359e8bd6622" containerName="maas-api"
Jul 29 07:05:48.697400 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:48.697345    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="50a1eba3-c5cd-4c97-ae5e-c359e8bd6622" containerName="maas-api"
Jul 29 07:05:48.697400 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:48.697357    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="64f4b837-e8b8-423f-9fc1-15463bb2814a" containerName="revoke-keys"
Jul 29 07:05:48.700456 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:48.700436    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2"
Jul 29 07:05:48.702698 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:48.702676    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-4b144e-serving-cert\""
Jul 29 07:05:48.709933 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:48.709912    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2"]
Jul 29 07:05:48.857117 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:48.857081    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aabfd710-9cba-4b72-b675-3bf085a07d0c-openshift-service-ca\") pod \"maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2\" (UID: \"aabfd710-9cba-4b72-b675-3bf085a07d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2"
Jul 29 07:05:48.857322 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:48.857124    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfv2z\" (UniqueName: \"kubernetes.io/projected/aabfd710-9cba-4b72-b675-3bf085a07d0c-kube-api-access-qfv2z\") pod \"maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2\" (UID: \"aabfd710-9cba-4b72-b675-3bf085a07d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2"
Jul 29 07:05:48.857322 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:48.857247    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aabfd710-9cba-4b72-b675-3bf085a07d0c-maas-api-tls\") pod \"maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2\" (UID: \"aabfd710-9cba-4b72-b675-3bf085a07d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2"
Jul 29 07:05:48.958470 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:48.958376    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aabfd710-9cba-4b72-b675-3bf085a07d0c-openshift-service-ca\") pod \"maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2\" (UID: \"aabfd710-9cba-4b72-b675-3bf085a07d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2"
Jul 29 07:05:48.958470 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:48.958415    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfv2z\" (UniqueName: \"kubernetes.io/projected/aabfd710-9cba-4b72-b675-3bf085a07d0c-kube-api-access-qfv2z\") pod \"maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2\" (UID: \"aabfd710-9cba-4b72-b675-3bf085a07d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2"
Jul 29 07:05:48.958470 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:48.958467    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aabfd710-9cba-4b72-b675-3bf085a07d0c-maas-api-tls\") pod \"maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2\" (UID: \"aabfd710-9cba-4b72-b675-3bf085a07d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2"
Jul 29 07:05:48.959140 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:48.959114    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aabfd710-9cba-4b72-b675-3bf085a07d0c-openshift-service-ca\") pod \"maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2\" (UID: \"aabfd710-9cba-4b72-b675-3bf085a07d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2"
Jul 29 07:05:48.960986 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:48.960964    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aabfd710-9cba-4b72-b675-3bf085a07d0c-maas-api-tls\") pod \"maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2\" (UID: \"aabfd710-9cba-4b72-b675-3bf085a07d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2"
Jul 29 07:05:48.966178 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:48.966153    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfv2z\" (UniqueName: \"kubernetes.io/projected/aabfd710-9cba-4b72-b675-3bf085a07d0c-kube-api-access-qfv2z\") pod \"maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2\" (UID: \"aabfd710-9cba-4b72-b675-3bf085a07d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2"
Jul 29 07:05:49.014070 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:49.014046    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2"
Jul 29 07:05:49.152670 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:49.152644    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2"]
Jul 29 07:05:49.155902 ip-10-0-129-223 kubenswrapper[2563]: W0729 07:05:49.155868    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaabfd710_9cba_4b72_b675_3bf085a07d0c.slice/crio-abb3f4186b3a1ea85387af40892d004c2cd063eef330496e7a3f54627f759c76 WatchSource:0}: Error finding container abb3f4186b3a1ea85387af40892d004c2cd063eef330496e7a3f54627f759c76: Status 404 returned error can't find the container with id abb3f4186b3a1ea85387af40892d004c2cd063eef330496e7a3f54627f759c76
Jul 29 07:05:49.157762 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:49.157740    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 07:05:49.588205 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:49.588181    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2" event={"ID":"aabfd710-9cba-4b72-b675-3bf085a07d0c","Type":"ContainerStarted","Data":"abb3f4186b3a1ea85387af40892d004c2cd063eef330496e7a3f54627f759c76"}
Jul 29 07:05:50.595462 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:50.595429    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2" event={"ID":"aabfd710-9cba-4b72-b675-3bf085a07d0c","Type":"ContainerStarted","Data":"7625f9a72e43c0d3e7bf96a1d1c21e611493b2009c1adcef2d185276a2c2c4cb"}
Jul 29 07:05:50.595878 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:50.595490    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2"
Jul 29 07:05:50.612868 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:50.612812    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2" podStartSLOduration=2.203627072 podStartE2EDuration="2.612797447s" totalImagesPullingTime="409.170375ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:05:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:05:50.61094973 +0000 UTC m=+2593.688581497" watchObservedRunningTime="2026-07-29 07:05:50.612797447 +0000 UTC m=+2593.690429277"
Jul 29 07:05:56.606552 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:05:56.606521    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2"
Jul 29 07:06:16.761784 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:06:16.761706    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln"]
Jul 29 07:06:16.765673 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:06:16.765651    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln"
Jul 29 07:06:16.768143 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:06:16.768122    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-e87d49-serving-cert\""
Jul 29 07:06:16.776555 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:06:16.776443    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln"]
Jul 29 07:06:16.819529 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:06:16.819493    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb0a3f16-2ae5-466e-80e8-3f8f5f641e58-openshift-service-ca\") pod \"maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln\" (UID: \"cb0a3f16-2ae5-466e-80e8-3f8f5f641e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln"
Jul 29 07:06:16.819669 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:06:16.819536    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft9rl\" (UniqueName: \"kubernetes.io/projected/cb0a3f16-2ae5-466e-80e8-3f8f5f641e58-kube-api-access-ft9rl\") pod \"maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln\" (UID: \"cb0a3f16-2ae5-466e-80e8-3f8f5f641e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln"
Jul 29 07:06:16.819669 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:06:16.819605    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb0a3f16-2ae5-466e-80e8-3f8f5f641e58-maas-api-tls\") pod \"maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln\" (UID: \"cb0a3f16-2ae5-466e-80e8-3f8f5f641e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln"
Jul 29 07:06:16.920571 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:06:16.920534    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb0a3f16-2ae5-466e-80e8-3f8f5f641e58-openshift-service-ca\") pod \"maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln\" (UID: \"cb0a3f16-2ae5-466e-80e8-3f8f5f641e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln"
Jul 29 07:06:16.920741 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:06:16.920578    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft9rl\" (UniqueName: \"kubernetes.io/projected/cb0a3f16-2ae5-466e-80e8-3f8f5f641e58-kube-api-access-ft9rl\") pod \"maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln\" (UID: \"cb0a3f16-2ae5-466e-80e8-3f8f5f641e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln"
Jul 29 07:06:16.920741 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:06:16.920643    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb0a3f16-2ae5-466e-80e8-3f8f5f641e58-maas-api-tls\") pod \"maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln\" (UID: \"cb0a3f16-2ae5-466e-80e8-3f8f5f641e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln"
Jul 29 07:06:16.920901 ip-10-0-129-223 kubenswrapper[2563]: E0729 07:06:16.920792    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-e87d49-serving-cert: secret "maas-api-e2e-api-b-e87d49-serving-cert" not found
Jul 29 07:06:16.920966 ip-10-0-129-223 kubenswrapper[2563]: E0729 07:06:16.920903    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb0a3f16-2ae5-466e-80e8-3f8f5f641e58-maas-api-tls podName:cb0a3f16-2ae5-466e-80e8-3f8f5f641e58 nodeName:}" failed. No retries permitted until 2026-07-29 07:06:17.420881309 +0000 UTC m=+2620.498513068 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cb0a3f16-2ae5-466e-80e8-3f8f5f641e58-maas-api-tls") pod "maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln" (UID: "cb0a3f16-2ae5-466e-80e8-3f8f5f641e58") : secret "maas-api-e2e-api-b-e87d49-serving-cert" not found
Jul 29 07:06:16.921205 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:06:16.921183    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb0a3f16-2ae5-466e-80e8-3f8f5f641e58-openshift-service-ca\") pod \"maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln\" (UID: \"cb0a3f16-2ae5-466e-80e8-3f8f5f641e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln"
Jul 29 07:06:16.928658 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:06:16.928635    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft9rl\" (UniqueName: \"kubernetes.io/projected/cb0a3f16-2ae5-466e-80e8-3f8f5f641e58-kube-api-access-ft9rl\") pod \"maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln\" (UID: \"cb0a3f16-2ae5-466e-80e8-3f8f5f641e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln"
Jul 29 07:06:17.425267 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:06:17.425228    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb0a3f16-2ae5-466e-80e8-3f8f5f641e58-maas-api-tls\") pod \"maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln\" (UID: \"cb0a3f16-2ae5-466e-80e8-3f8f5f641e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln"
Jul 29 07:06:17.428161 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:06:17.428134    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb0a3f16-2ae5-466e-80e8-3f8f5f641e58-maas-api-tls\") pod \"maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln\" (UID: \"cb0a3f16-2ae5-466e-80e8-3f8f5f641e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln"
Jul 29 07:06:17.680751 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:06:17.680661    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln"
Jul 29 07:06:17.835119 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:06:17.835091    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln"]
Jul 29 07:06:18.743875 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:06:18.743810    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln" event={"ID":"cb0a3f16-2ae5-466e-80e8-3f8f5f641e58","Type":"ContainerStarted","Data":"3988d66b094440ef0412469ee82c12a6fbc35fa898923a0d0f1dfca04bb62f6b"}
Jul 29 07:06:18.744074 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:06:18.743905    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln"
Jul 29 07:06:18.744074 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:06:18.743924    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln" event={"ID":"cb0a3f16-2ae5-466e-80e8-3f8f5f641e58","Type":"ContainerStarted","Data":"39438f17d9f6b48dda033617cb54a6da188a3ec0be058cc469131f099ba694f4"}
Jul 29 07:06:18.759534 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:06:18.759486    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln" podStartSLOduration=2.401779427 podStartE2EDuration="2.759471108s" totalImagesPullingTime="357.691681ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:06:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:06:18.757589253 +0000 UTC m=+2621.835221012" watchObservedRunningTime="2026-07-29 07:06:18.759471108 +0000 UTC m=+2621.837102875"
Jul 29 07:06:24.755035 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:06:24.755004    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln"
Jul 29 07:06:37.637509 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:06:37.637461    2563 scope.go:117] "RemoveContainer" containerID="256c4d16c63217be818eeba641e5c77acf1b99dd3232351d08c471d906d59526"
Jul 29 07:06:59.912800 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:06:59.912759    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2"]
Jul 29 07:06:59.913519 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:06:59.913113    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2" podUID="aabfd710-9cba-4b72-b675-3bf085a07d0c" containerName="maas-api" containerID="cri-o://7625f9a72e43c0d3e7bf96a1d1c21e611493b2009c1adcef2d185276a2c2c4cb" gracePeriod=30
Jul 29 07:07:00.163729 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:00.163674    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2"
Jul 29 07:07:00.229018 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:00.228988    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aabfd710-9cba-4b72-b675-3bf085a07d0c-openshift-service-ca\") pod \"aabfd710-9cba-4b72-b675-3bf085a07d0c\" (UID: \"aabfd710-9cba-4b72-b675-3bf085a07d0c\") "
Jul 29 07:07:00.229018 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:00.229020    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aabfd710-9cba-4b72-b675-3bf085a07d0c-maas-api-tls\") pod \"aabfd710-9cba-4b72-b675-3bf085a07d0c\" (UID: \"aabfd710-9cba-4b72-b675-3bf085a07d0c\") "
Jul 29 07:07:00.229194 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:00.229072    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfv2z\" (UniqueName: \"kubernetes.io/projected/aabfd710-9cba-4b72-b675-3bf085a07d0c-kube-api-access-qfv2z\") pod \"aabfd710-9cba-4b72-b675-3bf085a07d0c\" (UID: \"aabfd710-9cba-4b72-b675-3bf085a07d0c\") "
Jul 29 07:07:00.229397 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:00.229364    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aabfd710-9cba-4b72-b675-3bf085a07d0c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aabfd710-9cba-4b72-b675-3bf085a07d0c" (UID: "aabfd710-9cba-4b72-b675-3bf085a07d0c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 07:07:00.231311 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:00.231287    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aabfd710-9cba-4b72-b675-3bf085a07d0c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aabfd710-9cba-4b72-b675-3bf085a07d0c" (UID: "aabfd710-9cba-4b72-b675-3bf085a07d0c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 07:07:00.231385 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:00.231332    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aabfd710-9cba-4b72-b675-3bf085a07d0c-kube-api-access-qfv2z" (OuterVolumeSpecName: "kube-api-access-qfv2z") pod "aabfd710-9cba-4b72-b675-3bf085a07d0c" (UID: "aabfd710-9cba-4b72-b675-3bf085a07d0c"). InnerVolumeSpecName "kube-api-access-qfv2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 07:07:00.330469 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:00.330433    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aabfd710-9cba-4b72-b675-3bf085a07d0c-openshift-service-ca\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:07:00.330469 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:00.330469    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aabfd710-9cba-4b72-b675-3bf085a07d0c-maas-api-tls\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:07:00.330643 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:00.330487    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfv2z\" (UniqueName: \"kubernetes.io/projected/aabfd710-9cba-4b72-b675-3bf085a07d0c-kube-api-access-qfv2z\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:07:00.957783 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:00.957751    2563 generic.go:356] "Generic (PLEG): container finished" podID="aabfd710-9cba-4b72-b675-3bf085a07d0c" containerID="7625f9a72e43c0d3e7bf96a1d1c21e611493b2009c1adcef2d185276a2c2c4cb" exitCode=0
Jul 29 07:07:00.958278 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:00.957801    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2"
Jul 29 07:07:00.958278 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:00.957841    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2" event={"ID":"aabfd710-9cba-4b72-b675-3bf085a07d0c","Type":"ContainerDied","Data":"7625f9a72e43c0d3e7bf96a1d1c21e611493b2009c1adcef2d185276a2c2c4cb"}
Jul 29 07:07:00.958278 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:00.957923    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2" event={"ID":"aabfd710-9cba-4b72-b675-3bf085a07d0c","Type":"ContainerDied","Data":"abb3f4186b3a1ea85387af40892d004c2cd063eef330496e7a3f54627f759c76"}
Jul 29 07:07:00.958278 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:00.957945    2563 scope.go:117] "RemoveContainer" containerID="7625f9a72e43c0d3e7bf96a1d1c21e611493b2009c1adcef2d185276a2c2c4cb"
Jul 29 07:07:00.967114 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:00.967100    2563 scope.go:117] "RemoveContainer" containerID="7625f9a72e43c0d3e7bf96a1d1c21e611493b2009c1adcef2d185276a2c2c4cb"
Jul 29 07:07:00.967366 ip-10-0-129-223 kubenswrapper[2563]: E0729 07:07:00.967347    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7625f9a72e43c0d3e7bf96a1d1c21e611493b2009c1adcef2d185276a2c2c4cb\": container with ID starting with 7625f9a72e43c0d3e7bf96a1d1c21e611493b2009c1adcef2d185276a2c2c4cb not found: ID does not exist" containerID="7625f9a72e43c0d3e7bf96a1d1c21e611493b2009c1adcef2d185276a2c2c4cb"
Jul 29 07:07:00.967419 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:00.967373    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7625f9a72e43c0d3e7bf96a1d1c21e611493b2009c1adcef2d185276a2c2c4cb"} err="failed to get container status \"7625f9a72e43c0d3e7bf96a1d1c21e611493b2009c1adcef2d185276a2c2c4cb\": rpc error: code = NotFound desc = could not find container \"7625f9a72e43c0d3e7bf96a1d1c21e611493b2009c1adcef2d185276a2c2c4cb\": container with ID starting with 7625f9a72e43c0d3e7bf96a1d1c21e611493b2009c1adcef2d185276a2c2c4cb not found: ID does not exist"
Jul 29 07:07:00.977208 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:00.977189    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2"]
Jul 29 07:07:00.981171 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:00.981152    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4b144e-849ff464f6-v6vl2"]
Jul 29 07:07:01.397155 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:01.397120    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aabfd710-9cba-4b72-b675-3bf085a07d0c" path="/var/lib/kubelet/pods/aabfd710-9cba-4b72-b675-3bf085a07d0c/volumes"
Jul 29 07:07:22.201340 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:22.201310    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln"]
Jul 29 07:07:22.201802 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:22.201593    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln" podUID="cb0a3f16-2ae5-466e-80e8-3f8f5f641e58" containerName="maas-api" containerID="cri-o://3988d66b094440ef0412469ee82c12a6fbc35fa898923a0d0f1dfca04bb62f6b" gracePeriod=30
Jul 29 07:07:22.450098 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:22.450073    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln"
Jul 29 07:07:22.634872 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:22.634824    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb0a3f16-2ae5-466e-80e8-3f8f5f641e58-openshift-service-ca\") pod \"cb0a3f16-2ae5-466e-80e8-3f8f5f641e58\" (UID: \"cb0a3f16-2ae5-466e-80e8-3f8f5f641e58\") "
Jul 29 07:07:22.635034 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:22.634889    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb0a3f16-2ae5-466e-80e8-3f8f5f641e58-maas-api-tls\") pod \"cb0a3f16-2ae5-466e-80e8-3f8f5f641e58\" (UID: \"cb0a3f16-2ae5-466e-80e8-3f8f5f641e58\") "
Jul 29 07:07:22.635034 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:22.634951    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ft9rl\" (UniqueName: \"kubernetes.io/projected/cb0a3f16-2ae5-466e-80e8-3f8f5f641e58-kube-api-access-ft9rl\") pod \"cb0a3f16-2ae5-466e-80e8-3f8f5f641e58\" (UID: \"cb0a3f16-2ae5-466e-80e8-3f8f5f641e58\") "
Jul 29 07:07:22.635200 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:22.635178    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb0a3f16-2ae5-466e-80e8-3f8f5f641e58-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cb0a3f16-2ae5-466e-80e8-3f8f5f641e58" (UID: "cb0a3f16-2ae5-466e-80e8-3f8f5f641e58"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 07:07:22.635303 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:22.635280    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb0a3f16-2ae5-466e-80e8-3f8f5f641e58-openshift-service-ca\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:07:22.637171 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:22.637145    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb0a3f16-2ae5-466e-80e8-3f8f5f641e58-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cb0a3f16-2ae5-466e-80e8-3f8f5f641e58" (UID: "cb0a3f16-2ae5-466e-80e8-3f8f5f641e58"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 07:07:22.637256 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:22.637182    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb0a3f16-2ae5-466e-80e8-3f8f5f641e58-kube-api-access-ft9rl" (OuterVolumeSpecName: "kube-api-access-ft9rl") pod "cb0a3f16-2ae5-466e-80e8-3f8f5f641e58" (UID: "cb0a3f16-2ae5-466e-80e8-3f8f5f641e58"). InnerVolumeSpecName "kube-api-access-ft9rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 07:07:22.736287 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:22.736256    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb0a3f16-2ae5-466e-80e8-3f8f5f641e58-maas-api-tls\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:07:22.736287 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:22.736282    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft9rl\" (UniqueName: \"kubernetes.io/projected/cb0a3f16-2ae5-466e-80e8-3f8f5f641e58-kube-api-access-ft9rl\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:07:23.078940 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:23.078912    2563 generic.go:356] "Generic (PLEG): container finished" podID="cb0a3f16-2ae5-466e-80e8-3f8f5f641e58" containerID="3988d66b094440ef0412469ee82c12a6fbc35fa898923a0d0f1dfca04bb62f6b" exitCode=0
Jul 29 07:07:23.079070 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:23.078957    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln"
Jul 29 07:07:23.079070 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:23.078997    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln" event={"ID":"cb0a3f16-2ae5-466e-80e8-3f8f5f641e58","Type":"ContainerDied","Data":"3988d66b094440ef0412469ee82c12a6fbc35fa898923a0d0f1dfca04bb62f6b"}
Jul 29 07:07:23.079070 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:23.079050    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln" event={"ID":"cb0a3f16-2ae5-466e-80e8-3f8f5f641e58","Type":"ContainerDied","Data":"39438f17d9f6b48dda033617cb54a6da188a3ec0be058cc469131f099ba694f4"}
Jul 29 07:07:23.079212 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:23.079072    2563 scope.go:117] "RemoveContainer" containerID="3988d66b094440ef0412469ee82c12a6fbc35fa898923a0d0f1dfca04bb62f6b"
Jul 29 07:07:23.088361 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:23.088347    2563 scope.go:117] "RemoveContainer" containerID="3988d66b094440ef0412469ee82c12a6fbc35fa898923a0d0f1dfca04bb62f6b"
Jul 29 07:07:23.088608 ip-10-0-129-223 kubenswrapper[2563]: E0729 07:07:23.088591    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3988d66b094440ef0412469ee82c12a6fbc35fa898923a0d0f1dfca04bb62f6b\": container with ID starting with 3988d66b094440ef0412469ee82c12a6fbc35fa898923a0d0f1dfca04bb62f6b not found: ID does not exist" containerID="3988d66b094440ef0412469ee82c12a6fbc35fa898923a0d0f1dfca04bb62f6b"
Jul 29 07:07:23.088658 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:23.088617    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3988d66b094440ef0412469ee82c12a6fbc35fa898923a0d0f1dfca04bb62f6b"} err="failed to get container status \"3988d66b094440ef0412469ee82c12a6fbc35fa898923a0d0f1dfca04bb62f6b\": rpc error: code = NotFound desc = could not find container \"3988d66b094440ef0412469ee82c12a6fbc35fa898923a0d0f1dfca04bb62f6b\": container with ID starting with 3988d66b094440ef0412469ee82c12a6fbc35fa898923a0d0f1dfca04bb62f6b not found: ID does not exist"
Jul 29 07:07:23.099817 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:23.099798    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln"]
Jul 29 07:07:23.103505 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:23.103485    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e87d49-74bcbb66c8-lh5ln"]
Jul 29 07:07:23.397225 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:23.397153    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb0a3f16-2ae5-466e-80e8-3f8f5f641e58" path="/var/lib/kubelet/pods/cb0a3f16-2ae5-466e-80e8-3f8f5f641e58/volumes"
Jul 29 07:07:33.121026 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:33.120816    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z"]
Jul 29 07:07:33.121819 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:33.121794    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb0a3f16-2ae5-466e-80e8-3f8f5f641e58" containerName="maas-api"
Jul 29 07:07:33.121927 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:33.121823    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb0a3f16-2ae5-466e-80e8-3f8f5f641e58" containerName="maas-api"
Jul 29 07:07:33.126530 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:33.121847    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aabfd710-9cba-4b72-b675-3bf085a07d0c" containerName="maas-api"
Jul 29 07:07:33.126691 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:33.126673    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aabfd710-9cba-4b72-b675-3bf085a07d0c" containerName="maas-api"
Jul 29 07:07:33.126870 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:33.126837    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb0a3f16-2ae5-466e-80e8-3f8f5f641e58" containerName="maas-api"
Jul 29 07:07:33.126952 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:33.126885    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="aabfd710-9cba-4b72-b675-3bf085a07d0c" containerName="maas-api"
Jul 29 07:07:33.131757 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:33.131731    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z"
Jul 29 07:07:33.132315 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:33.132269    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z"]
Jul 29 07:07:33.133692 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:33.133650    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-50e693-serving-cert\""
Jul 29 07:07:33.219167 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:33.219137    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq8tc\" (UniqueName: \"kubernetes.io/projected/7b711f5d-bd42-47d1-a020-1e24dd0d4f2b-kube-api-access-jq8tc\") pod \"maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z\" (UID: \"7b711f5d-bd42-47d1-a020-1e24dd0d4f2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z"
Jul 29 07:07:33.219167 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:33.219172    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b711f5d-bd42-47d1-a020-1e24dd0d4f2b-maas-api-tls\") pod \"maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z\" (UID: \"7b711f5d-bd42-47d1-a020-1e24dd0d4f2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z"
Jul 29 07:07:33.219380 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:33.219239    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b711f5d-bd42-47d1-a020-1e24dd0d4f2b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z\" (UID: \"7b711f5d-bd42-47d1-a020-1e24dd0d4f2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z"
Jul 29 07:07:33.320359 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:33.320329    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq8tc\" (UniqueName: \"kubernetes.io/projected/7b711f5d-bd42-47d1-a020-1e24dd0d4f2b-kube-api-access-jq8tc\") pod \"maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z\" (UID: \"7b711f5d-bd42-47d1-a020-1e24dd0d4f2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z"
Jul 29 07:07:33.320359 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:33.320361    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b711f5d-bd42-47d1-a020-1e24dd0d4f2b-maas-api-tls\") pod \"maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z\" (UID: \"7b711f5d-bd42-47d1-a020-1e24dd0d4f2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z"
Jul 29 07:07:33.320547 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:33.320385    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b711f5d-bd42-47d1-a020-1e24dd0d4f2b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z\" (UID: \"7b711f5d-bd42-47d1-a020-1e24dd0d4f2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z"
Jul 29 07:07:33.320547 ip-10-0-129-223 kubenswrapper[2563]: E0729 07:07:33.320484    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-50e693-serving-cert: secret "maas-api-e2e-shared-a-50e693-serving-cert" not found
Jul 29 07:07:33.320614 ip-10-0-129-223 kubenswrapper[2563]: E0729 07:07:33.320555    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b711f5d-bd42-47d1-a020-1e24dd0d4f2b-maas-api-tls podName:7b711f5d-bd42-47d1-a020-1e24dd0d4f2b nodeName:}" failed. No retries permitted until 2026-07-29 07:07:33.820534974 +0000 UTC m=+2696.898166718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7b711f5d-bd42-47d1-a020-1e24dd0d4f2b-maas-api-tls") pod "maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z" (UID: "7b711f5d-bd42-47d1-a020-1e24dd0d4f2b") : secret "maas-api-e2e-shared-a-50e693-serving-cert" not found
Jul 29 07:07:33.321081 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:33.321064    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b711f5d-bd42-47d1-a020-1e24dd0d4f2b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z\" (UID: \"7b711f5d-bd42-47d1-a020-1e24dd0d4f2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z"
Jul 29 07:07:33.329056 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:33.329040    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq8tc\" (UniqueName: \"kubernetes.io/projected/7b711f5d-bd42-47d1-a020-1e24dd0d4f2b-kube-api-access-jq8tc\") pod \"maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z\" (UID: \"7b711f5d-bd42-47d1-a020-1e24dd0d4f2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z"
Jul 29 07:07:33.825422 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:33.825391    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b711f5d-bd42-47d1-a020-1e24dd0d4f2b-maas-api-tls\") pod \"maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z\" (UID: \"7b711f5d-bd42-47d1-a020-1e24dd0d4f2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z"
Jul 29 07:07:33.827987 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:33.827964    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b711f5d-bd42-47d1-a020-1e24dd0d4f2b-maas-api-tls\") pod \"maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z\" (UID: \"7b711f5d-bd42-47d1-a020-1e24dd0d4f2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z"
Jul 29 07:07:33.891430 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:33.891400    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-50e693-55cff4bdd8-2mtt4"]
Jul 29 07:07:33.895308 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:33.895293    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-50e693-55cff4bdd8-2mtt4"
Jul 29 07:07:33.897536 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:33.897514    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-50e693\""
Jul 29 07:07:33.897656 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:33.897636    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-50e693-dockercfg-ljpkl\""
Jul 29 07:07:33.910792 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:33.910770    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-50e693-55cff4bdd8-2mtt4"]
Jul 29 07:07:33.926202 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:33.926175    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/102edb4c-50b3-47d1-ac60-71e41b1df80d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-50e693-55cff4bdd8-2mtt4\" (UID: \"102edb4c-50b3-47d1-ac60-71e41b1df80d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-50e693-55cff4bdd8-2mtt4"
Jul 29 07:07:33.926327 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:33.926248    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9b5n\" (UniqueName: \"kubernetes.io/projected/102edb4c-50b3-47d1-ac60-71e41b1df80d-kube-api-access-b9b5n\") pod \"payload-pre-processing-e2e-shared-a-50e693-55cff4bdd8-2mtt4\" (UID: \"102edb4c-50b3-47d1-ac60-71e41b1df80d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-50e693-55cff4bdd8-2mtt4"
Jul 29 07:07:34.026924 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:34.026887    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/102edb4c-50b3-47d1-ac60-71e41b1df80d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-50e693-55cff4bdd8-2mtt4\" (UID: \"102edb4c-50b3-47d1-ac60-71e41b1df80d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-50e693-55cff4bdd8-2mtt4"
Jul 29 07:07:34.027126 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:34.026973    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9b5n\" (UniqueName: \"kubernetes.io/projected/102edb4c-50b3-47d1-ac60-71e41b1df80d-kube-api-access-b9b5n\") pod \"payload-pre-processing-e2e-shared-a-50e693-55cff4bdd8-2mtt4\" (UID: \"102edb4c-50b3-47d1-ac60-71e41b1df80d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-50e693-55cff4bdd8-2mtt4"
Jul 29 07:07:34.027643 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:34.027623    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/102edb4c-50b3-47d1-ac60-71e41b1df80d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-50e693-55cff4bdd8-2mtt4\" (UID: \"102edb4c-50b3-47d1-ac60-71e41b1df80d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-50e693-55cff4bdd8-2mtt4"
Jul 29 07:07:34.035247 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:34.035218    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9b5n\" (UniqueName: \"kubernetes.io/projected/102edb4c-50b3-47d1-ac60-71e41b1df80d-kube-api-access-b9b5n\") pod \"payload-pre-processing-e2e-shared-a-50e693-55cff4bdd8-2mtt4\" (UID: \"102edb4c-50b3-47d1-ac60-71e41b1df80d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-50e693-55cff4bdd8-2mtt4"
Jul 29 07:07:34.042949 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:34.042922    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z"
Jul 29 07:07:34.176805 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:34.176737    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z"]
Jul 29 07:07:34.180427 ip-10-0-129-223 kubenswrapper[2563]: W0729 07:07:34.180386    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b711f5d_bd42_47d1_a020_1e24dd0d4f2b.slice/crio-c9275b3a1b76487fdd996dc271571d120c0ba79460cba697879aa5f7ccb28fde WatchSource:0}: Error finding container c9275b3a1b76487fdd996dc271571d120c0ba79460cba697879aa5f7ccb28fde: Status 404 returned error can't find the container with id c9275b3a1b76487fdd996dc271571d120c0ba79460cba697879aa5f7ccb28fde
Jul 29 07:07:34.205806 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:34.205786    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-50e693-55cff4bdd8-2mtt4"
Jul 29 07:07:34.349107 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:34.349080    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-50e693-55cff4bdd8-2mtt4"]
Jul 29 07:07:34.350974 ip-10-0-129-223 kubenswrapper[2563]: W0729 07:07:34.350951    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod102edb4c_50b3_47d1_ac60_71e41b1df80d.slice/crio-967ee6bf6dc72d7255268b8b109519adb8b09d9d7c90f122e144d2926ad9036d WatchSource:0}: Error finding container 967ee6bf6dc72d7255268b8b109519adb8b09d9d7c90f122e144d2926ad9036d: Status 404 returned error can't find the container with id 967ee6bf6dc72d7255268b8b109519adb8b09d9d7c90f122e144d2926ad9036d
Jul 29 07:07:35.150182 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:35.150141    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z" event={"ID":"7b711f5d-bd42-47d1-a020-1e24dd0d4f2b","Type":"ContainerStarted","Data":"2dbf40434aed9140e0899c5447173bf53820c0ce91ce081a793ba96297ba0090"}
Jul 29 07:07:35.150392 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:35.150303    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z"
Jul 29 07:07:35.150392 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:35.150349    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z" event={"ID":"7b711f5d-bd42-47d1-a020-1e24dd0d4f2b","Type":"ContainerStarted","Data":"c9275b3a1b76487fdd996dc271571d120c0ba79460cba697879aa5f7ccb28fde"}
Jul 29 07:07:35.151648 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:35.151626    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-50e693-55cff4bdd8-2mtt4" event={"ID":"102edb4c-50b3-47d1-ac60-71e41b1df80d","Type":"ContainerStarted","Data":"23501ca20b3c97fd08f12d67e9df473901b465fbda6060692f35c2ff9e6e2fa2"}
Jul 29 07:07:35.151750 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:35.151662    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-50e693-55cff4bdd8-2mtt4" event={"ID":"102edb4c-50b3-47d1-ac60-71e41b1df80d","Type":"ContainerStarted","Data":"967ee6bf6dc72d7255268b8b109519adb8b09d9d7c90f122e144d2926ad9036d"}
Jul 29 07:07:35.151814 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:35.151763    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-50e693-55cff4bdd8-2mtt4"
Jul 29 07:07:35.166907 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:35.166845    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z" podStartSLOduration=1.735957671 podStartE2EDuration="2.166832346s" totalImagesPullingTime="430.874675ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:07:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:07:35.165051321 +0000 UTC m=+2698.242683090" watchObservedRunningTime="2026-07-29 07:07:35.166832346 +0000 UTC m=+2698.244464121"
Jul 29 07:07:35.179259 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:35.179218    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-50e693-55cff4bdd8-2mtt4" podStartSLOduration=2.179206558 podStartE2EDuration="2.179206558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:07:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:07:35.177765602 +0000 UTC m=+2698.255397369" watchObservedRunningTime="2026-07-29 07:07:35.179206558 +0000 UTC m=+2698.256838324"
Jul 29 07:07:41.162485 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:41.162454    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z"
Jul 29 07:07:46.159367 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:07:46.159339    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-50e693-55cff4bdd8-2mtt4"
Jul 29 07:08:02.698416 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:08:02.698377    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc"]
Jul 29 07:08:02.702053 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:08:02.702029    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc"
Jul 29 07:08:02.705207 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:08:02.705185    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-1e6a9b-serving-cert\""
Jul 29 07:08:02.725829 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:08:02.725806    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc"]
Jul 29 07:08:02.799298 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:08:02.799269    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/559a6d58-0e54-4400-8d31-2e46fe248f96-maas-api-tls\") pod \"maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc\" (UID: \"559a6d58-0e54-4400-8d31-2e46fe248f96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc"
Jul 29 07:08:02.799298 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:08:02.799300    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5ksc\" (UniqueName: \"kubernetes.io/projected/559a6d58-0e54-4400-8d31-2e46fe248f96-kube-api-access-l5ksc\") pod \"maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc\" (UID: \"559a6d58-0e54-4400-8d31-2e46fe248f96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc"
Jul 29 07:08:02.799487 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:08:02.799325    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/559a6d58-0e54-4400-8d31-2e46fe248f96-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc\" (UID: \"559a6d58-0e54-4400-8d31-2e46fe248f96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc"
Jul 29 07:08:02.900151 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:08:02.900119    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/559a6d58-0e54-4400-8d31-2e46fe248f96-maas-api-tls\") pod \"maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc\" (UID: \"559a6d58-0e54-4400-8d31-2e46fe248f96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc"
Jul 29 07:08:02.900151 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:08:02.900151    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5ksc\" (UniqueName: \"kubernetes.io/projected/559a6d58-0e54-4400-8d31-2e46fe248f96-kube-api-access-l5ksc\") pod \"maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc\" (UID: \"559a6d58-0e54-4400-8d31-2e46fe248f96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc"
Jul 29 07:08:02.900362 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:08:02.900188    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/559a6d58-0e54-4400-8d31-2e46fe248f96-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc\" (UID: \"559a6d58-0e54-4400-8d31-2e46fe248f96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc"
Jul 29 07:08:02.900362 ip-10-0-129-223 kubenswrapper[2563]: E0729 07:08:02.900275    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-1e6a9b-serving-cert: secret "maas-api-e2e-shared-b-1e6a9b-serving-cert" not found
Jul 29 07:08:02.900362 ip-10-0-129-223 kubenswrapper[2563]: E0729 07:08:02.900359    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/559a6d58-0e54-4400-8d31-2e46fe248f96-maas-api-tls podName:559a6d58-0e54-4400-8d31-2e46fe248f96 nodeName:}" failed. No retries permitted until 2026-07-29 07:08:03.400342349 +0000 UTC m=+2726.477974095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/559a6d58-0e54-4400-8d31-2e46fe248f96-maas-api-tls") pod "maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc" (UID: "559a6d58-0e54-4400-8d31-2e46fe248f96") : secret "maas-api-e2e-shared-b-1e6a9b-serving-cert" not found
Jul 29 07:08:02.900893 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:08:02.900873    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/559a6d58-0e54-4400-8d31-2e46fe248f96-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc\" (UID: \"559a6d58-0e54-4400-8d31-2e46fe248f96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc"
Jul 29 07:08:02.911389 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:08:02.911357    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5ksc\" (UniqueName: \"kubernetes.io/projected/559a6d58-0e54-4400-8d31-2e46fe248f96-kube-api-access-l5ksc\") pod \"maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc\" (UID: \"559a6d58-0e54-4400-8d31-2e46fe248f96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc"
Jul 29 07:08:03.404182 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:08:03.404154    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/559a6d58-0e54-4400-8d31-2e46fe248f96-maas-api-tls\") pod \"maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc\" (UID: \"559a6d58-0e54-4400-8d31-2e46fe248f96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc"
Jul 29 07:08:03.404383 ip-10-0-129-223 kubenswrapper[2563]: E0729 07:08:03.404312    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-1e6a9b-serving-cert: secret "maas-api-e2e-shared-b-1e6a9b-serving-cert" not found
Jul 29 07:08:03.404383 ip-10-0-129-223 kubenswrapper[2563]: E0729 07:08:03.404370    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/559a6d58-0e54-4400-8d31-2e46fe248f96-maas-api-tls podName:559a6d58-0e54-4400-8d31-2e46fe248f96 nodeName:}" failed. No retries permitted until 2026-07-29 07:08:04.40435532 +0000 UTC m=+2727.481987065 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/559a6d58-0e54-4400-8d31-2e46fe248f96-maas-api-tls") pod "maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc" (UID: "559a6d58-0e54-4400-8d31-2e46fe248f96") : secret "maas-api-e2e-shared-b-1e6a9b-serving-cert" not found
Jul 29 07:08:04.416150 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:08:04.416116    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/559a6d58-0e54-4400-8d31-2e46fe248f96-maas-api-tls\") pod \"maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc\" (UID: \"559a6d58-0e54-4400-8d31-2e46fe248f96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc"
Jul 29 07:08:04.418812 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:08:04.418788    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/559a6d58-0e54-4400-8d31-2e46fe248f96-maas-api-tls\") pod \"maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc\" (UID: \"559a6d58-0e54-4400-8d31-2e46fe248f96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc"
Jul 29 07:08:04.513822 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:08:04.513790    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc"
Jul 29 07:08:04.665175 ip-10-0-129-223 kubenswrapper[2563]: W0729 07:08:04.665145    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod559a6d58_0e54_4400_8d31_2e46fe248f96.slice/crio-69054019796d547b6cd52b1f3f7805b10b3aa57df38e58fe43732a4ef50dd07e WatchSource:0}: Error finding container 69054019796d547b6cd52b1f3f7805b10b3aa57df38e58fe43732a4ef50dd07e: Status 404 returned error can't find the container with id 69054019796d547b6cd52b1f3f7805b10b3aa57df38e58fe43732a4ef50dd07e
Jul 29 07:08:04.666538 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:08:04.666480    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc"]
Jul 29 07:08:05.316246 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:08:05.316224    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc" event={"ID":"559a6d58-0e54-4400-8d31-2e46fe248f96","Type":"ContainerStarted","Data":"562ea3d117841cc928cd813b00dee5ff69e321146e0f2c11dc173d5a69caca3d"}
Jul 29 07:08:05.316354 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:08:05.316268    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc" event={"ID":"559a6d58-0e54-4400-8d31-2e46fe248f96","Type":"ContainerStarted","Data":"69054019796d547b6cd52b1f3f7805b10b3aa57df38e58fe43732a4ef50dd07e"}
Jul 29 07:08:05.316400 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:08:05.316365    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc"
Jul 29 07:08:05.353639 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:08:05.353586    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc" podStartSLOduration=2.773899791 podStartE2EDuration="3.353569891s" totalImagesPullingTime="579.6701ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:08:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:08:05.352520394 +0000 UTC m=+2728.430152161" watchObservedRunningTime="2026-07-29 07:08:05.353569891 +0000 UTC m=+2728.431201658"
Jul 29 07:08:11.328410 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:08:11.328383    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc"
Jul 29 07:11:37.705006 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:11:37.704976    2563 scope.go:117] "RemoveContainer" containerID="72b575d964078e19fd05c3d9190f7d72c061f25a611765c85f8ab08e86ecb1df"
Jul 29 07:15:00.137723 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:15:00.137648    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1e6a9b-29755155-cd2rg"]
Jul 29 07:15:00.141440 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:15:00.141417    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1e6a9b-29755155-cd2rg"
Jul 29 07:15:00.143619 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:15:00.143587    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bj47g\""
Jul 29 07:15:00.149681 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:15:00.149656    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1e6a9b-29755155-cd2rg"]
Jul 29 07:15:00.154358 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:15:00.154334    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1e6a9b-29755155-cd2rg"
Jul 29 07:15:00.289316 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:15:00.289289    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1e6a9b-29755155-cd2rg"]
Jul 29 07:15:00.291411 ip-10-0-129-223 kubenswrapper[2563]: W0729 07:15:00.291380    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabbe197d_fd92_48c2_9694_1f2388864613.slice/crio-41c6721e2af9a129abee832dc82ba2bcd8c83cadc96e0cd3ac888e82eb86ffaa WatchSource:0}: Error finding container 41c6721e2af9a129abee832dc82ba2bcd8c83cadc96e0cd3ac888e82eb86ffaa: Status 404 returned error can't find the container with id 41c6721e2af9a129abee832dc82ba2bcd8c83cadc96e0cd3ac888e82eb86ffaa
Jul 29 07:15:00.448903 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:15:00.448845    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1e6a9b-29755155-cd2rg" event={"ID":"abbe197d-fd92-48c2-9694-1f2388864613","Type":"ContainerStarted","Data":"41c6721e2af9a129abee832dc82ba2bcd8c83cadc96e0cd3ac888e82eb86ffaa"}
Jul 29 07:15:01.456080 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:15:01.456040    2563 generic.go:356] "Generic (PLEG): container finished" podID="abbe197d-fd92-48c2-9694-1f2388864613" containerID="3240fd04a6b7e41a554d2c742bc6fab627c04d9c9aba63b3222695f0274d2294" exitCode=0
Jul 29 07:15:01.456498 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:15:01.456125    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1e6a9b-29755155-cd2rg" event={"ID":"abbe197d-fd92-48c2-9694-1f2388864613","Type":"ContainerDied","Data":"3240fd04a6b7e41a554d2c742bc6fab627c04d9c9aba63b3222695f0274d2294"}
Jul 29 07:15:02.593996 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:15:02.593971    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1e6a9b-29755155-cd2rg"
Jul 29 07:15:03.466749 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:15:03.466714    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1e6a9b-29755155-cd2rg" event={"ID":"abbe197d-fd92-48c2-9694-1f2388864613","Type":"ContainerDied","Data":"41c6721e2af9a129abee832dc82ba2bcd8c83cadc96e0cd3ac888e82eb86ffaa"}
Jul 29 07:15:03.466749 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:15:03.466733    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1e6a9b-29755155-cd2rg"
Jul 29 07:15:03.466972 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:15:03.466767    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41c6721e2af9a129abee832dc82ba2bcd8c83cadc96e0cd3ac888e82eb86ffaa"
Jul 29 07:17:46.350138 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:46.350104    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7822-140659763015856"]
Jul 29 07:17:46.350610 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:46.350499    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abbe197d-fd92-48c2-9694-1f2388864613" containerName="cleanup"
Jul 29 07:17:46.350610 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:46.350509    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="abbe197d-fd92-48c2-9694-1f2388864613" containerName="cleanup"
Jul 29 07:17:46.350610 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:46.350589    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="abbe197d-fd92-48c2-9694-1f2388864613" containerName="cleanup"
Jul 29 07:17:46.353814 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:46.353797    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7822-140659763015856"
Jul 29 07:17:46.356054 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:46.356030    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8cb9r\""
Jul 29 07:17:46.359231 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:46.359210    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7822-140659763015856"]
Jul 29 07:17:46.458692 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:46.458655    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqjbh\" (UniqueName: \"kubernetes.io/projected/eb1bc999-96fc-4de7-a661-6f594c3d69b9-kube-api-access-kqjbh\") pod \"test-curl-7822-140659763015856\" (UID: \"eb1bc999-96fc-4de7-a661-6f594c3d69b9\") " pod="opendatahub/test-curl-7822-140659763015856"
Jul 29 07:17:46.559894 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:46.559864    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqjbh\" (UniqueName: \"kubernetes.io/projected/eb1bc999-96fc-4de7-a661-6f594c3d69b9-kube-api-access-kqjbh\") pod \"test-curl-7822-140659763015856\" (UID: \"eb1bc999-96fc-4de7-a661-6f594c3d69b9\") " pod="opendatahub/test-curl-7822-140659763015856"
Jul 29 07:17:46.570183 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:46.570152    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqjbh\" (UniqueName: \"kubernetes.io/projected/eb1bc999-96fc-4de7-a661-6f594c3d69b9-kube-api-access-kqjbh\") pod \"test-curl-7822-140659763015856\" (UID: \"eb1bc999-96fc-4de7-a661-6f594c3d69b9\") " pod="opendatahub/test-curl-7822-140659763015856"
Jul 29 07:17:46.664952 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:46.664877    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7822-140659763015856"
Jul 29 07:17:46.799804 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:46.799769    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7822-140659763015856"]
Jul 29 07:17:46.800912 ip-10-0-129-223 kubenswrapper[2563]: W0729 07:17:46.800881    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb1bc999_96fc_4de7_a661_6f594c3d69b9.slice/crio-196d8c04c96acfbf582ad976a4826133a7a6c220c48c5cf9779de97e0508a2c4 WatchSource:0}: Error finding container 196d8c04c96acfbf582ad976a4826133a7a6c220c48c5cf9779de97e0508a2c4: Status 404 returned error can't find the container with id 196d8c04c96acfbf582ad976a4826133a7a6c220c48c5cf9779de97e0508a2c4
Jul 29 07:17:46.802641 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:46.802621    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 07:17:47.311146 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:47.311109    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7822-140659763015856" event={"ID":"eb1bc999-96fc-4de7-a661-6f594c3d69b9","Type":"ContainerStarted","Data":"196d8c04c96acfbf582ad976a4826133a7a6c220c48c5cf9779de97e0508a2c4"}
Jul 29 07:17:48.317787 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:48.317748    2563 generic.go:356] "Generic (PLEG): container finished" podID="eb1bc999-96fc-4de7-a661-6f594c3d69b9" containerID="1febdcd21b1bbb27098b4204609c4786781714cf4077c79fc14529f8ebabf374" exitCode=0
Jul 29 07:17:48.318203 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:48.317796    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7822-140659763015856" event={"ID":"eb1bc999-96fc-4de7-a661-6f594c3d69b9","Type":"ContainerDied","Data":"1febdcd21b1bbb27098b4204609c4786781714cf4077c79fc14529f8ebabf374"}
Jul 29 07:17:49.461034 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:49.461000    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7822-140659763015856"
Jul 29 07:17:49.487083 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:49.487061    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqjbh\" (UniqueName: \"kubernetes.io/projected/eb1bc999-96fc-4de7-a661-6f594c3d69b9-kube-api-access-kqjbh\") pod \"eb1bc999-96fc-4de7-a661-6f594c3d69b9\" (UID: \"eb1bc999-96fc-4de7-a661-6f594c3d69b9\") "
Jul 29 07:17:49.489164 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:49.489135    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb1bc999-96fc-4de7-a661-6f594c3d69b9-kube-api-access-kqjbh" (OuterVolumeSpecName: "kube-api-access-kqjbh") pod "eb1bc999-96fc-4de7-a661-6f594c3d69b9" (UID: "eb1bc999-96fc-4de7-a661-6f594c3d69b9"). InnerVolumeSpecName "kube-api-access-kqjbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 07:17:49.496389 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:49.496369    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7822-140659763015856_eb1bc999-96fc-4de7-a661-6f594c3d69b9/test-curl-7822-140659763015856/0.log"
Jul 29 07:17:49.520178 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:49.520146    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7822-140659763015856"]
Jul 29 07:17:49.525036 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:49.525010    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7822-140659763015856"]
Jul 29 07:17:49.588562 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:49.588488    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqjbh\" (UniqueName: \"kubernetes.io/projected/eb1bc999-96fc-4de7-a661-6f594c3d69b9-kube-api-access-kqjbh\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:17:49.661444 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:49.661415    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7822-140659763017136"]
Jul 29 07:17:49.661909 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:49.661894    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb1bc999-96fc-4de7-a661-6f594c3d69b9" containerName="test-curl-7822-140659763015856"
Jul 29 07:17:49.661951 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:49.661912    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb1bc999-96fc-4de7-a661-6f594c3d69b9" containerName="test-curl-7822-140659763015856"
Jul 29 07:17:49.661989 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:49.661976    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb1bc999-96fc-4de7-a661-6f594c3d69b9" containerName="test-curl-7822-140659763015856"
Jul 29 07:17:49.664477 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:49.664464    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7822-140659763017136"
Jul 29 07:17:49.668024 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:49.668000    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7822-140659763017136"]
Jul 29 07:17:49.689267 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:49.689243    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r8l9\" (UniqueName: \"kubernetes.io/projected/7382fbbc-bbd9-471e-80e6-e66a7a9a9615-kube-api-access-2r8l9\") pod \"test-curl-7822-140659763017136\" (UID: \"7382fbbc-bbd9-471e-80e6-e66a7a9a9615\") " pod="opendatahub/test-curl-7822-140659763017136"
Jul 29 07:17:49.790193 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:49.790164    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r8l9\" (UniqueName: \"kubernetes.io/projected/7382fbbc-bbd9-471e-80e6-e66a7a9a9615-kube-api-access-2r8l9\") pod \"test-curl-7822-140659763017136\" (UID: \"7382fbbc-bbd9-471e-80e6-e66a7a9a9615\") " pod="opendatahub/test-curl-7822-140659763017136"
Jul 29 07:17:49.798044 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:49.798011    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r8l9\" (UniqueName: \"kubernetes.io/projected/7382fbbc-bbd9-471e-80e6-e66a7a9a9615-kube-api-access-2r8l9\") pod \"test-curl-7822-140659763017136\" (UID: \"7382fbbc-bbd9-471e-80e6-e66a7a9a9615\") " pod="opendatahub/test-curl-7822-140659763017136"
Jul 29 07:17:49.976008 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:49.975921    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7822-140659763017136"
Jul 29 07:17:50.109218 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:50.109192    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7822-140659763017136"]
Jul 29 07:17:50.111731 ip-10-0-129-223 kubenswrapper[2563]: W0729 07:17:50.111697    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7382fbbc_bbd9_471e_80e6_e66a7a9a9615.slice/crio-97dd5bb518dd4434c70ec81f6ca3b500da379058a16daba06e2fe9ff03663ff1 WatchSource:0}: Error finding container 97dd5bb518dd4434c70ec81f6ca3b500da379058a16daba06e2fe9ff03663ff1: Status 404 returned error can't find the container with id 97dd5bb518dd4434c70ec81f6ca3b500da379058a16daba06e2fe9ff03663ff1
Jul 29 07:17:50.333523 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:50.333493    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7822-140659763015856"
Jul 29 07:17:50.333695 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:50.333528    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="196d8c04c96acfbf582ad976a4826133a7a6c220c48c5cf9779de97e0508a2c4"
Jul 29 07:17:50.334720 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:50.334696    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7822-140659763017136" event={"ID":"7382fbbc-bbd9-471e-80e6-e66a7a9a9615","Type":"ContainerStarted","Data":"97dd5bb518dd4434c70ec81f6ca3b500da379058a16daba06e2fe9ff03663ff1"}
Jul 29 07:17:50.344073 ip-10-0-129-223 kubenswrapper[2563]: E0729 07:17:50.344049    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7822-140659763015856\" is forbidden: User \"system:node:ip-10-0-129-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-223.ec2.internal' and this object" podUID="eb1bc999-96fc-4de7-a661-6f594c3d69b9" pod="opendatahub/test-curl-7822-140659763015856"
Jul 29 07:17:51.341442 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:51.341413    2563 generic.go:356] "Generic (PLEG): container finished" podID="7382fbbc-bbd9-471e-80e6-e66a7a9a9615" containerID="3b6be84daa8ba5f308ed39a15629c29eea3bb2681a71db352f5270f1ed6a1891" exitCode=0
Jul 29 07:17:51.341812 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:51.341502    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7822-140659763017136" event={"ID":"7382fbbc-bbd9-471e-80e6-e66a7a9a9615","Type":"ContainerDied","Data":"3b6be84daa8ba5f308ed39a15629c29eea3bb2681a71db352f5270f1ed6a1891"}
Jul 29 07:17:51.355325 ip-10-0-129-223 kubenswrapper[2563]: E0729 07:17:51.355294    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7822-140659763015856\" is forbidden: User \"system:node:ip-10-0-129-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-223.ec2.internal' and this object" podUID="eb1bc999-96fc-4de7-a661-6f594c3d69b9" pod="opendatahub/test-curl-7822-140659763015856"
Jul 29 07:17:51.397447 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:51.397419    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb1bc999-96fc-4de7-a661-6f594c3d69b9" path="/var/lib/kubelet/pods/eb1bc999-96fc-4de7-a661-6f594c3d69b9/volumes"
Jul 29 07:17:52.484449 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:52.484428    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7822-140659763017136"
Jul 29 07:17:52.502636 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:52.502612    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7822-140659763017136_7382fbbc-bbd9-471e-80e6-e66a7a9a9615/test-curl-7822-140659763017136/0.log"
Jul 29 07:17:52.518897 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:52.518848    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2r8l9\" (UniqueName: \"kubernetes.io/projected/7382fbbc-bbd9-471e-80e6-e66a7a9a9615-kube-api-access-2r8l9\") pod \"7382fbbc-bbd9-471e-80e6-e66a7a9a9615\" (UID: \"7382fbbc-bbd9-471e-80e6-e66a7a9a9615\") "
Jul 29 07:17:52.521026 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:52.520998    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7382fbbc-bbd9-471e-80e6-e66a7a9a9615-kube-api-access-2r8l9" (OuterVolumeSpecName: "kube-api-access-2r8l9") pod "7382fbbc-bbd9-471e-80e6-e66a7a9a9615" (UID: "7382fbbc-bbd9-471e-80e6-e66a7a9a9615"). InnerVolumeSpecName "kube-api-access-2r8l9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 07:17:52.525477 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:52.525452    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7822-140659763017136"]
Jul 29 07:17:52.529137 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:52.529118    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7822-140659763017136"]
Jul 29 07:17:52.620218 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:52.620150    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r8l9\" (UniqueName: \"kubernetes.io/projected/7382fbbc-bbd9-471e-80e6-e66a7a9a9615-kube-api-access-2r8l9\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:17:53.352973 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:53.352945    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7822-140659763017136"
Jul 29 07:17:53.353152 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:53.352944    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97dd5bb518dd4434c70ec81f6ca3b500da379058a16daba06e2fe9ff03663ff1"
Jul 29 07:17:53.361939 ip-10-0-129-223 kubenswrapper[2563]: E0729 07:17:53.361913    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7822-140659763017136\" is forbidden: User \"system:node:ip-10-0-129-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-223.ec2.internal' and this object" podUID="7382fbbc-bbd9-471e-80e6-e66a7a9a9615" pod="opendatahub/test-curl-7822-140659763017136"
Jul 29 07:17:53.398151 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:17:53.398119    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7382fbbc-bbd9-471e-80e6-e66a7a9a9615" path="/var/lib/kubelet/pods/7382fbbc-bbd9-471e-80e6-e66a7a9a9615/volumes"
Jul 29 07:18:09.173741 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:09.173642    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab086-664f494947-7v674"]
Jul 29 07:18:09.174309 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:09.174237    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7382fbbc-bbd9-471e-80e6-e66a7a9a9615" containerName="test-curl-7822-140659763017136"
Jul 29 07:18:09.174309 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:09.174259    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7382fbbc-bbd9-471e-80e6-e66a7a9a9615" containerName="test-curl-7822-140659763017136"
Jul 29 07:18:09.174393 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:09.174381    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7382fbbc-bbd9-471e-80e6-e66a7a9a9615" containerName="test-curl-7822-140659763017136"
Jul 29 07:18:09.179323 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:09.179299    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab086-664f494947-7v674"
Jul 29 07:18:09.181246 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:09.181226    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-aab086-serving-cert\""
Jul 29 07:18:09.186806 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:09.186780    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab086-664f494947-7v674"]
Jul 29 07:18:09.280076 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:09.280038    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/845cff92-9c98-4a83-afeb-e7360e562f06-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-aab086-664f494947-7v674\" (UID: \"845cff92-9c98-4a83-afeb-e7360e562f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab086-664f494947-7v674"
Jul 29 07:18:09.280285 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:09.280183    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqvlq\" (UniqueName: \"kubernetes.io/projected/845cff92-9c98-4a83-afeb-e7360e562f06-kube-api-access-sqvlq\") pod \"maas-api-e2e-ipp-a-aab086-664f494947-7v674\" (UID: \"845cff92-9c98-4a83-afeb-e7360e562f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab086-664f494947-7v674"
Jul 29 07:18:09.280285 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:09.280236    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/845cff92-9c98-4a83-afeb-e7360e562f06-maas-api-tls\") pod \"maas-api-e2e-ipp-a-aab086-664f494947-7v674\" (UID: \"845cff92-9c98-4a83-afeb-e7360e562f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab086-664f494947-7v674"
Jul 29 07:18:09.380936 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:09.380904    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/845cff92-9c98-4a83-afeb-e7360e562f06-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-aab086-664f494947-7v674\" (UID: \"845cff92-9c98-4a83-afeb-e7360e562f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab086-664f494947-7v674"
Jul 29 07:18:09.381125 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:09.381007    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqvlq\" (UniqueName: \"kubernetes.io/projected/845cff92-9c98-4a83-afeb-e7360e562f06-kube-api-access-sqvlq\") pod \"maas-api-e2e-ipp-a-aab086-664f494947-7v674\" (UID: \"845cff92-9c98-4a83-afeb-e7360e562f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab086-664f494947-7v674"
Jul 29 07:18:09.381125 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:09.381042    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/845cff92-9c98-4a83-afeb-e7360e562f06-maas-api-tls\") pod \"maas-api-e2e-ipp-a-aab086-664f494947-7v674\" (UID: \"845cff92-9c98-4a83-afeb-e7360e562f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab086-664f494947-7v674"
Jul 29 07:18:09.381242 ip-10-0-129-223 kubenswrapper[2563]: E0729 07:18:09.381156    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab086-serving-cert: secret "maas-api-e2e-ipp-a-aab086-serving-cert" not found
Jul 29 07:18:09.381242 ip-10-0-129-223 kubenswrapper[2563]: E0729 07:18:09.381240    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/845cff92-9c98-4a83-afeb-e7360e562f06-maas-api-tls podName:845cff92-9c98-4a83-afeb-e7360e562f06 nodeName:}" failed. No retries permitted until 2026-07-29 07:18:09.881218228 +0000 UTC m=+3332.958849977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/845cff92-9c98-4a83-afeb-e7360e562f06-maas-api-tls") pod "maas-api-e2e-ipp-a-aab086-664f494947-7v674" (UID: "845cff92-9c98-4a83-afeb-e7360e562f06") : secret "maas-api-e2e-ipp-a-aab086-serving-cert" not found
Jul 29 07:18:09.381661 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:09.381639    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/845cff92-9c98-4a83-afeb-e7360e562f06-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-aab086-664f494947-7v674\" (UID: \"845cff92-9c98-4a83-afeb-e7360e562f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab086-664f494947-7v674"
Jul 29 07:18:09.389731 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:09.389705    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqvlq\" (UniqueName: \"kubernetes.io/projected/845cff92-9c98-4a83-afeb-e7360e562f06-kube-api-access-sqvlq\") pod \"maas-api-e2e-ipp-a-aab086-664f494947-7v674\" (UID: \"845cff92-9c98-4a83-afeb-e7360e562f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab086-664f494947-7v674"
Jul 29 07:18:09.885983 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:09.885950    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/845cff92-9c98-4a83-afeb-e7360e562f06-maas-api-tls\") pod \"maas-api-e2e-ipp-a-aab086-664f494947-7v674\" (UID: \"845cff92-9c98-4a83-afeb-e7360e562f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab086-664f494947-7v674"
Jul 29 07:18:09.888561 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:09.888540    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/845cff92-9c98-4a83-afeb-e7360e562f06-maas-api-tls\") pod \"maas-api-e2e-ipp-a-aab086-664f494947-7v674\" (UID: \"845cff92-9c98-4a83-afeb-e7360e562f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab086-664f494947-7v674"
Jul 29 07:18:10.092450 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:10.092415    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab086-664f494947-7v674"
Jul 29 07:18:10.235825 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:10.235799    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab086-664f494947-7v674"]
Jul 29 07:18:10.237948 ip-10-0-129-223 kubenswrapper[2563]: W0729 07:18:10.237921    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod845cff92_9c98_4a83_afeb_e7360e562f06.slice/crio-9e3629732963eabf803c3714de7e9623300b0dde5df0e79220678ee08d12f6ff WatchSource:0}: Error finding container 9e3629732963eabf803c3714de7e9623300b0dde5df0e79220678ee08d12f6ff: Status 404 returned error can't find the container with id 9e3629732963eabf803c3714de7e9623300b0dde5df0e79220678ee08d12f6ff
Jul 29 07:18:10.443525 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:10.443448    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab086-664f494947-7v674" event={"ID":"845cff92-9c98-4a83-afeb-e7360e562f06","Type":"ContainerStarted","Data":"9e3629732963eabf803c3714de7e9623300b0dde5df0e79220678ee08d12f6ff"}
Jul 29 07:18:11.449643 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:11.449610    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab086-664f494947-7v674" event={"ID":"845cff92-9c98-4a83-afeb-e7360e562f06","Type":"ContainerStarted","Data":"30a9ca4e47dfce0f10af272b1f68f5caa7ff3831f7160124ce3ea5955f006b24"}
Jul 29 07:18:11.450081 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:11.449677    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab086-664f494947-7v674"
Jul 29 07:18:11.467448 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:11.467387    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab086-664f494947-7v674" podStartSLOduration=1.8648111379999999 podStartE2EDuration="2.467372418s" totalImagesPullingTime="602.56128ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:18:11.465110189 +0000 UTC m=+3334.542741970" watchObservedRunningTime="2026-07-29 07:18:11.467372418 +0000 UTC m=+3334.545004185"
Jul 29 07:18:17.461784 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:17.461751    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab086-664f494947-7v674"
Jul 29 07:18:38.172380 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:38.172347    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9"]
Jul 29 07:18:38.176311 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:38.176287    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9"
Jul 29 07:18:38.179432 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:38.179407    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-04c3b9-serving-cert\""
Jul 29 07:18:38.190841 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:38.190821    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9"]
Jul 29 07:18:38.260773 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:38.260743    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgbkz\" (UniqueName: \"kubernetes.io/projected/c89b4f41-723e-4e60-adb1-7788889aa156-kube-api-access-lgbkz\") pod \"maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9\" (UID: \"c89b4f41-723e-4e60-adb1-7788889aa156\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9"
Jul 29 07:18:38.260990 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:38.260795    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c89b4f41-723e-4e60-adb1-7788889aa156-maas-api-tls\") pod \"maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9\" (UID: \"c89b4f41-723e-4e60-adb1-7788889aa156\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9"
Jul 29 07:18:38.260990 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:38.260882    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89b4f41-723e-4e60-adb1-7788889aa156-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9\" (UID: \"c89b4f41-723e-4e60-adb1-7788889aa156\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9"
Jul 29 07:18:38.362280 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:38.362239    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgbkz\" (UniqueName: \"kubernetes.io/projected/c89b4f41-723e-4e60-adb1-7788889aa156-kube-api-access-lgbkz\") pod \"maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9\" (UID: \"c89b4f41-723e-4e60-adb1-7788889aa156\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9"
Jul 29 07:18:38.362485 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:38.362302    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c89b4f41-723e-4e60-adb1-7788889aa156-maas-api-tls\") pod \"maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9\" (UID: \"c89b4f41-723e-4e60-adb1-7788889aa156\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9"
Jul 29 07:18:38.362485 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:38.362339    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89b4f41-723e-4e60-adb1-7788889aa156-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9\" (UID: \"c89b4f41-723e-4e60-adb1-7788889aa156\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9"
Jul 29 07:18:38.363140 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:38.363072    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89b4f41-723e-4e60-adb1-7788889aa156-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9\" (UID: \"c89b4f41-723e-4e60-adb1-7788889aa156\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9"
Jul 29 07:18:38.364938 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:38.364917    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c89b4f41-723e-4e60-adb1-7788889aa156-maas-api-tls\") pod \"maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9\" (UID: \"c89b4f41-723e-4e60-adb1-7788889aa156\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9"
Jul 29 07:18:38.369993 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:38.369967    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgbkz\" (UniqueName: \"kubernetes.io/projected/c89b4f41-723e-4e60-adb1-7788889aa156-kube-api-access-lgbkz\") pod \"maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9\" (UID: \"c89b4f41-723e-4e60-adb1-7788889aa156\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9"
Jul 29 07:18:38.486775 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:38.486691    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9"
Jul 29 07:18:38.662135 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:38.662105    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9"]
Jul 29 07:18:38.965745 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:38.965704    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-04c3b9-5554d46df6-fnq6c"]
Jul 29 07:18:38.970699 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:38.970676    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-04c3b9-5554d46df6-fnq6c"
Jul 29 07:18:38.972770 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:38.972749    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-04c3b9\""
Jul 29 07:18:38.978804 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:38.978776    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-04c3b9-5554d46df6-fnq6c"]
Jul 29 07:18:38.991140 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:38.991113    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-04c3b9-5f58c78b7d-wf6ws"]
Jul 29 07:18:38.995542 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:38.995523    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-04c3b9-5f58c78b7d-wf6ws"
Jul 29 07:18:38.997755 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:38.997731    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-04c3b9-dockercfg-zhq8p\""
Jul 29 07:18:39.011046 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:39.011023    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-04c3b9-5f58c78b7d-wf6ws"]
Jul 29 07:18:39.069134 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:39.069101    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb8hr\" (UniqueName: \"kubernetes.io/projected/e2713431-5709-4c72-87b3-5b8b37a7816d-kube-api-access-cb8hr\") pod \"payload-pre-processing-e2e-ipp-b-04c3b9-5554d46df6-fnq6c\" (UID: \"e2713431-5709-4c72-87b3-5b8b37a7816d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-04c3b9-5554d46df6-fnq6c"
Jul 29 07:18:39.069134 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:39.069134    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2713431-5709-4c72-87b3-5b8b37a7816d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-04c3b9-5554d46df6-fnq6c\" (UID: \"e2713431-5709-4c72-87b3-5b8b37a7816d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-04c3b9-5554d46df6-fnq6c"
Jul 29 07:18:39.069317 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:39.069162    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxlgw\" (UniqueName: \"kubernetes.io/projected/dd5e5f16-bec9-48a4-80f3-73662f2ba97c-kube-api-access-xxlgw\") pod \"payload-processing-e2e-ipp-b-04c3b9-5f58c78b7d-wf6ws\" (UID: \"dd5e5f16-bec9-48a4-80f3-73662f2ba97c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-04c3b9-5f58c78b7d-wf6ws"
Jul 29 07:18:39.069317 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:39.069288    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd5e5f16-bec9-48a4-80f3-73662f2ba97c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-04c3b9-5f58c78b7d-wf6ws\" (UID: \"dd5e5f16-bec9-48a4-80f3-73662f2ba97c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-04c3b9-5f58c78b7d-wf6ws"
Jul 29 07:18:39.170458 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:39.170425    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd5e5f16-bec9-48a4-80f3-73662f2ba97c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-04c3b9-5f58c78b7d-wf6ws\" (UID: \"dd5e5f16-bec9-48a4-80f3-73662f2ba97c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-04c3b9-5f58c78b7d-wf6ws"
Jul 29 07:18:39.170601 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:39.170496    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb8hr\" (UniqueName: \"kubernetes.io/projected/e2713431-5709-4c72-87b3-5b8b37a7816d-kube-api-access-cb8hr\") pod \"payload-pre-processing-e2e-ipp-b-04c3b9-5554d46df6-fnq6c\" (UID: \"e2713431-5709-4c72-87b3-5b8b37a7816d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-04c3b9-5554d46df6-fnq6c"
Jul 29 07:18:39.170677 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:39.170655    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2713431-5709-4c72-87b3-5b8b37a7816d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-04c3b9-5554d46df6-fnq6c\" (UID: \"e2713431-5709-4c72-87b3-5b8b37a7816d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-04c3b9-5554d46df6-fnq6c"
Jul 29 07:18:39.170728 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:39.170716    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxlgw\" (UniqueName: \"kubernetes.io/projected/dd5e5f16-bec9-48a4-80f3-73662f2ba97c-kube-api-access-xxlgw\") pod \"payload-processing-e2e-ipp-b-04c3b9-5f58c78b7d-wf6ws\" (UID: \"dd5e5f16-bec9-48a4-80f3-73662f2ba97c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-04c3b9-5f58c78b7d-wf6ws"
Jul 29 07:18:39.171247 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:39.171229    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd5e5f16-bec9-48a4-80f3-73662f2ba97c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-04c3b9-5f58c78b7d-wf6ws\" (UID: \"dd5e5f16-bec9-48a4-80f3-73662f2ba97c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-04c3b9-5f58c78b7d-wf6ws"
Jul 29 07:18:39.171387 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:39.171370    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2713431-5709-4c72-87b3-5b8b37a7816d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-04c3b9-5554d46df6-fnq6c\" (UID: \"e2713431-5709-4c72-87b3-5b8b37a7816d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-04c3b9-5554d46df6-fnq6c"
Jul 29 07:18:39.178888 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:39.178843    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxlgw\" (UniqueName: \"kubernetes.io/projected/dd5e5f16-bec9-48a4-80f3-73662f2ba97c-kube-api-access-xxlgw\") pod \"payload-processing-e2e-ipp-b-04c3b9-5f58c78b7d-wf6ws\" (UID: \"dd5e5f16-bec9-48a4-80f3-73662f2ba97c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-04c3b9-5f58c78b7d-wf6ws"
Jul 29 07:18:39.179156 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:39.178845    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb8hr\" (UniqueName: \"kubernetes.io/projected/e2713431-5709-4c72-87b3-5b8b37a7816d-kube-api-access-cb8hr\") pod \"payload-pre-processing-e2e-ipp-b-04c3b9-5554d46df6-fnq6c\" (UID: \"e2713431-5709-4c72-87b3-5b8b37a7816d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-04c3b9-5554d46df6-fnq6c"
Jul 29 07:18:39.284100 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:39.284015    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-04c3b9-5554d46df6-fnq6c"
Jul 29 07:18:39.306203 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:39.306171    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-04c3b9-5f58c78b7d-wf6ws"
Jul 29 07:18:39.456927 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:39.456728    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-04c3b9-5554d46df6-fnq6c"]
Jul 29 07:18:39.460922 ip-10-0-129-223 kubenswrapper[2563]: W0729 07:18:39.460896    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2713431_5709_4c72_87b3_5b8b37a7816d.slice/crio-75dc621a6cbedbde8fc8f44bf8ae75b69cc620179c77f61d6a0e7ab6c9383c7c WatchSource:0}: Error finding container 75dc621a6cbedbde8fc8f44bf8ae75b69cc620179c77f61d6a0e7ab6c9383c7c: Status 404 returned error can't find the container with id 75dc621a6cbedbde8fc8f44bf8ae75b69cc620179c77f61d6a0e7ab6c9383c7c
Jul 29 07:18:39.474327 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:39.474303    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-04c3b9-5f58c78b7d-wf6ws"]
Jul 29 07:18:39.477268 ip-10-0-129-223 kubenswrapper[2563]: W0729 07:18:39.477244    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd5e5f16_bec9_48a4_80f3_73662f2ba97c.slice/crio-35db4d13ccad93c47873644cc2c491c20a057df8ea0ee3d76b4b478185065294 WatchSource:0}: Error finding container 35db4d13ccad93c47873644cc2c491c20a057df8ea0ee3d76b4b478185065294: Status 404 returned error can't find the container with id 35db4d13ccad93c47873644cc2c491c20a057df8ea0ee3d76b4b478185065294
Jul 29 07:18:39.597380 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:39.597343    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9" event={"ID":"c89b4f41-723e-4e60-adb1-7788889aa156","Type":"ContainerStarted","Data":"424df5e1fc3c166f37daca08a00e52af193914d4a7af63d36398c9fa193b8a87"}
Jul 29 07:18:39.597581 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:39.597405    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9"
Jul 29 07:18:39.597581 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:39.597429    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9" event={"ID":"c89b4f41-723e-4e60-adb1-7788889aa156","Type":"ContainerStarted","Data":"45532473a6697ff5700889e566d98acc8ca5a475f7e2e1e3d9d988919fb04389"}
Jul 29 07:18:39.598992 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:39.598966    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-04c3b9-5f58c78b7d-wf6ws" event={"ID":"dd5e5f16-bec9-48a4-80f3-73662f2ba97c","Type":"ContainerStarted","Data":"d62ecf985c860272b0c134301c0cfec42926ea83b2980cefe208b552a7310a2b"}
Jul 29 07:18:39.599127 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:39.599009    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-04c3b9-5f58c78b7d-wf6ws" event={"ID":"dd5e5f16-bec9-48a4-80f3-73662f2ba97c","Type":"ContainerStarted","Data":"35db4d13ccad93c47873644cc2c491c20a057df8ea0ee3d76b4b478185065294"}
Jul 29 07:18:39.599127 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:39.599033    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-04c3b9-5f58c78b7d-wf6ws"
Jul 29 07:18:39.602825 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:39.602791    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-04c3b9-5554d46df6-fnq6c" event={"ID":"e2713431-5709-4c72-87b3-5b8b37a7816d","Type":"ContainerStarted","Data":"3e0ab1273ae936e223044dc8ce5a0f241db97dda37a7d635d72299fc31cb0831"}
Jul 29 07:18:39.602934 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:39.602839    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-04c3b9-5554d46df6-fnq6c" event={"ID":"e2713431-5709-4c72-87b3-5b8b37a7816d","Type":"ContainerStarted","Data":"75dc621a6cbedbde8fc8f44bf8ae75b69cc620179c77f61d6a0e7ab6c9383c7c"}
Jul 29 07:18:39.603782 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:39.603031    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-04c3b9-5554d46df6-fnq6c"
Jul 29 07:18:39.622317 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:39.622258    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9" podStartSLOduration=1.2087323620000001 podStartE2EDuration="1.622236389s" totalImagesPullingTime="413.504027ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:18:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:18:39.619605597 +0000 UTC m=+3362.697237378" watchObservedRunningTime="2026-07-29 07:18:39.622236389 +0000 UTC m=+3362.699868158"
Jul 29 07:18:39.640640 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:39.640592    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-04c3b9-5554d46df6-fnq6c" podStartSLOduration=1.640577796 podStartE2EDuration="1.640577796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:18:39.638683001 +0000 UTC m=+3362.716314770" watchObservedRunningTime="2026-07-29 07:18:39.640577796 +0000 UTC m=+3362.718209562"
Jul 29 07:18:39.662944 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:39.662900    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-04c3b9-5f58c78b7d-wf6ws" podStartSLOduration=1.6628866979999999 podStartE2EDuration="1.662886698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:18:39.660830224 +0000 UTC m=+3362.738461992" watchObservedRunningTime="2026-07-29 07:18:39.662886698 +0000 UTC m=+3362.740518464"
Jul 29 07:18:45.616248 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:45.616219    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9"
Jul 29 07:18:50.613083 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:50.613054    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-04c3b9-5f58c78b7d-wf6ws"
Jul 29 07:18:50.613489 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:18:50.613125    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-04c3b9-5554d46df6-fnq6c"
Jul 29 07:19:09.900462 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:19:09.900436    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-csc8g_75e1377f-ca60-47c3-b91b-f328413bd9f6/payload-processing/0.log"
Jul 29 07:19:10.011776 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:19:10.011748    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-04c3b9-5f58c78b7d-wf6ws_dd5e5f16-bec9-48a4-80f3-73662f2ba97c/payload-processing/0.log"
Jul 29 07:20:00.212290 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:00.212261    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-csc8g_75e1377f-ca60-47c3-b91b-f328413bd9f6/payload-processing/0.log"
Jul 29 07:20:00.341334 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:00.341306    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-04c3b9-5f58c78b7d-wf6ws_dd5e5f16-bec9-48a4-80f3-73662f2ba97c/payload-processing/0.log"
Jul 29 07:20:07.484297 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:07.484265    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr"]
Jul 29 07:20:07.492484 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:07.489298    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr"
Jul 29 07:20:07.493075 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:07.493047    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-43705a-serving-cert\""
Jul 29 07:20:07.500472 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:07.500450    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr"]
Jul 29 07:20:07.621974 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:07.621936    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt8mc\" (UniqueName: \"kubernetes.io/projected/d48b4ad8-c49a-41e4-b466-662f5a9ad76d-kube-api-access-qt8mc\") pod \"maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr\" (UID: \"d48b4ad8-c49a-41e4-b466-662f5a9ad76d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr"
Jul 29 07:20:07.622180 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:07.622103    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d48b4ad8-c49a-41e4-b466-662f5a9ad76d-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr\" (UID: \"d48b4ad8-c49a-41e4-b466-662f5a9ad76d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr"
Jul 29 07:20:07.622180 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:07.622152    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d48b4ad8-c49a-41e4-b466-662f5a9ad76d-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr\" (UID: \"d48b4ad8-c49a-41e4-b466-662f5a9ad76d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr"
Jul 29 07:20:07.723463 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:07.723431    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d48b4ad8-c49a-41e4-b466-662f5a9ad76d-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr\" (UID: \"d48b4ad8-c49a-41e4-b466-662f5a9ad76d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr"
Jul 29 07:20:07.723623 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:07.723479    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d48b4ad8-c49a-41e4-b466-662f5a9ad76d-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr\" (UID: \"d48b4ad8-c49a-41e4-b466-662f5a9ad76d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr"
Jul 29 07:20:07.723623 ip-10-0-129-223 kubenswrapper[2563]: E0729 07:20:07.723587    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-43705a-serving-cert: secret "maas-api-e2e-ipp-cleanup-43705a-serving-cert" not found
Jul 29 07:20:07.723623 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:07.723605    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt8mc\" (UniqueName: \"kubernetes.io/projected/d48b4ad8-c49a-41e4-b466-662f5a9ad76d-kube-api-access-qt8mc\") pod \"maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr\" (UID: \"d48b4ad8-c49a-41e4-b466-662f5a9ad76d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr"
Jul 29 07:20:07.723738 ip-10-0-129-223 kubenswrapper[2563]: E0729 07:20:07.723680    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d48b4ad8-c49a-41e4-b466-662f5a9ad76d-maas-api-tls podName:d48b4ad8-c49a-41e4-b466-662f5a9ad76d nodeName:}" failed. No retries permitted until 2026-07-29 07:20:08.223658596 +0000 UTC m=+3451.301290343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d48b4ad8-c49a-41e4-b466-662f5a9ad76d-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr" (UID: "d48b4ad8-c49a-41e4-b466-662f5a9ad76d") : secret "maas-api-e2e-ipp-cleanup-43705a-serving-cert" not found
Jul 29 07:20:07.724127 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:07.724111    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d48b4ad8-c49a-41e4-b466-662f5a9ad76d-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr\" (UID: \"d48b4ad8-c49a-41e4-b466-662f5a9ad76d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr"
Jul 29 07:20:07.734583 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:07.734522    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt8mc\" (UniqueName: \"kubernetes.io/projected/d48b4ad8-c49a-41e4-b466-662f5a9ad76d-kube-api-access-qt8mc\") pod \"maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr\" (UID: \"d48b4ad8-c49a-41e4-b466-662f5a9ad76d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr"
Jul 29 07:20:08.198169 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:08.198138    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-43705a-7bcb4798d4-wssxm"]
Jul 29 07:20:08.202152 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:08.202133    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-43705a-7bcb4798d4-wssxm"
Jul 29 07:20:08.204913 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:08.204888    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-43705a\""
Jul 29 07:20:08.205036 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:08.204998    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-43705a-dockercfg-tz87s\""
Jul 29 07:20:08.213251 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:08.213233    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-43705a-7bcb4798d4-wssxm"]
Jul 29 07:20:08.229905 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:08.229878    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d48b4ad8-c49a-41e4-b466-662f5a9ad76d-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr\" (UID: \"d48b4ad8-c49a-41e4-b466-662f5a9ad76d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr"
Jul 29 07:20:08.232541 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:08.232518    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d48b4ad8-c49a-41e4-b466-662f5a9ad76d-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr\" (UID: \"d48b4ad8-c49a-41e4-b466-662f5a9ad76d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr"
Jul 29 07:20:08.330919 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:08.330890    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzmcm\" (UniqueName: \"kubernetes.io/projected/c5be84d4-e67d-4abb-91fb-062564c1d063-kube-api-access-dzmcm\") pod \"payload-processing-e2e-ipp-cleanup-43705a-7bcb4798d4-wssxm\" (UID: \"c5be84d4-e67d-4abb-91fb-062564c1d063\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-43705a-7bcb4798d4-wssxm"
Jul 29 07:20:08.331078 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:08.330957    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5be84d4-e67d-4abb-91fb-062564c1d063-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-43705a-7bcb4798d4-wssxm\" (UID: \"c5be84d4-e67d-4abb-91fb-062564c1d063\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-43705a-7bcb4798d4-wssxm"
Jul 29 07:20:08.403865 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:08.403820    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr"
Jul 29 07:20:08.432119 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:08.432091    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzmcm\" (UniqueName: \"kubernetes.io/projected/c5be84d4-e67d-4abb-91fb-062564c1d063-kube-api-access-dzmcm\") pod \"payload-processing-e2e-ipp-cleanup-43705a-7bcb4798d4-wssxm\" (UID: \"c5be84d4-e67d-4abb-91fb-062564c1d063\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-43705a-7bcb4798d4-wssxm"
Jul 29 07:20:08.432251 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:08.432149    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5be84d4-e67d-4abb-91fb-062564c1d063-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-43705a-7bcb4798d4-wssxm\" (UID: \"c5be84d4-e67d-4abb-91fb-062564c1d063\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-43705a-7bcb4798d4-wssxm"
Jul 29 07:20:08.432837 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:08.432819    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5be84d4-e67d-4abb-91fb-062564c1d063-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-43705a-7bcb4798d4-wssxm\" (UID: \"c5be84d4-e67d-4abb-91fb-062564c1d063\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-43705a-7bcb4798d4-wssxm"
Jul 29 07:20:08.440247 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:08.440215    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzmcm\" (UniqueName: \"kubernetes.io/projected/c5be84d4-e67d-4abb-91fb-062564c1d063-kube-api-access-dzmcm\") pod \"payload-processing-e2e-ipp-cleanup-43705a-7bcb4798d4-wssxm\" (UID: \"c5be84d4-e67d-4abb-91fb-062564c1d063\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-43705a-7bcb4798d4-wssxm"
Jul 29 07:20:08.514161 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:08.514129    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-43705a-7bcb4798d4-wssxm"
Jul 29 07:20:08.552230 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:08.551842    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr"]
Jul 29 07:20:08.554093 ip-10-0-129-223 kubenswrapper[2563]: W0729 07:20:08.554061    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd48b4ad8_c49a_41e4_b466_662f5a9ad76d.slice/crio-b6de80613974f239d580a16e52bdf2c1d45f57b05d32397be385ef39c7d1eae8 WatchSource:0}: Error finding container b6de80613974f239d580a16e52bdf2c1d45f57b05d32397be385ef39c7d1eae8: Status 404 returned error can't find the container with id b6de80613974f239d580a16e52bdf2c1d45f57b05d32397be385ef39c7d1eae8
Jul 29 07:20:08.660186 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:08.660163    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-43705a-7bcb4798d4-wssxm"]
Jul 29 07:20:08.662583 ip-10-0-129-223 kubenswrapper[2563]: W0729 07:20:08.662551    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5be84d4_e67d_4abb_91fb_062564c1d063.slice/crio-235578e4ab76b6e360b7aa55059de20419529d22df979f0e1694a19c611a5b4d WatchSource:0}: Error finding container 235578e4ab76b6e360b7aa55059de20419529d22df979f0e1694a19c611a5b4d: Status 404 returned error can't find the container with id 235578e4ab76b6e360b7aa55059de20419529d22df979f0e1694a19c611a5b4d
Jul 29 07:20:09.083436 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:09.083405    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-43705a-7bcb4798d4-wssxm" event={"ID":"c5be84d4-e67d-4abb-91fb-062564c1d063","Type":"ContainerStarted","Data":"e381376930d4a7db7a14ed0ea8014225863add9b2339693b691a5c2fcf420a5e"}
Jul 29 07:20:09.083632 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:09.083464    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-43705a-7bcb4798d4-wssxm" event={"ID":"c5be84d4-e67d-4abb-91fb-062564c1d063","Type":"ContainerStarted","Data":"235578e4ab76b6e360b7aa55059de20419529d22df979f0e1694a19c611a5b4d"}
Jul 29 07:20:09.083632 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:09.083493    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-43705a-7bcb4798d4-wssxm"
Jul 29 07:20:09.084707 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:09.084686    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr" event={"ID":"d48b4ad8-c49a-41e4-b466-662f5a9ad76d","Type":"ContainerStarted","Data":"b6de80613974f239d580a16e52bdf2c1d45f57b05d32397be385ef39c7d1eae8"}
Jul 29 07:20:09.100717 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:09.100670    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-43705a-7bcb4798d4-wssxm" podStartSLOduration=1.100655038 podStartE2EDuration="1.100655038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:20:09.09847394 +0000 UTC m=+3452.176105721" watchObservedRunningTime="2026-07-29 07:20:09.100655038 +0000 UTC m=+3452.178286804"
Jul 29 07:20:10.092057 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:10.092016    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr" event={"ID":"d48b4ad8-c49a-41e4-b466-662f5a9ad76d","Type":"ContainerStarted","Data":"daebdc280678a58256a608fe4f4495b92f8a81aa3861a9f91619428da37430bb"}
Jul 29 07:20:10.092500 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:10.092078    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr"
Jul 29 07:20:10.109293 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:10.109247    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr" podStartSLOduration=2.542733587 podStartE2EDuration="3.109231794s" totalImagesPullingTime="566.498207ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:20:10.10833978 +0000 UTC m=+3453.185971548" watchObservedRunningTime="2026-07-29 07:20:10.109231794 +0000 UTC m=+3453.186863560"
Jul 29 07:20:16.103043 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:16.103014    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr"
Jul 29 07:20:20.094121 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:20.094092    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-43705a-7bcb4798d4-wssxm"
Jul 29 07:20:48.110555 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:48.110497    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr"]
Jul 29 07:20:48.111195 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:48.110748    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr" podUID="d48b4ad8-c49a-41e4-b466-662f5a9ad76d" containerName="maas-api" containerID="cri-o://daebdc280678a58256a608fe4f4495b92f8a81aa3861a9f91619428da37430bb" gracePeriod=30
Jul 29 07:20:48.170686 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:48.170653    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-43705a-7bcb4798d4-wssxm"]
Jul 29 07:20:48.170909 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:48.170880    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-43705a-7bcb4798d4-wssxm" podUID="c5be84d4-e67d-4abb-91fb-062564c1d063" containerName="payload-processing" containerID="cri-o://e381376930d4a7db7a14ed0ea8014225863add9b2339693b691a5c2fcf420a5e" gracePeriod=30
Jul 29 07:20:48.302076 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:48.302028    2563 generic.go:356] "Generic (PLEG): container finished" podID="d48b4ad8-c49a-41e4-b466-662f5a9ad76d" containerID="daebdc280678a58256a608fe4f4495b92f8a81aa3861a9f91619428da37430bb" exitCode=0
Jul 29 07:20:48.302256 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:48.302140    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr" event={"ID":"d48b4ad8-c49a-41e4-b466-662f5a9ad76d","Type":"ContainerDied","Data":"daebdc280678a58256a608fe4f4495b92f8a81aa3861a9f91619428da37430bb"}
Jul 29 07:20:48.303908 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:48.303883    2563 generic.go:356] "Generic (PLEG): container finished" podID="c5be84d4-e67d-4abb-91fb-062564c1d063" containerID="e381376930d4a7db7a14ed0ea8014225863add9b2339693b691a5c2fcf420a5e" exitCode=0
Jul 29 07:20:48.304048 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:48.303959    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-43705a-7bcb4798d4-wssxm" event={"ID":"c5be84d4-e67d-4abb-91fb-062564c1d063","Type":"ContainerDied","Data":"e381376930d4a7db7a14ed0ea8014225863add9b2339693b691a5c2fcf420a5e"}
Jul 29 07:20:48.445382 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:48.445359    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr"
Jul 29 07:20:48.460899 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:48.460879    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-43705a-7bcb4798d4-wssxm"
Jul 29 07:20:48.631509 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:48.631469    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5be84d4-e67d-4abb-91fb-062564c1d063-plugins-config-volume\") pod \"c5be84d4-e67d-4abb-91fb-062564c1d063\" (UID: \"c5be84d4-e67d-4abb-91fb-062564c1d063\") "
Jul 29 07:20:48.631722 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:48.631586    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzmcm\" (UniqueName: \"kubernetes.io/projected/c5be84d4-e67d-4abb-91fb-062564c1d063-kube-api-access-dzmcm\") pod \"c5be84d4-e67d-4abb-91fb-062564c1d063\" (UID: \"c5be84d4-e67d-4abb-91fb-062564c1d063\") "
Jul 29 07:20:48.631722 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:48.631615    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qt8mc\" (UniqueName: \"kubernetes.io/projected/d48b4ad8-c49a-41e4-b466-662f5a9ad76d-kube-api-access-qt8mc\") pod \"d48b4ad8-c49a-41e4-b466-662f5a9ad76d\" (UID: \"d48b4ad8-c49a-41e4-b466-662f5a9ad76d\") "
Jul 29 07:20:48.631722 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:48.631664    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d48b4ad8-c49a-41e4-b466-662f5a9ad76d-maas-api-tls\") pod \"d48b4ad8-c49a-41e4-b466-662f5a9ad76d\" (UID: \"d48b4ad8-c49a-41e4-b466-662f5a9ad76d\") "
Jul 29 07:20:48.631722 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:48.631688    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d48b4ad8-c49a-41e4-b466-662f5a9ad76d-openshift-service-ca\") pod \"d48b4ad8-c49a-41e4-b466-662f5a9ad76d\" (UID: \"d48b4ad8-c49a-41e4-b466-662f5a9ad76d\") "
Jul 29 07:20:48.631975 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:48.631904    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5be84d4-e67d-4abb-91fb-062564c1d063-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c5be84d4-e67d-4abb-91fb-062564c1d063" (UID: "c5be84d4-e67d-4abb-91fb-062564c1d063"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 07:20:48.632180 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:48.632152    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d48b4ad8-c49a-41e4-b466-662f5a9ad76d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d48b4ad8-c49a-41e4-b466-662f5a9ad76d" (UID: "d48b4ad8-c49a-41e4-b466-662f5a9ad76d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 07:20:48.632306 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:48.632263    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d48b4ad8-c49a-41e4-b466-662f5a9ad76d-openshift-service-ca\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:20:48.632306 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:48.632293    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5be84d4-e67d-4abb-91fb-062564c1d063-plugins-config-volume\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:20:48.633964 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:48.633938    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d48b4ad8-c49a-41e4-b466-662f5a9ad76d-kube-api-access-qt8mc" (OuterVolumeSpecName: "kube-api-access-qt8mc") pod "d48b4ad8-c49a-41e4-b466-662f5a9ad76d" (UID: "d48b4ad8-c49a-41e4-b466-662f5a9ad76d"). InnerVolumeSpecName "kube-api-access-qt8mc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 07:20:48.634056 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:48.633963    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d48b4ad8-c49a-41e4-b466-662f5a9ad76d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d48b4ad8-c49a-41e4-b466-662f5a9ad76d" (UID: "d48b4ad8-c49a-41e4-b466-662f5a9ad76d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 07:20:48.634056 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:48.633964    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5be84d4-e67d-4abb-91fb-062564c1d063-kube-api-access-dzmcm" (OuterVolumeSpecName: "kube-api-access-dzmcm") pod "c5be84d4-e67d-4abb-91fb-062564c1d063" (UID: "c5be84d4-e67d-4abb-91fb-062564c1d063"). InnerVolumeSpecName "kube-api-access-dzmcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 07:20:48.733827 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:48.733739    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzmcm\" (UniqueName: \"kubernetes.io/projected/c5be84d4-e67d-4abb-91fb-062564c1d063-kube-api-access-dzmcm\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:20:48.733827 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:48.733769    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt8mc\" (UniqueName: \"kubernetes.io/projected/d48b4ad8-c49a-41e4-b466-662f5a9ad76d-kube-api-access-qt8mc\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:20:48.733827 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:48.733780    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d48b4ad8-c49a-41e4-b466-662f5a9ad76d-maas-api-tls\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:20:49.317834 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:49.317805    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr"
Jul 29 07:20:49.318323 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:49.317797    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr" event={"ID":"d48b4ad8-c49a-41e4-b466-662f5a9ad76d","Type":"ContainerDied","Data":"b6de80613974f239d580a16e52bdf2c1d45f57b05d32397be385ef39c7d1eae8"}
Jul 29 07:20:49.318323 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:49.317940    2563 scope.go:117] "RemoveContainer" containerID="daebdc280678a58256a608fe4f4495b92f8a81aa3861a9f91619428da37430bb"
Jul 29 07:20:49.319606 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:49.319461    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-43705a-7bcb4798d4-wssxm"
Jul 29 07:20:49.319606 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:49.319480    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-43705a-7bcb4798d4-wssxm" event={"ID":"c5be84d4-e67d-4abb-91fb-062564c1d063","Type":"ContainerDied","Data":"235578e4ab76b6e360b7aa55059de20419529d22df979f0e1694a19c611a5b4d"}
Jul 29 07:20:49.328601 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:49.328579    2563 scope.go:117] "RemoveContainer" containerID="e381376930d4a7db7a14ed0ea8014225863add9b2339693b691a5c2fcf420a5e"
Jul 29 07:20:49.342393 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:49.342367    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr"]
Jul 29 07:20:49.345878 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:49.345840    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-43705a-798b9cb84-r5knr"]
Jul 29 07:20:49.354921 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:49.354896    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-43705a-7bcb4798d4-wssxm"]
Jul 29 07:20:49.357553 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:49.357535    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-43705a-7bcb4798d4-wssxm"]
Jul 29 07:20:49.398873 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:49.398786    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5be84d4-e67d-4abb-91fb-062564c1d063" path="/var/lib/kubelet/pods/c5be84d4-e67d-4abb-91fb-062564c1d063/volumes"
Jul 29 07:20:49.399227 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:49.399213    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d48b4ad8-c49a-41e4-b466-662f5a9ad76d" path="/var/lib/kubelet/pods/d48b4ad8-c49a-41e4-b466-662f5a9ad76d/volumes"
Jul 29 07:20:59.062156 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:59.062082    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-aab086-jpsck"]
Jul 29 07:20:59.062545 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:59.062531    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5be84d4-e67d-4abb-91fb-062564c1d063" containerName="payload-processing"
Jul 29 07:20:59.062590 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:59.062546    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5be84d4-e67d-4abb-91fb-062564c1d063" containerName="payload-processing"
Jul 29 07:20:59.062590 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:59.062556    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d48b4ad8-c49a-41e4-b466-662f5a9ad76d" containerName="maas-api"
Jul 29 07:20:59.062590 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:59.062562    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d48b4ad8-c49a-41e4-b466-662f5a9ad76d" containerName="maas-api"
Jul 29 07:20:59.062683 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:59.062625    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5be84d4-e67d-4abb-91fb-062564c1d063" containerName="payload-processing"
Jul 29 07:20:59.062683 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:59.062639    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d48b4ad8-c49a-41e4-b466-662f5a9ad76d" containerName="maas-api"
Jul 29 07:20:59.065564 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:59.065547    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-aab086-jpsck"
Jul 29 07:20:59.067717 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:59.067694    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bj47g\""
Jul 29 07:20:59.071722 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:59.071700    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-aab086-jpsck"]
Jul 29 07:20:59.127294 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:59.127263    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67e3e7d-3238-418e-84d8-828150a38b43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-aab086-jpsck\" (UID: \"c67e3e7d-3238-418e-84d8-828150a38b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-aab086-jpsck"
Jul 29 07:20:59.228492 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:59.228459    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67e3e7d-3238-418e-84d8-828150a38b43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-aab086-jpsck\" (UID: \"c67e3e7d-3238-418e-84d8-828150a38b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-aab086-jpsck"
Jul 29 07:20:59.229113 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:59.229096    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67e3e7d-3238-418e-84d8-828150a38b43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-aab086-jpsck\" (UID: \"c67e3e7d-3238-418e-84d8-828150a38b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-aab086-jpsck"
Jul 29 07:20:59.381927 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:59.378207    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-aab086-jpsck"
Jul 29 07:20:59.521830 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:20:59.521804    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-aab086-jpsck"]
Jul 29 07:20:59.524458 ip-10-0-129-223 kubenswrapper[2563]: W0729 07:20:59.524430    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc67e3e7d_3238_418e_84d8_828150a38b43.slice/crio-4b63e00b865f09d78d5755a8ca534bc890244a27887f7bda6039005cc4c15de3 WatchSource:0}: Error finding container 4b63e00b865f09d78d5755a8ca534bc890244a27887f7bda6039005cc4c15de3: Status 404 returned error can't find the container with id 4b63e00b865f09d78d5755a8ca534bc890244a27887f7bda6039005cc4c15de3
Jul 29 07:21:00.392002 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:21:00.391969    2563 generic.go:356] "Generic (PLEG): container finished" podID="c67e3e7d-3238-418e-84d8-828150a38b43" containerID="223846001a54725e096bd951b6a9a3ab42f93f238110bd61d05a5d7ec60f0c09" exitCode=0
Jul 29 07:21:00.392386 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:21:00.392055    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-aab086-jpsck" event={"ID":"c67e3e7d-3238-418e-84d8-828150a38b43","Type":"ContainerDied","Data":"223846001a54725e096bd951b6a9a3ab42f93f238110bd61d05a5d7ec60f0c09"}
Jul 29 07:21:00.392386 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:21:00.392103    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-aab086-jpsck" event={"ID":"c67e3e7d-3238-418e-84d8-828150a38b43","Type":"ContainerStarted","Data":"4b63e00b865f09d78d5755a8ca534bc890244a27887f7bda6039005cc4c15de3"}
Jul 29 07:21:01.533522 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:21:01.533500    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-aab086-jpsck"
Jul 29 07:21:01.551229 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:21:01.551112    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67e3e7d-3238-418e-84d8-828150a38b43-maas-api-service-ca\") pod \"c67e3e7d-3238-418e-84d8-828150a38b43\" (UID: \"c67e3e7d-3238-418e-84d8-828150a38b43\") "
Jul 29 07:21:01.551658 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:21:01.551630    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c67e3e7d-3238-418e-84d8-828150a38b43-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c67e3e7d-3238-418e-84d8-828150a38b43" (UID: "c67e3e7d-3238-418e-84d8-828150a38b43"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 07:21:01.652287 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:21:01.652210    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67e3e7d-3238-418e-84d8-828150a38b43-maas-api-service-ca\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:21:02.404042 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:21:02.404012    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-aab086-jpsck"
Jul 29 07:21:02.404217 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:21:02.404053    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-aab086-jpsck" event={"ID":"c67e3e7d-3238-418e-84d8-828150a38b43","Type":"ContainerDied","Data":"4b63e00b865f09d78d5755a8ca534bc890244a27887f7bda6039005cc4c15de3"}
Jul 29 07:21:02.404217 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:21:02.404094    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b63e00b865f09d78d5755a8ca534bc890244a27887f7bda6039005cc4c15de3"
Jul 29 07:21:14.518845 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:21:14.518807    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab086-664f494947-7v674"]
Jul 29 07:21:14.519342 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:21:14.519071    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab086-664f494947-7v674" podUID="845cff92-9c98-4a83-afeb-e7360e562f06" containerName="maas-api" containerID="cri-o://30a9ca4e47dfce0f10af272b1f68f5caa7ff3831f7160124ce3ea5955f006b24" gracePeriod=30
Jul 29 07:21:14.750203 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:21:14.750165    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-aab086-jpsck"]
Jul 29 07:21:14.752214 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:21:14.752192    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-aab086-jpsck"]
Jul 29 07:21:14.782978 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:21:14.782915    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab086-664f494947-7v674"
Jul 29 07:21:14.874211 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:21:14.874177    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/845cff92-9c98-4a83-afeb-e7360e562f06-openshift-service-ca\") pod \"845cff92-9c98-4a83-afeb-e7360e562f06\" (UID: \"845cff92-9c98-4a83-afeb-e7360e562f06\") "
Jul 29 07:21:14.874378 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:21:14.874301    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/845cff92-9c98-4a83-afeb-e7360e562f06-maas-api-tls\") pod \"845cff92-9c98-4a83-afeb-e7360e562f06\" (UID: \"845cff92-9c98-4a83-afeb-e7360e562f06\") "
Jul 29 07:21:14.874378 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:21:14.874325    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqvlq\" (UniqueName: \"kubernetes.io/projected/845cff92-9c98-4a83-afeb-e7360e562f06-kube-api-access-sqvlq\") pod \"845cff92-9c98-4a83-afeb-e7360e562f06\" (UID: \"845cff92-9c98-4a83-afeb-e7360e562f06\") "
Jul 29 07:21:14.874554 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:21:14.874534    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/845cff92-9c98-4a83-afeb-e7360e562f06-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "845cff92-9c98-4a83-afeb-e7360e562f06" (UID: "845cff92-9c98-4a83-afeb-e7360e562f06"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 07:21:14.876620 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:21:14.876597    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/845cff92-9c98-4a83-afeb-e7360e562f06-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "845cff92-9c98-4a83-afeb-e7360e562f06" (UID: "845cff92-9c98-4a83-afeb-e7360e562f06"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 07:21:14.876717 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:21:14.876698    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/845cff92-9c98-4a83-afeb-e7360e562f06-kube-api-access-sqvlq" (OuterVolumeSpecName: "kube-api-access-sqvlq") pod "845cff92-9c98-4a83-afeb-e7360e562f06" (UID: "845cff92-9c98-4a83-afeb-e7360e562f06"). InnerVolumeSpecName "kube-api-access-sqvlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 07:21:14.975392 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:21:14.975355    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/845cff92-9c98-4a83-afeb-e7360e562f06-openshift-service-ca\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:21:14.975392 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:21:14.975385    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/845cff92-9c98-4a83-afeb-e7360e562f06-maas-api-tls\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:21:14.975392 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:21:14.975394    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqvlq\" (UniqueName: \"kubernetes.io/projected/845cff92-9c98-4a83-afeb-e7360e562f06-kube-api-access-sqvlq\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:21:15.399019 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:21:15.398980    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c67e3e7d-3238-418e-84d8-828150a38b43" path="/var/lib/kubelet/pods/c67e3e7d-3238-418e-84d8-828150a38b43/volumes"
Jul 29 07:21:15.475514 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:21:15.475480    2563 generic.go:356] "Generic (PLEG): container finished" podID="845cff92-9c98-4a83-afeb-e7360e562f06" containerID="30a9ca4e47dfce0f10af272b1f68f5caa7ff3831f7160124ce3ea5955f006b24" exitCode=0
Jul 29 07:21:15.475706 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:21:15.475529    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab086-664f494947-7v674"
Jul 29 07:21:15.475706 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:21:15.475570    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab086-664f494947-7v674" event={"ID":"845cff92-9c98-4a83-afeb-e7360e562f06","Type":"ContainerDied","Data":"30a9ca4e47dfce0f10af272b1f68f5caa7ff3831f7160124ce3ea5955f006b24"}
Jul 29 07:21:15.475706 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:21:15.475629    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab086-664f494947-7v674" event={"ID":"845cff92-9c98-4a83-afeb-e7360e562f06","Type":"ContainerDied","Data":"9e3629732963eabf803c3714de7e9623300b0dde5df0e79220678ee08d12f6ff"}
Jul 29 07:21:15.475706 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:21:15.475668    2563 scope.go:117] "RemoveContainer" containerID="30a9ca4e47dfce0f10af272b1f68f5caa7ff3831f7160124ce3ea5955f006b24"
Jul 29 07:21:15.485263 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:21:15.485240    2563 scope.go:117] "RemoveContainer" containerID="30a9ca4e47dfce0f10af272b1f68f5caa7ff3831f7160124ce3ea5955f006b24"
Jul 29 07:21:15.485519 ip-10-0-129-223 kubenswrapper[2563]: E0729 07:21:15.485500    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30a9ca4e47dfce0f10af272b1f68f5caa7ff3831f7160124ce3ea5955f006b24\": container with ID starting with 30a9ca4e47dfce0f10af272b1f68f5caa7ff3831f7160124ce3ea5955f006b24 not found: ID does not exist" containerID="30a9ca4e47dfce0f10af272b1f68f5caa7ff3831f7160124ce3ea5955f006b24"
Jul 29 07:21:15.485601 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:21:15.485525    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30a9ca4e47dfce0f10af272b1f68f5caa7ff3831f7160124ce3ea5955f006b24"} err="failed to get container status \"30a9ca4e47dfce0f10af272b1f68f5caa7ff3831f7160124ce3ea5955f006b24\": rpc error: code = NotFound desc = could not find container \"30a9ca4e47dfce0f10af272b1f68f5caa7ff3831f7160124ce3ea5955f006b24\": container with ID starting with 30a9ca4e47dfce0f10af272b1f68f5caa7ff3831f7160124ce3ea5955f006b24 not found: ID does not exist"
Jul 29 07:21:15.491923 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:21:15.491904    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab086-664f494947-7v674"]
Jul 29 07:21:15.495081 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:21:15.495060    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-aab086-664f494947-7v674"]
Jul 29 07:21:17.401566 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:21:17.401527    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="845cff92-9c98-4a83-afeb-e7360e562f06" path="/var/lib/kubelet/pods/845cff92-9c98-4a83-afeb-e7360e562f06/volumes"
Jul 29 07:22:08.873660 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:08.873623    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9"]
Jul 29 07:22:08.874169 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:08.873980    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9" podUID="c89b4f41-723e-4e60-adb1-7788889aa156" containerName="maas-api" containerID="cri-o://424df5e1fc3c166f37daca08a00e52af193914d4a7af63d36398c9fa193b8a87" gracePeriod=30
Jul 29 07:22:08.920397 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:08.920365    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-04c3b9-5f58c78b7d-wf6ws"]
Jul 29 07:22:08.920675 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:08.920630    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-04c3b9-5f58c78b7d-wf6ws" podUID="dd5e5f16-bec9-48a4-80f3-73662f2ba97c" containerName="payload-processing" containerID="cri-o://d62ecf985c860272b0c134301c0cfec42926ea83b2980cefe208b552a7310a2b" gracePeriod=30
Jul 29 07:22:08.929096 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:08.929071    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-04c3b9-5554d46df6-fnq6c"]
Jul 29 07:22:08.929660 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:08.929617    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-04c3b9-5554d46df6-fnq6c" podUID="e2713431-5709-4c72-87b3-5b8b37a7816d" containerName="payload-pre-processing" containerID="cri-o://3e0ab1273ae936e223044dc8ce5a0f241db97dda37a7d635d72299fc31cb0831" gracePeriod=30
Jul 29 07:22:09.249492 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.249467    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-04c3b9-5f58c78b7d-wf6ws"
Jul 29 07:22:09.252059 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.252038    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9"
Jul 29 07:22:09.255679 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.255659    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-04c3b9-5554d46df6-fnq6c"
Jul 29 07:22:09.400630 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.400603    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd5e5f16-bec9-48a4-80f3-73662f2ba97c-plugins-config-volume\") pod \"dd5e5f16-bec9-48a4-80f3-73662f2ba97c\" (UID: \"dd5e5f16-bec9-48a4-80f3-73662f2ba97c\") "
Jul 29 07:22:09.400823 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.400642    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2713431-5709-4c72-87b3-5b8b37a7816d-plugins-config-volume\") pod \"e2713431-5709-4c72-87b3-5b8b37a7816d\" (UID: \"e2713431-5709-4c72-87b3-5b8b37a7816d\") "
Jul 29 07:22:09.400823 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.400796    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxlgw\" (UniqueName: \"kubernetes.io/projected/dd5e5f16-bec9-48a4-80f3-73662f2ba97c-kube-api-access-xxlgw\") pod \"dd5e5f16-bec9-48a4-80f3-73662f2ba97c\" (UID: \"dd5e5f16-bec9-48a4-80f3-73662f2ba97c\") "
Jul 29 07:22:09.400972 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.400841    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgbkz\" (UniqueName: \"kubernetes.io/projected/c89b4f41-723e-4e60-adb1-7788889aa156-kube-api-access-lgbkz\") pod \"c89b4f41-723e-4e60-adb1-7788889aa156\" (UID: \"c89b4f41-723e-4e60-adb1-7788889aa156\") "
Jul 29 07:22:09.400972 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.400938    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cb8hr\" (UniqueName: \"kubernetes.io/projected/e2713431-5709-4c72-87b3-5b8b37a7816d-kube-api-access-cb8hr\") pod \"e2713431-5709-4c72-87b3-5b8b37a7816d\" (UID: \"e2713431-5709-4c72-87b3-5b8b37a7816d\") "
Jul 29 07:22:09.401086 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.400973    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c89b4f41-723e-4e60-adb1-7788889aa156-maas-api-tls\") pod \"c89b4f41-723e-4e60-adb1-7788889aa156\" (UID: \"c89b4f41-723e-4e60-adb1-7788889aa156\") "
Jul 29 07:22:09.401086 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.401006    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89b4f41-723e-4e60-adb1-7788889aa156-openshift-service-ca\") pod \"c89b4f41-723e-4e60-adb1-7788889aa156\" (UID: \"c89b4f41-723e-4e60-adb1-7788889aa156\") "
Jul 29 07:22:09.401086 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.401069    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd5e5f16-bec9-48a4-80f3-73662f2ba97c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dd5e5f16-bec9-48a4-80f3-73662f2ba97c" (UID: "dd5e5f16-bec9-48a4-80f3-73662f2ba97c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 07:22:09.401306 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.401284    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd5e5f16-bec9-48a4-80f3-73662f2ba97c-plugins-config-volume\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:22:09.401390 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.401324    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2713431-5709-4c72-87b3-5b8b37a7816d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e2713431-5709-4c72-87b3-5b8b37a7816d" (UID: "e2713431-5709-4c72-87b3-5b8b37a7816d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 07:22:09.401785 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.401696    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c89b4f41-723e-4e60-adb1-7788889aa156-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c89b4f41-723e-4e60-adb1-7788889aa156" (UID: "c89b4f41-723e-4e60-adb1-7788889aa156"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 07:22:09.403280 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.403249    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2713431-5709-4c72-87b3-5b8b37a7816d-kube-api-access-cb8hr" (OuterVolumeSpecName: "kube-api-access-cb8hr") pod "e2713431-5709-4c72-87b3-5b8b37a7816d" (UID: "e2713431-5709-4c72-87b3-5b8b37a7816d"). InnerVolumeSpecName "kube-api-access-cb8hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 07:22:09.403371 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.403262    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c89b4f41-723e-4e60-adb1-7788889aa156-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c89b4f41-723e-4e60-adb1-7788889aa156" (UID: "c89b4f41-723e-4e60-adb1-7788889aa156"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 07:22:09.403371 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.403315    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd5e5f16-bec9-48a4-80f3-73662f2ba97c-kube-api-access-xxlgw" (OuterVolumeSpecName: "kube-api-access-xxlgw") pod "dd5e5f16-bec9-48a4-80f3-73662f2ba97c" (UID: "dd5e5f16-bec9-48a4-80f3-73662f2ba97c"). InnerVolumeSpecName "kube-api-access-xxlgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 07:22:09.403371 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.403327    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c89b4f41-723e-4e60-adb1-7788889aa156-kube-api-access-lgbkz" (OuterVolumeSpecName: "kube-api-access-lgbkz") pod "c89b4f41-723e-4e60-adb1-7788889aa156" (UID: "c89b4f41-723e-4e60-adb1-7788889aa156"). InnerVolumeSpecName "kube-api-access-lgbkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 07:22:09.502792 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.502707    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxlgw\" (UniqueName: \"kubernetes.io/projected/dd5e5f16-bec9-48a4-80f3-73662f2ba97c-kube-api-access-xxlgw\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:22:09.502792 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.502737    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgbkz\" (UniqueName: \"kubernetes.io/projected/c89b4f41-723e-4e60-adb1-7788889aa156-kube-api-access-lgbkz\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:22:09.502792 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.502747    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb8hr\" (UniqueName: \"kubernetes.io/projected/e2713431-5709-4c72-87b3-5b8b37a7816d-kube-api-access-cb8hr\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:22:09.502792 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.502756    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c89b4f41-723e-4e60-adb1-7788889aa156-maas-api-tls\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:22:09.502792 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.502766    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89b4f41-723e-4e60-adb1-7788889aa156-openshift-service-ca\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:22:09.502792 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.502776    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2713431-5709-4c72-87b3-5b8b37a7816d-plugins-config-volume\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:22:09.770265 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.770175    2563 generic.go:356] "Generic (PLEG): container finished" podID="dd5e5f16-bec9-48a4-80f3-73662f2ba97c" containerID="d62ecf985c860272b0c134301c0cfec42926ea83b2980cefe208b552a7310a2b" exitCode=0
Jul 29 07:22:09.770265 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.770221    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-04c3b9-5f58c78b7d-wf6ws"
Jul 29 07:22:09.770265 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.770257    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-04c3b9-5f58c78b7d-wf6ws" event={"ID":"dd5e5f16-bec9-48a4-80f3-73662f2ba97c","Type":"ContainerDied","Data":"d62ecf985c860272b0c134301c0cfec42926ea83b2980cefe208b552a7310a2b"}
Jul 29 07:22:09.770546 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.770308    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-04c3b9-5f58c78b7d-wf6ws" event={"ID":"dd5e5f16-bec9-48a4-80f3-73662f2ba97c","Type":"ContainerDied","Data":"35db4d13ccad93c47873644cc2c491c20a057df8ea0ee3d76b4b478185065294"}
Jul 29 07:22:09.770546 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.770326    2563 scope.go:117] "RemoveContainer" containerID="d62ecf985c860272b0c134301c0cfec42926ea83b2980cefe208b552a7310a2b"
Jul 29 07:22:09.774476 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.774450    2563 generic.go:356] "Generic (PLEG): container finished" podID="e2713431-5709-4c72-87b3-5b8b37a7816d" containerID="3e0ab1273ae936e223044dc8ce5a0f241db97dda37a7d635d72299fc31cb0831" exitCode=0
Jul 29 07:22:09.774601 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.774512    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-04c3b9-5554d46df6-fnq6c"
Jul 29 07:22:09.774601 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.774532    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-04c3b9-5554d46df6-fnq6c" event={"ID":"e2713431-5709-4c72-87b3-5b8b37a7816d","Type":"ContainerDied","Data":"3e0ab1273ae936e223044dc8ce5a0f241db97dda37a7d635d72299fc31cb0831"}
Jul 29 07:22:09.774601 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.774592    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-04c3b9-5554d46df6-fnq6c" event={"ID":"e2713431-5709-4c72-87b3-5b8b37a7816d","Type":"ContainerDied","Data":"75dc621a6cbedbde8fc8f44bf8ae75b69cc620179c77f61d6a0e7ab6c9383c7c"}
Jul 29 07:22:09.776130 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.776109    2563 generic.go:356] "Generic (PLEG): container finished" podID="c89b4f41-723e-4e60-adb1-7788889aa156" containerID="424df5e1fc3c166f37daca08a00e52af193914d4a7af63d36398c9fa193b8a87" exitCode=0
Jul 29 07:22:09.776250 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.776161    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9"
Jul 29 07:22:09.776250 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.776189    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9" event={"ID":"c89b4f41-723e-4e60-adb1-7788889aa156","Type":"ContainerDied","Data":"424df5e1fc3c166f37daca08a00e52af193914d4a7af63d36398c9fa193b8a87"}
Jul 29 07:22:09.776250 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.776234    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9" event={"ID":"c89b4f41-723e-4e60-adb1-7788889aa156","Type":"ContainerDied","Data":"45532473a6697ff5700889e566d98acc8ca5a475f7e2e1e3d9d988919fb04389"}
Jul 29 07:22:09.783380 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.783355    2563 scope.go:117] "RemoveContainer" containerID="d62ecf985c860272b0c134301c0cfec42926ea83b2980cefe208b552a7310a2b"
Jul 29 07:22:09.783674 ip-10-0-129-223 kubenswrapper[2563]: E0729 07:22:09.783648    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d62ecf985c860272b0c134301c0cfec42926ea83b2980cefe208b552a7310a2b\": container with ID starting with d62ecf985c860272b0c134301c0cfec42926ea83b2980cefe208b552a7310a2b not found: ID does not exist" containerID="d62ecf985c860272b0c134301c0cfec42926ea83b2980cefe208b552a7310a2b"
Jul 29 07:22:09.783766 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.783686    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d62ecf985c860272b0c134301c0cfec42926ea83b2980cefe208b552a7310a2b"} err="failed to get container status \"d62ecf985c860272b0c134301c0cfec42926ea83b2980cefe208b552a7310a2b\": rpc error: code = NotFound desc = could not find container \"d62ecf985c860272b0c134301c0cfec42926ea83b2980cefe208b552a7310a2b\": container with ID starting with d62ecf985c860272b0c134301c0cfec42926ea83b2980cefe208b552a7310a2b not found: ID does not exist"
Jul 29 07:22:09.783766 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.783711    2563 scope.go:117] "RemoveContainer" containerID="3e0ab1273ae936e223044dc8ce5a0f241db97dda37a7d635d72299fc31cb0831"
Jul 29 07:22:09.792263 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.792245    2563 scope.go:117] "RemoveContainer" containerID="3e0ab1273ae936e223044dc8ce5a0f241db97dda37a7d635d72299fc31cb0831"
Jul 29 07:22:09.792501 ip-10-0-129-223 kubenswrapper[2563]: E0729 07:22:09.792472    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e0ab1273ae936e223044dc8ce5a0f241db97dda37a7d635d72299fc31cb0831\": container with ID starting with 3e0ab1273ae936e223044dc8ce5a0f241db97dda37a7d635d72299fc31cb0831 not found: ID does not exist" containerID="3e0ab1273ae936e223044dc8ce5a0f241db97dda37a7d635d72299fc31cb0831"
Jul 29 07:22:09.792553 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.792499    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e0ab1273ae936e223044dc8ce5a0f241db97dda37a7d635d72299fc31cb0831"} err="failed to get container status \"3e0ab1273ae936e223044dc8ce5a0f241db97dda37a7d635d72299fc31cb0831\": rpc error: code = NotFound desc = could not find container \"3e0ab1273ae936e223044dc8ce5a0f241db97dda37a7d635d72299fc31cb0831\": container with ID starting with 3e0ab1273ae936e223044dc8ce5a0f241db97dda37a7d635d72299fc31cb0831 not found: ID does not exist"
Jul 29 07:22:09.792553 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.792518    2563 scope.go:117] "RemoveContainer" containerID="424df5e1fc3c166f37daca08a00e52af193914d4a7af63d36398c9fa193b8a87"
Jul 29 07:22:09.798766 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.798739    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-04c3b9-5f58c78b7d-wf6ws"]
Jul 29 07:22:09.801830 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.801816    2563 scope.go:117] "RemoveContainer" containerID="424df5e1fc3c166f37daca08a00e52af193914d4a7af63d36398c9fa193b8a87"
Jul 29 07:22:09.802160 ip-10-0-129-223 kubenswrapper[2563]: E0729 07:22:09.802140    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"424df5e1fc3c166f37daca08a00e52af193914d4a7af63d36398c9fa193b8a87\": container with ID starting with 424df5e1fc3c166f37daca08a00e52af193914d4a7af63d36398c9fa193b8a87 not found: ID does not exist" containerID="424df5e1fc3c166f37daca08a00e52af193914d4a7af63d36398c9fa193b8a87"
Jul 29 07:22:09.802220 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.802168    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"424df5e1fc3c166f37daca08a00e52af193914d4a7af63d36398c9fa193b8a87"} err="failed to get container status \"424df5e1fc3c166f37daca08a00e52af193914d4a7af63d36398c9fa193b8a87\": rpc error: code = NotFound desc = could not find container \"424df5e1fc3c166f37daca08a00e52af193914d4a7af63d36398c9fa193b8a87\": container with ID starting with 424df5e1fc3c166f37daca08a00e52af193914d4a7af63d36398c9fa193b8a87 not found: ID does not exist"
Jul 29 07:22:09.810946 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.810915    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-04c3b9-5f58c78b7d-wf6ws"]
Jul 29 07:22:09.821838 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.821814    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9"]
Jul 29 07:22:09.825460 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.825439    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-04c3b9-84df796f7c-77pl9"]
Jul 29 07:22:09.836991 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.836969    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-04c3b9-5554d46df6-fnq6c"]
Jul 29 07:22:09.841102 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:09.841082    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-04c3b9-5554d46df6-fnq6c"]
Jul 29 07:22:11.397608 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:11.397575    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c89b4f41-723e-4e60-adb1-7788889aa156" path="/var/lib/kubelet/pods/c89b4f41-723e-4e60-adb1-7788889aa156/volumes"
Jul 29 07:22:11.398017 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:11.397954    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd5e5f16-bec9-48a4-80f3-73662f2ba97c" path="/var/lib/kubelet/pods/dd5e5f16-bec9-48a4-80f3-73662f2ba97c/volumes"
Jul 29 07:22:11.398271 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:11.398259    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2713431-5709-4c72-87b3-5b8b37a7816d" path="/var/lib/kubelet/pods/e2713431-5709-4c72-87b3-5b8b37a7816d/volumes"
Jul 29 07:22:20.835486 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:20.835453    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z"]
Jul 29 07:22:20.836105 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:20.835699    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z" podUID="7b711f5d-bd42-47d1-a020-1e24dd0d4f2b" containerName="maas-api" containerID="cri-o://2dbf40434aed9140e0899c5447173bf53820c0ce91ce081a793ba96297ba0090" gracePeriod=30
Jul 29 07:22:20.971019 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:20.970984    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-50e693-55cff4bdd8-2mtt4"]
Jul 29 07:22:20.971311 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:20.971250    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-50e693-55cff4bdd8-2mtt4" podUID="102edb4c-50b3-47d1-ac60-71e41b1df80d" containerName="payload-pre-processing" containerID="cri-o://23501ca20b3c97fd08f12d67e9df473901b465fbda6060692f35c2ff9e6e2fa2" gracePeriod=30
Jul 29 07:22:21.094432 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:21.094410    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z"
Jul 29 07:22:21.112472 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:21.112365    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b711f5d-bd42-47d1-a020-1e24dd0d4f2b-maas-api-tls\") pod \"7b711f5d-bd42-47d1-a020-1e24dd0d4f2b\" (UID: \"7b711f5d-bd42-47d1-a020-1e24dd0d4f2b\") "
Jul 29 07:22:21.112665 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:21.112473    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jq8tc\" (UniqueName: \"kubernetes.io/projected/7b711f5d-bd42-47d1-a020-1e24dd0d4f2b-kube-api-access-jq8tc\") pod \"7b711f5d-bd42-47d1-a020-1e24dd0d4f2b\" (UID: \"7b711f5d-bd42-47d1-a020-1e24dd0d4f2b\") "
Jul 29 07:22:21.112665 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:21.112528    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b711f5d-bd42-47d1-a020-1e24dd0d4f2b-openshift-service-ca\") pod \"7b711f5d-bd42-47d1-a020-1e24dd0d4f2b\" (UID: \"7b711f5d-bd42-47d1-a020-1e24dd0d4f2b\") "
Jul 29 07:22:21.113255 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:21.113227    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b711f5d-bd42-47d1-a020-1e24dd0d4f2b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7b711f5d-bd42-47d1-a020-1e24dd0d4f2b" (UID: "7b711f5d-bd42-47d1-a020-1e24dd0d4f2b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 07:22:21.116089 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:21.116053    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b711f5d-bd42-47d1-a020-1e24dd0d4f2b-kube-api-access-jq8tc" (OuterVolumeSpecName: "kube-api-access-jq8tc") pod "7b711f5d-bd42-47d1-a020-1e24dd0d4f2b" (UID: "7b711f5d-bd42-47d1-a020-1e24dd0d4f2b"). InnerVolumeSpecName "kube-api-access-jq8tc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 07:22:21.116331 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:21.116307    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b711f5d-bd42-47d1-a020-1e24dd0d4f2b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7b711f5d-bd42-47d1-a020-1e24dd0d4f2b" (UID: "7b711f5d-bd42-47d1-a020-1e24dd0d4f2b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 07:22:21.213954 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:21.213922    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq8tc\" (UniqueName: \"kubernetes.io/projected/7b711f5d-bd42-47d1-a020-1e24dd0d4f2b-kube-api-access-jq8tc\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:22:21.213954 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:21.213950    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b711f5d-bd42-47d1-a020-1e24dd0d4f2b-openshift-service-ca\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:22:21.213954 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:21.213963    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b711f5d-bd42-47d1-a020-1e24dd0d4f2b-maas-api-tls\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:22:21.234709 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:21.234685    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-50e693-55cff4bdd8-2mtt4"
Jul 29 07:22:21.315010 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:21.314985    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/102edb4c-50b3-47d1-ac60-71e41b1df80d-plugins-config-volume\") pod \"102edb4c-50b3-47d1-ac60-71e41b1df80d\" (UID: \"102edb4c-50b3-47d1-ac60-71e41b1df80d\") "
Jul 29 07:22:21.315207 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:21.315034    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9b5n\" (UniqueName: \"kubernetes.io/projected/102edb4c-50b3-47d1-ac60-71e41b1df80d-kube-api-access-b9b5n\") pod \"102edb4c-50b3-47d1-ac60-71e41b1df80d\" (UID: \"102edb4c-50b3-47d1-ac60-71e41b1df80d\") "
Jul 29 07:22:21.315392 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:21.315368    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/102edb4c-50b3-47d1-ac60-71e41b1df80d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "102edb4c-50b3-47d1-ac60-71e41b1df80d" (UID: "102edb4c-50b3-47d1-ac60-71e41b1df80d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 07:22:21.317207 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:21.317186    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/102edb4c-50b3-47d1-ac60-71e41b1df80d-kube-api-access-b9b5n" (OuterVolumeSpecName: "kube-api-access-b9b5n") pod "102edb4c-50b3-47d1-ac60-71e41b1df80d" (UID: "102edb4c-50b3-47d1-ac60-71e41b1df80d"). InnerVolumeSpecName "kube-api-access-b9b5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 07:22:21.416635 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:21.416609    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/102edb4c-50b3-47d1-ac60-71e41b1df80d-plugins-config-volume\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:22:21.416635 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:21.416637    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9b5n\" (UniqueName: \"kubernetes.io/projected/102edb4c-50b3-47d1-ac60-71e41b1df80d-kube-api-access-b9b5n\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:22:21.847402 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:21.847367    2563 generic.go:356] "Generic (PLEG): container finished" podID="7b711f5d-bd42-47d1-a020-1e24dd0d4f2b" containerID="2dbf40434aed9140e0899c5447173bf53820c0ce91ce081a793ba96297ba0090" exitCode=0
Jul 29 07:22:21.847879 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:21.847409    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z"
Jul 29 07:22:21.847879 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:21.847449    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z" event={"ID":"7b711f5d-bd42-47d1-a020-1e24dd0d4f2b","Type":"ContainerDied","Data":"2dbf40434aed9140e0899c5447173bf53820c0ce91ce081a793ba96297ba0090"}
Jul 29 07:22:21.847879 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:21.847501    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z" event={"ID":"7b711f5d-bd42-47d1-a020-1e24dd0d4f2b","Type":"ContainerDied","Data":"c9275b3a1b76487fdd996dc271571d120c0ba79460cba697879aa5f7ccb28fde"}
Jul 29 07:22:21.847879 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:21.847531    2563 scope.go:117] "RemoveContainer" containerID="2dbf40434aed9140e0899c5447173bf53820c0ce91ce081a793ba96297ba0090"
Jul 29 07:22:21.849031 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:21.849004    2563 generic.go:356] "Generic (PLEG): container finished" podID="102edb4c-50b3-47d1-ac60-71e41b1df80d" containerID="23501ca20b3c97fd08f12d67e9df473901b465fbda6060692f35c2ff9e6e2fa2" exitCode=0
Jul 29 07:22:21.849096 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:21.849055    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-50e693-55cff4bdd8-2mtt4" event={"ID":"102edb4c-50b3-47d1-ac60-71e41b1df80d","Type":"ContainerDied","Data":"23501ca20b3c97fd08f12d67e9df473901b465fbda6060692f35c2ff9e6e2fa2"}
Jul 29 07:22:21.849096 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:21.849093    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-50e693-55cff4bdd8-2mtt4" event={"ID":"102edb4c-50b3-47d1-ac60-71e41b1df80d","Type":"ContainerDied","Data":"967ee6bf6dc72d7255268b8b109519adb8b09d9d7c90f122e144d2926ad9036d"}
Jul 29 07:22:21.849169 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:21.849055    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-50e693-55cff4bdd8-2mtt4"
Jul 29 07:22:21.861723 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:21.861704    2563 scope.go:117] "RemoveContainer" containerID="2dbf40434aed9140e0899c5447173bf53820c0ce91ce081a793ba96297ba0090"
Jul 29 07:22:21.862057 ip-10-0-129-223 kubenswrapper[2563]: E0729 07:22:21.862029    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dbf40434aed9140e0899c5447173bf53820c0ce91ce081a793ba96297ba0090\": container with ID starting with 2dbf40434aed9140e0899c5447173bf53820c0ce91ce081a793ba96297ba0090 not found: ID does not exist" containerID="2dbf40434aed9140e0899c5447173bf53820c0ce91ce081a793ba96297ba0090"
Jul 29 07:22:21.862159 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:21.862067    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dbf40434aed9140e0899c5447173bf53820c0ce91ce081a793ba96297ba0090"} err="failed to get container status \"2dbf40434aed9140e0899c5447173bf53820c0ce91ce081a793ba96297ba0090\": rpc error: code = NotFound desc = could not find container \"2dbf40434aed9140e0899c5447173bf53820c0ce91ce081a793ba96297ba0090\": container with ID starting with 2dbf40434aed9140e0899c5447173bf53820c0ce91ce081a793ba96297ba0090 not found: ID does not exist"
Jul 29 07:22:21.862159 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:21.862093    2563 scope.go:117] "RemoveContainer" containerID="23501ca20b3c97fd08f12d67e9df473901b465fbda6060692f35c2ff9e6e2fa2"
Jul 29 07:22:21.870725 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:21.870709    2563 scope.go:117] "RemoveContainer" containerID="23501ca20b3c97fd08f12d67e9df473901b465fbda6060692f35c2ff9e6e2fa2"
Jul 29 07:22:21.871013 ip-10-0-129-223 kubenswrapper[2563]: E0729 07:22:21.870992    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23501ca20b3c97fd08f12d67e9df473901b465fbda6060692f35c2ff9e6e2fa2\": container with ID starting with 23501ca20b3c97fd08f12d67e9df473901b465fbda6060692f35c2ff9e6e2fa2 not found: ID does not exist" containerID="23501ca20b3c97fd08f12d67e9df473901b465fbda6060692f35c2ff9e6e2fa2"
Jul 29 07:22:21.871065 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:21.871023    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23501ca20b3c97fd08f12d67e9df473901b465fbda6060692f35c2ff9e6e2fa2"} err="failed to get container status \"23501ca20b3c97fd08f12d67e9df473901b465fbda6060692f35c2ff9e6e2fa2\": rpc error: code = NotFound desc = could not find container \"23501ca20b3c97fd08f12d67e9df473901b465fbda6060692f35c2ff9e6e2fa2\": container with ID starting with 23501ca20b3c97fd08f12d67e9df473901b465fbda6060692f35c2ff9e6e2fa2 not found: ID does not exist"
Jul 29 07:22:21.871940 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:21.871919    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-50e693-55cff4bdd8-2mtt4"]
Jul 29 07:22:21.877305 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:21.877286    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-50e693-55cff4bdd8-2mtt4"]
Jul 29 07:22:21.886375 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:21.886355    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z"]
Jul 29 07:22:21.890205 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:21.890186    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-50e693-747cbc8cd7-zw47z"]
Jul 29 07:22:23.397626 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:23.397593    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="102edb4c-50b3-47d1-ac60-71e41b1df80d" path="/var/lib/kubelet/pods/102edb4c-50b3-47d1-ac60-71e41b1df80d/volumes"
Jul 29 07:22:23.398014 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:23.397997    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b711f5d-bd42-47d1-a020-1e24dd0d4f2b" path="/var/lib/kubelet/pods/7b711f5d-bd42-47d1-a020-1e24dd0d4f2b/volumes"
Jul 29 07:22:28.044500 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:28.044425    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1e6a9b-sg8lq"]
Jul 29 07:22:28.044912 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:28.044829    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b711f5d-bd42-47d1-a020-1e24dd0d4f2b" containerName="maas-api"
Jul 29 07:22:28.044912 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:28.044841    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b711f5d-bd42-47d1-a020-1e24dd0d4f2b" containerName="maas-api"
Jul 29 07:22:28.044912 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:28.044867    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c89b4f41-723e-4e60-adb1-7788889aa156" containerName="maas-api"
Jul 29 07:22:28.044912 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:28.044873    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c89b4f41-723e-4e60-adb1-7788889aa156" containerName="maas-api"
Jul 29 07:22:28.044912 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:28.044882    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="102edb4c-50b3-47d1-ac60-71e41b1df80d" containerName="payload-pre-processing"
Jul 29 07:22:28.044912 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:28.044887    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="102edb4c-50b3-47d1-ac60-71e41b1df80d" containerName="payload-pre-processing"
Jul 29 07:22:28.044912 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:28.044898    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="845cff92-9c98-4a83-afeb-e7360e562f06" containerName="maas-api"
Jul 29 07:22:28.044912 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:28.044903    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="845cff92-9c98-4a83-afeb-e7360e562f06" containerName="maas-api"
Jul 29 07:22:28.044912 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:28.044911    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd5e5f16-bec9-48a4-80f3-73662f2ba97c" containerName="payload-processing"
Jul 29 07:22:28.044912 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:28.044917    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd5e5f16-bec9-48a4-80f3-73662f2ba97c" containerName="payload-processing"
Jul 29 07:22:28.045266 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:28.044924    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2713431-5709-4c72-87b3-5b8b37a7816d" containerName="payload-pre-processing"
Jul 29 07:22:28.045266 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:28.044929    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2713431-5709-4c72-87b3-5b8b37a7816d" containerName="payload-pre-processing"
Jul 29 07:22:28.045266 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:28.044937    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c67e3e7d-3238-418e-84d8-828150a38b43" containerName="revoke-keys"
Jul 29 07:22:28.045266 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:28.044942    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c67e3e7d-3238-418e-84d8-828150a38b43" containerName="revoke-keys"
Jul 29 07:22:28.045266 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:28.045007    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2713431-5709-4c72-87b3-5b8b37a7816d" containerName="payload-pre-processing"
Jul 29 07:22:28.045266 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:28.045013    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd5e5f16-bec9-48a4-80f3-73662f2ba97c" containerName="payload-processing"
Jul 29 07:22:28.045266 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:28.045020    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c89b4f41-723e-4e60-adb1-7788889aa156" containerName="maas-api"
Jul 29 07:22:28.045266 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:28.045025    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="845cff92-9c98-4a83-afeb-e7360e562f06" containerName="maas-api"
Jul 29 07:22:28.045266 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:28.045032    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="102edb4c-50b3-47d1-ac60-71e41b1df80d" containerName="payload-pre-processing"
Jul 29 07:22:28.045266 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:28.045037    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c67e3e7d-3238-418e-84d8-828150a38b43" containerName="revoke-keys"
Jul 29 07:22:28.045266 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:28.045042    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b711f5d-bd42-47d1-a020-1e24dd0d4f2b" containerName="maas-api"
Jul 29 07:22:28.049828 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:28.049808    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1e6a9b-sg8lq"
Jul 29 07:22:28.052015 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:28.051993    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bj47g\""
Jul 29 07:22:28.055160 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:28.055137    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1e6a9b-sg8lq"]
Jul 29 07:22:28.078512 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:28.078485    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/558745d0-708d-4645-bb88-adf802fca19d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-1e6a9b-sg8lq\" (UID: \"558745d0-708d-4645-bb88-adf802fca19d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1e6a9b-sg8lq"
Jul 29 07:22:28.180017 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:28.179983    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/558745d0-708d-4645-bb88-adf802fca19d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-1e6a9b-sg8lq\" (UID: \"558745d0-708d-4645-bb88-adf802fca19d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1e6a9b-sg8lq"
Jul 29 07:22:28.180628 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:28.180604    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/558745d0-708d-4645-bb88-adf802fca19d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-1e6a9b-sg8lq\" (UID: \"558745d0-708d-4645-bb88-adf802fca19d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1e6a9b-sg8lq"
Jul 29 07:22:28.362724 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:28.362698    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1e6a9b-sg8lq"
Jul 29 07:22:28.502676 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:28.502649    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1e6a9b-sg8lq"]
Jul 29 07:22:28.504390 ip-10-0-129-223 kubenswrapper[2563]: W0729 07:22:28.504364    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod558745d0_708d_4645_bb88_adf802fca19d.slice/crio-51cd70e82920e51765c8a5d60d4a7f8acf889962812b59510e62981ff1312699 WatchSource:0}: Error finding container 51cd70e82920e51765c8a5d60d4a7f8acf889962812b59510e62981ff1312699: Status 404 returned error can't find the container with id 51cd70e82920e51765c8a5d60d4a7f8acf889962812b59510e62981ff1312699
Jul 29 07:22:28.888071 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:28.887990    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1e6a9b-sg8lq" event={"ID":"558745d0-708d-4645-bb88-adf802fca19d","Type":"ContainerStarted","Data":"39753ddc242720737132e315eec384219f54e0f528c85e889fc72025f3c715cf"}
Jul 29 07:22:28.888071 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:28.888042    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1e6a9b-sg8lq" event={"ID":"558745d0-708d-4645-bb88-adf802fca19d","Type":"ContainerStarted","Data":"51cd70e82920e51765c8a5d60d4a7f8acf889962812b59510e62981ff1312699"}
Jul 29 07:22:28.907379 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:28.907328    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1e6a9b-sg8lq" podStartSLOduration=0.907311812 podStartE2EDuration="907.311812ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 07:22:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:22:28.90444404 +0000 UTC m=+3591.982075809" watchObservedRunningTime="2026-07-29 07:22:28.907311812 +0000 UTC m=+3591.984943579"
Jul 29 07:22:29.895964 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:29.895928    2563 generic.go:356] "Generic (PLEG): container finished" podID="558745d0-708d-4645-bb88-adf802fca19d" containerID="39753ddc242720737132e315eec384219f54e0f528c85e889fc72025f3c715cf" exitCode=0
Jul 29 07:22:29.896331 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:29.896000    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1e6a9b-sg8lq" event={"ID":"558745d0-708d-4645-bb88-adf802fca19d","Type":"ContainerDied","Data":"39753ddc242720737132e315eec384219f54e0f528c85e889fc72025f3c715cf"}
Jul 29 07:22:31.036144 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:31.036120    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1e6a9b-sg8lq"
Jul 29 07:22:31.106745 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:31.106717    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/558745d0-708d-4645-bb88-adf802fca19d-maas-api-service-ca\") pod \"558745d0-708d-4645-bb88-adf802fca19d\" (UID: \"558745d0-708d-4645-bb88-adf802fca19d\") "
Jul 29 07:22:31.107108 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:31.107086    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/558745d0-708d-4645-bb88-adf802fca19d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "558745d0-708d-4645-bb88-adf802fca19d" (UID: "558745d0-708d-4645-bb88-adf802fca19d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 07:22:31.208204 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:31.208117    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/558745d0-708d-4645-bb88-adf802fca19d-maas-api-service-ca\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:22:31.907119 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:31.907082    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1e6a9b-sg8lq"
Jul 29 07:22:31.907308 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:31.907125    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1e6a9b-sg8lq" event={"ID":"558745d0-708d-4645-bb88-adf802fca19d","Type":"ContainerDied","Data":"51cd70e82920e51765c8a5d60d4a7f8acf889962812b59510e62981ff1312699"}
Jul 29 07:22:31.907308 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:31.907208    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51cd70e82920e51765c8a5d60d4a7f8acf889962812b59510e62981ff1312699"
Jul 29 07:22:38.636636 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:38.636605    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc"]
Jul 29 07:22:38.637066 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:38.636816    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc" podUID="559a6d58-0e54-4400-8d31-2e46fe248f96" containerName="maas-api" containerID="cri-o://562ea3d117841cc928cd813b00dee5ff69e321146e0f2c11dc173d5a69caca3d" gracePeriod=30
Jul 29 07:22:38.656763 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:38.656742    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1e6a9b-29755155-cd2rg"]
Jul 29 07:22:38.662571 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:38.662549    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1e6a9b-29755155-cd2rg"]
Jul 29 07:22:38.845980 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:38.845948    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1e6a9b-sg8lq"]
Jul 29 07:22:38.847903 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:38.847876    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1e6a9b-sg8lq"]
Jul 29 07:22:38.887052 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:38.887000    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc"
Jul 29 07:22:38.945359 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:38.945331    2563 generic.go:356] "Generic (PLEG): container finished" podID="559a6d58-0e54-4400-8d31-2e46fe248f96" containerID="562ea3d117841cc928cd813b00dee5ff69e321146e0f2c11dc173d5a69caca3d" exitCode=0
Jul 29 07:22:38.945520 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:38.945379    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc"
Jul 29 07:22:38.945520 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:38.945379    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc" event={"ID":"559a6d58-0e54-4400-8d31-2e46fe248f96","Type":"ContainerDied","Data":"562ea3d117841cc928cd813b00dee5ff69e321146e0f2c11dc173d5a69caca3d"}
Jul 29 07:22:38.945668 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:38.945530    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc" event={"ID":"559a6d58-0e54-4400-8d31-2e46fe248f96","Type":"ContainerDied","Data":"69054019796d547b6cd52b1f3f7805b10b3aa57df38e58fe43732a4ef50dd07e"}
Jul 29 07:22:38.945668 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:38.945570    2563 scope.go:117] "RemoveContainer" containerID="562ea3d117841cc928cd813b00dee5ff69e321146e0f2c11dc173d5a69caca3d"
Jul 29 07:22:38.954191 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:38.954174    2563 scope.go:117] "RemoveContainer" containerID="562ea3d117841cc928cd813b00dee5ff69e321146e0f2c11dc173d5a69caca3d"
Jul 29 07:22:38.954452 ip-10-0-129-223 kubenswrapper[2563]: E0729 07:22:38.954432    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"562ea3d117841cc928cd813b00dee5ff69e321146e0f2c11dc173d5a69caca3d\": container with ID starting with 562ea3d117841cc928cd813b00dee5ff69e321146e0f2c11dc173d5a69caca3d not found: ID does not exist" containerID="562ea3d117841cc928cd813b00dee5ff69e321146e0f2c11dc173d5a69caca3d"
Jul 29 07:22:38.954514 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:38.954465    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"562ea3d117841cc928cd813b00dee5ff69e321146e0f2c11dc173d5a69caca3d"} err="failed to get container status \"562ea3d117841cc928cd813b00dee5ff69e321146e0f2c11dc173d5a69caca3d\": rpc error: code = NotFound desc = could not find container \"562ea3d117841cc928cd813b00dee5ff69e321146e0f2c11dc173d5a69caca3d\": container with ID starting with 562ea3d117841cc928cd813b00dee5ff69e321146e0f2c11dc173d5a69caca3d not found: ID does not exist"
Jul 29 07:22:38.978759 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:38.978731    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/559a6d58-0e54-4400-8d31-2e46fe248f96-openshift-service-ca\") pod \"559a6d58-0e54-4400-8d31-2e46fe248f96\" (UID: \"559a6d58-0e54-4400-8d31-2e46fe248f96\") "
Jul 29 07:22:38.978875 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:38.978844    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5ksc\" (UniqueName: \"kubernetes.io/projected/559a6d58-0e54-4400-8d31-2e46fe248f96-kube-api-access-l5ksc\") pod \"559a6d58-0e54-4400-8d31-2e46fe248f96\" (UID: \"559a6d58-0e54-4400-8d31-2e46fe248f96\") "
Jul 29 07:22:38.978928 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:38.978915    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/559a6d58-0e54-4400-8d31-2e46fe248f96-maas-api-tls\") pod \"559a6d58-0e54-4400-8d31-2e46fe248f96\" (UID: \"559a6d58-0e54-4400-8d31-2e46fe248f96\") "
Jul 29 07:22:38.979125 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:38.979104    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/559a6d58-0e54-4400-8d31-2e46fe248f96-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "559a6d58-0e54-4400-8d31-2e46fe248f96" (UID: "559a6d58-0e54-4400-8d31-2e46fe248f96"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 07:22:38.979206 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:38.979191    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/559a6d58-0e54-4400-8d31-2e46fe248f96-openshift-service-ca\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:22:38.980894 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:38.980874    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/559a6d58-0e54-4400-8d31-2e46fe248f96-kube-api-access-l5ksc" (OuterVolumeSpecName: "kube-api-access-l5ksc") pod "559a6d58-0e54-4400-8d31-2e46fe248f96" (UID: "559a6d58-0e54-4400-8d31-2e46fe248f96"). InnerVolumeSpecName "kube-api-access-l5ksc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 07:22:38.981050 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:38.981024    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/559a6d58-0e54-4400-8d31-2e46fe248f96-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "559a6d58-0e54-4400-8d31-2e46fe248f96" (UID: "559a6d58-0e54-4400-8d31-2e46fe248f96"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 07:22:39.080235 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:39.080202    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5ksc\" (UniqueName: \"kubernetes.io/projected/559a6d58-0e54-4400-8d31-2e46fe248f96-kube-api-access-l5ksc\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:22:39.080235 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:39.080234    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/559a6d58-0e54-4400-8d31-2e46fe248f96-maas-api-tls\") on node \"ip-10-0-129-223.ec2.internal\" DevicePath \"\""
Jul 29 07:22:39.266533 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:39.266503    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc"]
Jul 29 07:22:39.272405 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:39.272383    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1e6a9b-5f55796cbb-ndtpc"]
Jul 29 07:22:39.397475 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:39.397444    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="558745d0-708d-4645-bb88-adf802fca19d" path="/var/lib/kubelet/pods/558745d0-708d-4645-bb88-adf802fca19d/volumes"
Jul 29 07:22:39.397837 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:39.397825    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="559a6d58-0e54-4400-8d31-2e46fe248f96" path="/var/lib/kubelet/pods/559a6d58-0e54-4400-8d31-2e46fe248f96/volumes"
Jul 29 07:22:39.398123 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:39.398111    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abbe197d-fd92-48c2-9694-1f2388864613" path="/var/lib/kubelet/pods/abbe197d-fd92-48c2-9694-1f2388864613/volumes"
Jul 29 07:22:46.491964 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:46.491932    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-586ff9c89-4tgtz_31c2224d-b518-44e9-97cf-2fc2cff44b23/authorino/0.log"
Jul 29 07:22:52.629990 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:52.629957    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-677c488c9b-8j2w7_014fe138-bc84-49bb-903e-dde1a4ddd7a6/manager/0.log"
Jul 29 07:22:53.236352 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:53.236325    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d6c7945d7-t79x9_b7ac749e-0dfb-498c-9294-01f55a0f2bb2/copy-manifests/0.log"
Jul 29 07:22:53.250004 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:53.249980    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d6c7945d7-t79x9_b7ac749e-0dfb-498c-9294-01f55a0f2bb2/manager/0.log"
Jul 29 07:22:53.620084 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:53.620050    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6bd9f8c959-scl8p_e970836d-8708-475d-82a2-55b2acbf2a6f/maas-api/0.log"
Jul 29 07:22:53.753803 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:53.753777    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29755125-8jwlj_05c89cce-87dc-4b43-b025-99153116c311/cleanup/0.log"
Jul 29 07:22:53.864609 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:53.864584    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29755140-7687w_ca26ab7d-4c33-4a9b-a510-d9f477cf199c/cleanup/0.log"
Jul 29 07:22:54.088404 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:54.088375    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-2dm84_bd49080d-e4f0-4620-8e4c-0632e2d426e2/postgres/0.log"
Jul 29 07:22:55.328324 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:55.328293    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-586ff9c89-4tgtz_31c2224d-b518-44e9-97cf-2fc2cff44b23/authorino/0.log"
Jul 29 07:22:56.727289 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:56.727263    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c857974d4-g2dnm_d60bb996-2dc5-4113-b14c-131e20292ed9/kube-auth-proxy/0.log"
Jul 29 07:22:57.078326 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:57.078298    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-v55gs_c2f9abbe-999d-4a04-b435-4cad541c4ff3/payload-pre-processing/0.log"
Jul 29 07:22:57.192875 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:57.192835    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-csc8g_75e1377f-ca60-47c3-b91b-f328413bd9f6/payload-processing/0.log"
Jul 29 07:22:57.653123 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:57.653085    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-zppk6_6101e8c2-fdd5-4de5-8fed-9d250dd10c9a/main/0.log"
Jul 29 07:22:57.990945 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:22:57.990845    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7hgmg_5805ac5a-18bf-4ac2-a496-54323bd42721/main/0.log"
Jul 29 07:23:06.866203 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:06.866178    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dzdr7_0667dd8b-11f7-4b28-8f63-2f1bbbf98c56/global-pull-secret-syncer/0.log"
Jul 29 07:23:07.200541 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:07.200454    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wfqxw_e191231b-69c1-4a7e-baf3-d46d529fe7d6/konnectivity-agent/0.log"
Jul 29 07:23:07.247729 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:07.247704    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-223.ec2.internal_2ff94682e942fadc8425e32a6af57aed/haproxy/0.log"
Jul 29 07:23:12.790098 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:12.790068    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-586ff9c89-4tgtz_31c2224d-b518-44e9-97cf-2fc2cff44b23/authorino/0.log"
Jul 29 07:23:15.126620 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:15.126541    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_36636153-364d-4f22-a2d9-37834e0152c9/alertmanager/0.log"
Jul 29 07:23:15.190601 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:15.190575    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_36636153-364d-4f22-a2d9-37834e0152c9/config-reloader/0.log"
Jul 29 07:23:15.240240 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:15.240214    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_36636153-364d-4f22-a2d9-37834e0152c9/kube-rbac-proxy-web/0.log"
Jul 29 07:23:15.291069 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:15.291047    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_36636153-364d-4f22-a2d9-37834e0152c9/kube-rbac-proxy/0.log"
Jul 29 07:23:15.350172 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:15.350146    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_36636153-364d-4f22-a2d9-37834e0152c9/kube-rbac-proxy-metric/0.log"
Jul 29 07:23:15.411997 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:15.411917    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_36636153-364d-4f22-a2d9-37834e0152c9/prom-label-proxy/0.log"
Jul 29 07:23:15.457722 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:15.457698    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_36636153-364d-4f22-a2d9-37834e0152c9/init-config-reloader/0.log"
Jul 29 07:23:15.518347 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:15.518317    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-87wjh_1786842b-1ae8-4c6b-9c4c-0d21298a4e15/cluster-monitoring-operator/0.log"
Jul 29 07:23:15.575146 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:15.575117    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-x7d4r_8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8/kube-state-metrics/0.log"
Jul 29 07:23:15.615593 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:15.615556    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-x7d4r_8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8/kube-rbac-proxy-main/0.log"
Jul 29 07:23:15.647580 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:15.647560    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-x7d4r_8227fcbe-b2cf-4ea7-bc3d-1d51bfe0fcf8/kube-rbac-proxy-self/0.log"
Jul 29 07:23:15.947550 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:15.947522    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8nmt6_1d2702bd-5995-4c4d-b90a-dc7e40df640d/node-exporter/0.log"
Jul 29 07:23:15.984597 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:15.984573    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8nmt6_1d2702bd-5995-4c4d-b90a-dc7e40df640d/kube-rbac-proxy/0.log"
Jul 29 07:23:16.026082 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.026061    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8nmt6_1d2702bd-5995-4c4d-b90a-dc7e40df640d/init-textfile/0.log"
Jul 29 07:23:16.195426 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.195398    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rtf28/perf-node-gather-daemonset-jvtwd"]
Jul 29 07:23:16.195991 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.195973    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="559a6d58-0e54-4400-8d31-2e46fe248f96" containerName="maas-api"
Jul 29 07:23:16.196082 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.195993    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="559a6d58-0e54-4400-8d31-2e46fe248f96" containerName="maas-api"
Jul 29 07:23:16.196082 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.196028    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="558745d0-708d-4645-bb88-adf802fca19d" containerName="revoke-keys"
Jul 29 07:23:16.196082 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.196037    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="558745d0-708d-4645-bb88-adf802fca19d" containerName="revoke-keys"
Jul 29 07:23:16.196238 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.196164    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="558745d0-708d-4645-bb88-adf802fca19d" containerName="revoke-keys"
Jul 29 07:23:16.196238 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.196184    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="559a6d58-0e54-4400-8d31-2e46fe248f96" containerName="maas-api"
Jul 29 07:23:16.199727 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.199670    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-jvtwd"
Jul 29 07:23:16.203139 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.203114    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rtf28\"/\"default-dockercfg-zbqqp\""
Jul 29 07:23:16.203646 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.203621    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rtf28\"/\"openshift-service-ca.crt\""
Jul 29 07:23:16.204328 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.204311    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rtf28\"/\"kube-root-ca.crt\""
Jul 29 07:23:16.219907 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.219884    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rtf28/perf-node-gather-daemonset-jvtwd"]
Jul 29 07:23:16.222293 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.222268    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-dcwzq_46f8ec36-334c-400b-aa93-b57679124eaf/kube-rbac-proxy-main/0.log"
Jul 29 07:23:16.294553 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.294525    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-dcwzq_46f8ec36-334c-400b-aa93-b57679124eaf/kube-rbac-proxy-self/0.log"
Jul 29 07:23:16.304067 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.304044    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3b50e95a-6277-4e11-9ab3-f4d212bec755-podres\") pod \"perf-node-gather-daemonset-jvtwd\" (UID: \"3b50e95a-6277-4e11-9ab3-f4d212bec755\") " pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-jvtwd"
Jul 29 07:23:16.304199 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.304094    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b50e95a-6277-4e11-9ab3-f4d212bec755-sys\") pod \"perf-node-gather-daemonset-jvtwd\" (UID: \"3b50e95a-6277-4e11-9ab3-f4d212bec755\") " pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-jvtwd"
Jul 29 07:23:16.304199 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.304109    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3b50e95a-6277-4e11-9ab3-f4d212bec755-proc\") pod \"perf-node-gather-daemonset-jvtwd\" (UID: \"3b50e95a-6277-4e11-9ab3-f4d212bec755\") " pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-jvtwd"
Jul 29 07:23:16.304199 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.304126    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m7td\" (UniqueName: \"kubernetes.io/projected/3b50e95a-6277-4e11-9ab3-f4d212bec755-kube-api-access-4m7td\") pod \"perf-node-gather-daemonset-jvtwd\" (UID: \"3b50e95a-6277-4e11-9ab3-f4d212bec755\") " pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-jvtwd"
Jul 29 07:23:16.304355 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.304224    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3b50e95a-6277-4e11-9ab3-f4d212bec755-lib-modules\") pod \"perf-node-gather-daemonset-jvtwd\" (UID: \"3b50e95a-6277-4e11-9ab3-f4d212bec755\") " pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-jvtwd"
Jul 29 07:23:16.332128 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.332107    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-dcwzq_46f8ec36-334c-400b-aa93-b57679124eaf/openshift-state-metrics/0.log"
Jul 29 07:23:16.384463 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.384437    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_526902c0-ec0e-4384-b718-24d9387d9fee/prometheus/0.log"
Jul 29 07:23:16.404730 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.404711    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3b50e95a-6277-4e11-9ab3-f4d212bec755-lib-modules\") pod \"perf-node-gather-daemonset-jvtwd\" (UID: \"3b50e95a-6277-4e11-9ab3-f4d212bec755\") " pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-jvtwd"
Jul 29 07:23:16.404826 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.404745    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3b50e95a-6277-4e11-9ab3-f4d212bec755-podres\") pod \"perf-node-gather-daemonset-jvtwd\" (UID: \"3b50e95a-6277-4e11-9ab3-f4d212bec755\") " pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-jvtwd"
Jul 29 07:23:16.404826 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.404788    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b50e95a-6277-4e11-9ab3-f4d212bec755-sys\") pod \"perf-node-gather-daemonset-jvtwd\" (UID: \"3b50e95a-6277-4e11-9ab3-f4d212bec755\") " pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-jvtwd"
Jul 29 07:23:16.404826 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.404801    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3b50e95a-6277-4e11-9ab3-f4d212bec755-proc\") pod \"perf-node-gather-daemonset-jvtwd\" (UID: \"3b50e95a-6277-4e11-9ab3-f4d212bec755\") " pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-jvtwd"
Jul 29 07:23:16.404826 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.404821    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m7td\" (UniqueName: \"kubernetes.io/projected/3b50e95a-6277-4e11-9ab3-f4d212bec755-kube-api-access-4m7td\") pod \"perf-node-gather-daemonset-jvtwd\" (UID: \"3b50e95a-6277-4e11-9ab3-f4d212bec755\") " pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-jvtwd"
Jul 29 07:23:16.405042 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.404907    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b50e95a-6277-4e11-9ab3-f4d212bec755-sys\") pod \"perf-node-gather-daemonset-jvtwd\" (UID: \"3b50e95a-6277-4e11-9ab3-f4d212bec755\") " pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-jvtwd"
Jul 29 07:23:16.405042 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.404904    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3b50e95a-6277-4e11-9ab3-f4d212bec755-lib-modules\") pod \"perf-node-gather-daemonset-jvtwd\" (UID: \"3b50e95a-6277-4e11-9ab3-f4d212bec755\") " pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-jvtwd"
Jul 29 07:23:16.405042 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.404931    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3b50e95a-6277-4e11-9ab3-f4d212bec755-proc\") pod \"perf-node-gather-daemonset-jvtwd\" (UID: \"3b50e95a-6277-4e11-9ab3-f4d212bec755\") " pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-jvtwd"
Jul 29 07:23:16.405042 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.404939    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3b50e95a-6277-4e11-9ab3-f4d212bec755-podres\") pod \"perf-node-gather-daemonset-jvtwd\" (UID: \"3b50e95a-6277-4e11-9ab3-f4d212bec755\") " pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-jvtwd"
Jul 29 07:23:16.414620 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.414596    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m7td\" (UniqueName: \"kubernetes.io/projected/3b50e95a-6277-4e11-9ab3-f4d212bec755-kube-api-access-4m7td\") pod \"perf-node-gather-daemonset-jvtwd\" (UID: \"3b50e95a-6277-4e11-9ab3-f4d212bec755\") " pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-jvtwd"
Jul 29 07:23:16.429572 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.429555    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_526902c0-ec0e-4384-b718-24d9387d9fee/config-reloader/0.log"
Jul 29 07:23:16.469463 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.469403    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_526902c0-ec0e-4384-b718-24d9387d9fee/thanos-sidecar/0.log"
Jul 29 07:23:16.501999 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.501976    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_526902c0-ec0e-4384-b718-24d9387d9fee/kube-rbac-proxy-web/0.log"
Jul 29 07:23:16.509151 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.509136    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-jvtwd"
Jul 29 07:23:16.536053 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.535997    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_526902c0-ec0e-4384-b718-24d9387d9fee/kube-rbac-proxy/0.log"
Jul 29 07:23:16.572521 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.572495    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_526902c0-ec0e-4384-b718-24d9387d9fee/kube-rbac-proxy-thanos/0.log"
Jul 29 07:23:16.616075 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.616050    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_526902c0-ec0e-4384-b718-24d9387d9fee/init-config-reloader/0.log"
Jul 29 07:23:16.651471 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.651446    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rtf28/perf-node-gather-daemonset-jvtwd"]
Jul 29 07:23:16.652920 ip-10-0-129-223 kubenswrapper[2563]: W0729 07:23:16.652892    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3b50e95a_6277_4e11_9ab3_f4d212bec755.slice/crio-94630c64a83d067d547f2b4258f8664161b99a1bd7c361c5b0206e7e9b77f7d6 WatchSource:0}: Error finding container 94630c64a83d067d547f2b4258f8664161b99a1bd7c361c5b0206e7e9b77f7d6: Status 404 returned error can't find the container with id 94630c64a83d067d547f2b4258f8664161b99a1bd7c361c5b0206e7e9b77f7d6
Jul 29 07:23:16.655144 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.655087    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-lm28j_9c169934-87cc-43de-b7e7-2d26c36ffd35/prometheus-operator/0.log"
Jul 29 07:23:16.694730 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.694711    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-lm28j_9c169934-87cc-43de-b7e7-2d26c36ffd35/kube-rbac-proxy/0.log"
Jul 29 07:23:16.727428 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:16.727410    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-2xhdp_34133ad8-194a-4352-af5f-667836d27cb2/prometheus-operator-admission-webhook/0.log"
Jul 29 07:23:17.149134 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:17.149103    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-jvtwd" event={"ID":"3b50e95a-6277-4e11-9ab3-f4d212bec755","Type":"ContainerStarted","Data":"5739888d7b4cd83a7df0861313537f1ce5d4cda09a567f8b76b50c7359f16ae1"}
Jul 29 07:23:17.149299 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:17.149155    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-jvtwd" event={"ID":"3b50e95a-6277-4e11-9ab3-f4d212bec755","Type":"ContainerStarted","Data":"94630c64a83d067d547f2b4258f8664161b99a1bd7c361c5b0206e7e9b77f7d6"}
Jul 29 07:23:17.149299 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:17.149187    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-jvtwd"
Jul 29 07:23:17.175339 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:17.175291    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-jvtwd" podStartSLOduration=1.175278303 podStartE2EDuration="1.175278303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 07:23:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 07:23:17.173380649 +0000 UTC m=+3640.251012415" watchObservedRunningTime="2026-07-29 07:23:17.175278303 +0000 UTC m=+3640.252910130"
Jul 29 07:23:18.202607 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:18.202582    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-mrklz_37763295-c6d9-4846-8891-f4c5b78ec095/networking-console-plugin/0.log"
Jul 29 07:23:21.183785 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:21.183760    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bd28d_4f4e0417-82ca-4f78-9043-85dc8af1d71b/dns/0.log"
Jul 29 07:23:21.217130 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:21.217110    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bd28d_4f4e0417-82ca-4f78-9043-85dc8af1d71b/kube-rbac-proxy/0.log"
Jul 29 07:23:21.340602 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:21.340572    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rkphn_4a2a21ed-615a-4783-9a2a-b57804724f79/dns-node-resolver/0.log"
Jul 29 07:23:22.162717 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:22.162691    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qrnmf_2079ff1a-5b9e-439d-bd6c-02c73c344340/node-ca/0.log"
Jul 29 07:23:23.164651 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:23.164624    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rtf28/perf-node-gather-daemonset-jvtwd"
Jul 29 07:23:23.377516 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:23.377487    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c857974d4-g2dnm_d60bb996-2dc5-4113-b14c-131e20292ed9/kube-auth-proxy/0.log"
Jul 29 07:23:23.554183 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:23.554107    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-v55gs_c2f9abbe-999d-4a04-b435-4cad541c4ff3/payload-pre-processing/0.log"
Jul 29 07:23:23.600150 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:23.600126    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-csc8g_75e1377f-ca60-47c3-b91b-f328413bd9f6/payload-processing/0.log"
Jul 29 07:23:24.532327 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:24.532301    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4pk46_2f05bf26-3930-4e41-ad15-78055946eb87/serve-healthcheck-canary/0.log"
Jul 29 07:23:25.252384 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:25.252357    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-mr497_cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4/insights-operator/0.log"
Jul 29 07:23:25.253186 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:25.253170    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-mr497_cf59d0f1-c3e6-4d35-aa15-250b8ca1cfb4/insights-operator/1.log"
Jul 29 07:23:25.519179 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:25.519104    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vt5qk_3fec1db3-1218-4a65-b5c1-1e7473869742/kube-rbac-proxy/0.log"
Jul 29 07:23:25.545398 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:25.545370    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vt5qk_3fec1db3-1218-4a65-b5c1-1e7473869742/exporter/0.log"
Jul 29 07:23:25.571033 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:25.571011    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vt5qk_3fec1db3-1218-4a65-b5c1-1e7473869742/extractor/0.log"
Jul 29 07:23:27.204327 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:27.204293    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c857974d4-g2dnm_d60bb996-2dc5-4113-b14c-131e20292ed9/kube-auth-proxy/0.log"
Jul 29 07:23:27.335245 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:27.335220    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-v55gs_c2f9abbe-999d-4a04-b435-4cad541c4ff3/payload-pre-processing/0.log"
Jul 29 07:23:27.360979 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:27.360951    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-csc8g_75e1377f-ca60-47c3-b91b-f328413bd9f6/payload-processing/0.log"
Jul 29 07:23:28.425100 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:28.425068    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-677c488c9b-8j2w7_014fe138-bc84-49bb-903e-dde1a4ddd7a6/manager/0.log"
Jul 29 07:23:28.829730 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:28.829701    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d6c7945d7-t79x9_b7ac749e-0dfb-498c-9294-01f55a0f2bb2/manager/0.log"
Jul 29 07:23:28.861914 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:28.861892    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d6c7945d7-t79x9_b7ac749e-0dfb-498c-9294-01f55a0f2bb2/copy-manifests/0.log"
Jul 29 07:23:28.948836 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:28.948808    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c857974d4-g2dnm_d60bb996-2dc5-4113-b14c-131e20292ed9/kube-auth-proxy/0.log"
Jul 29 07:23:29.096803 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:29.096727    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-v55gs_c2f9abbe-999d-4a04-b435-4cad541c4ff3/payload-pre-processing/0.log"
Jul 29 07:23:29.138981 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:29.138954    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-csc8g_75e1377f-ca60-47c3-b91b-f328413bd9f6/payload-processing/0.log"
Jul 29 07:23:30.208235 ip-10-0-129-223 kubenswrapper[2563]: I0729 07:23:30.208210    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-68c9b6dddc-gk4hv_9862f729-e68d-4647-bdf1-9fac3162c4de/manager/0.log"