Jul 09 18:04:14.183643 ip-10-0-129-181 systemd[1]: Starting Kubernetes Kubelet...
Jul 09 18:04:14.672408 ip-10-0-129-181 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:04:14.672408 ip-10-0-129-181 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 09 18:04:14.672408 ip-10-0-129-181 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:04:14.673041 ip-10-0-129-181 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 09 18:04:14.673041 ip-10-0-129-181 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:04:14.675301 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.675127    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 09 18:04:14.680965 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.680940    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:14.680965 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.680961    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:14.680965 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.680966    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:14.680965 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.680969    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:14.680965 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.680972    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:14.681171 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.680975    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:14.681171 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.680980    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:14.681171 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.680985    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:14.681171 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.680989    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:14.681171 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.680992    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:14.681171 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.680995    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:14.681171 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.680998    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:14.681171 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681001    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:14.681171 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681004    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:14.681171 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681006    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:14.681171 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681009    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:14.681171 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681012    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:14.681171 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681016    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:14.681171 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681018    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:14.681171 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681021    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:14.681171 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681024    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:14.681171 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681027    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:14.681171 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681029    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:14.681171 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681032    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:14.681685 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681035    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:14.681685 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681038    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:14.681685 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681041    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:14.681685 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681044    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:14.681685 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681047    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:14.681685 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681050    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:14.681685 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681053    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:14.681685 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681059    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:14.681685 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681062    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:14.681685 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681065    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:14.681685 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681068    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:14.681685 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681073    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:14.681685 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681077    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:14.681685 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681080    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:14.681685 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681084    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:14.681685 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681087    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:14.681685 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681090    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:14.681685 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681093    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:14.681685 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681096    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:14.682163 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681099    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:14.682163 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681102    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:14.682163 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681105    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:14.682163 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681108    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:14.682163 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681110    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:14.682163 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681113    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:14.682163 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681116    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:14.682163 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681119    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:14.682163 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681123    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:14.682163 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681127    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:14.682163 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681130    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:14.682163 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681133    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:14.682163 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681136    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:14.682163 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681139    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:14.682163 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681142    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:14.682163 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681144    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:14.682163 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681149    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:14.682163 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681151    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:14.682163 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681154    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:14.682163 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681158    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:14.682692 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681161    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:14.682692 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681164    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:14.682692 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681167    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:14.682692 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681170    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:14.682692 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681173    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:14.682692 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681176    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:14.682692 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681179    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:14.682692 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681183    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:14.682692 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681186    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:14.682692 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681188    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:14.682692 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681191    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:14.682692 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681194    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:14.682692 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681196    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:14.682692 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681199    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:14.682692 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681202    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:14.682692 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681204    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:14.682692 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681207    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:14.682692 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681210    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:14.682692 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681213    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:14.682692 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681215    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:14.683190 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681219    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:14.683190 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681222    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:14.683190 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681225    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:14.683190 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681227    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:14.683190 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681230    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:14.683190 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681234    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:14.683190 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681237    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:14.683190 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681239    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:14.683190 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681242    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:14.683190 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681245    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:14.683190 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681248    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:14.683190 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681251    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:14.683190 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681253    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:14.683190 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681256    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:14.683190 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681259    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:14.683190 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.681262    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:14.683190 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682069    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:14.683190 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682077    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:14.683190 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682082    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:14.683190 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682085    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:14.683711 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682088    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:14.683711 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682091    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:14.683711 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682094    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:14.683711 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682097    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:14.683711 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682100    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:14.683711 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682103    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:14.683711 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682106    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:14.683711 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682108    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:14.683711 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682112    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:14.683711 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682114    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:14.683711 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682117    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:14.683711 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682120    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:14.683711 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682123    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:14.683711 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682126    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:14.683711 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682129    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:14.683711 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682132    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:14.683711 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682134    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:14.683711 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682137    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:14.683711 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682140    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:14.683711 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682142    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:14.684259 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682145    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:14.684259 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682149    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:14.684259 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682152    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:14.684259 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682155    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:14.684259 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682157    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:14.684259 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682160    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:14.684259 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682163    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:14.684259 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682166    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:14.684259 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682169    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:14.684259 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682172    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:14.684259 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682175    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:14.684259 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682178    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:14.684259 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682181    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:14.684259 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682184    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:14.684259 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682187    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:14.684259 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682190    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:14.684259 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682193    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:14.684259 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682196    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:14.684259 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682198    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:14.684259 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682201    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:14.684797 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682205    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:14.684797 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682209    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:14.684797 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682214    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:14.684797 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682217    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:14.684797 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682220    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:14.684797 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682224    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:14.684797 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682227    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:14.684797 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682231    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:14.684797 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682234    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:14.684797 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682237    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:14.684797 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682240    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:14.684797 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682243    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:14.684797 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682246    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:14.684797 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682250    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:14.684797 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682253    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:14.684797 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682256    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:14.684797 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682259    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:14.684797 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682262    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:14.685261 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682264    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:14.685261 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682267    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:14.685261 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682270    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:14.685261 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682273    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:14.685261 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682276    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:14.685261 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682278    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:14.685261 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682281    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:14.685261 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682284    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:14.685261 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682311    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:14.685261 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682315    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:14.685261 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682318    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:14.685261 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682321    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:14.685261 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682324    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:14.685261 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682327    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:14.685261 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682331    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:14.685261 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682335    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:14.685261 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682338    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:14.685261 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682341    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:14.685261 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682344    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:14.685261 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682347    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:14.685781 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682350    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:14.685781 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682354    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:14.685781 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682358    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:14.685781 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682361    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:14.685781 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682364    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:14.685781 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682367    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:14.685781 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682370    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:14.685781 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682373    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:14.685781 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682376    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:14.685781 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682379    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:14.685781 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682382    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:14.685781 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682384    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:14.685781 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682387    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:14.685781 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682390    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:14.685781 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682393    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:14.685781 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682396    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:14.685781 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682400    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:14.685781 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682889    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:14.685781 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682896    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:14.685781 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682901    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:14.686306 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682906    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:14.686306 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682909    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:14.686306 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682912    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:14.686306 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682916    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:14.686306 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682919    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:14.686306 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682922    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:14.686306 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682926    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:14.686306 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682929    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:14.686306 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682932    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:14.686306 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682934    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:14.686306 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682937    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:14.686306 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682940    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:14.686306 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682944    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:14.686306 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682947    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:14.686306 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682950    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:14.686306 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682953    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:14.686306 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682956    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:14.686306 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682960    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:14.686306 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682963    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:14.686306 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682966    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:14.686807 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682969    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:14.686807 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682971    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:14.686807 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682976    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:14.686807 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682980    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:14.686807 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682983    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:14.686807 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682986    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:14.686807 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682990    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:14.686807 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682993    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:14.686807 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682995    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:14.686807 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.682998    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:14.686807 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683001    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:14.686807 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683004    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:14.686807 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683007    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:14.686807 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683010    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:14.686807 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683013    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:14.686807 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683016    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:14.686807 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683018    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:14.686807 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683021    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:14.686807 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683024    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:14.687315 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683026    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:14.687315 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683029    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:14.687315 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683032    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:14.687315 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683035    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:14.687315 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683038    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:14.687315 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683040    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:14.687315 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683043    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:14.687315 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683046    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:14.687315 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683049    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:14.687315 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683051    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:14.687315 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683054    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:14.687315 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683057    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:14.687315 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683060    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:14.687315 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683062    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:14.687315 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683065    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:14.687315 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683067    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:14.687315 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683070    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:14.687315 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683073    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:14.687315 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683077    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:14.687315 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683080    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:14.687819 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683083    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:14.687819 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683087    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:14.687819 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683090    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:14.687819 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683093    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:14.687819 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683095    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:14.687819 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683098    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:14.687819 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683101    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:14.687819 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683104    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:14.687819 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683107    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:14.687819 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683109    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:14.687819 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683112    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:14.687819 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683115    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:14.687819 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683117    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:14.687819 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683120    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:14.687819 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683123    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:14.687819 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683127    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:14.687819 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683131    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:14.687819 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683134    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:14.687819 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683137    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:14.687819 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683139    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:14.688337 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683143    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:14.688337 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683145    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:14.688337 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683148    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:14.688337 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683151    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:14.688337 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683154    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:14.688337 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683156    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:14.688337 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683159    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:14.688337 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683162    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:14.688337 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683164    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:14.688337 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683167    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:14.688337 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683170    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:14.688337 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683173    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:14.688337 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683175    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:14.688337 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683178    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:14.688337 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683180    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:14.688337 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683183    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:14.688337 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683186    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:14.688337 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683262    2563 flags.go:64] FLAG: --address="0.0.0.0"
Jul 09 18:04:14.688337 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683269    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 09 18:04:14.688337 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683277    2563 flags.go:64] FLAG: --anonymous-auth="true"
Jul 09 18:04:14.688337 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683282    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 09 18:04:14.688872 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683304    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 09 18:04:14.688872 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683310    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 09 18:04:14.688872 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683319    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 09 18:04:14.688872 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683328    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 09 18:04:14.688872 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683331    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 09 18:04:14.688872 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683334    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 09 18:04:14.688872 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683338    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 09 18:04:14.688872 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683341    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 09 18:04:14.688872 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683345    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 09 18:04:14.688872 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683348    2563 flags.go:64] FLAG: --cgroup-root=""
Jul 09 18:04:14.688872 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683351    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 09 18:04:14.688872 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683354    2563 flags.go:64] FLAG: --client-ca-file=""
Jul 09 18:04:14.688872 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683358    2563 flags.go:64] FLAG: --cloud-provider="external"
Jul 09 18:04:14.688872 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683361    2563 flags.go:64] FLAG: --cluster-dns="[]"
Jul 09 18:04:14.688872 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683368    2563 flags.go:64] FLAG: --cluster-domain=""
Jul 09 18:04:14.688872 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683371    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 09 18:04:14.688872 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683375    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 09 18:04:14.688872 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683378    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 09 18:04:14.688872 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683382    2563 flags.go:64] FLAG: --container-log-max-files="5"
Jul 09 18:04:14.688872 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683391    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 09 18:04:14.688872 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683395    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 09 18:04:14.688872 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683398    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 09 18:04:14.688872 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683402    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 09 18:04:14.689469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683406    2563 flags.go:64] FLAG: --contention-profiling="false"
Jul 09 18:04:14.689469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683409    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 09 18:04:14.689469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683412    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 09 18:04:14.689469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683416    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 09 18:04:14.689469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683419    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 09 18:04:14.689469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683424    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 09 18:04:14.689469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683427    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 09 18:04:14.689469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683430    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 09 18:04:14.689469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683433    2563 flags.go:64] FLAG: --enable-load-reader="false"
Jul 09 18:04:14.689469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683437    2563 flags.go:64] FLAG: --enable-server="true"
Jul 09 18:04:14.689469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683440    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 09 18:04:14.689469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683445    2563 flags.go:64] FLAG: --event-burst="100"
Jul 09 18:04:14.689469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683449    2563 flags.go:64] FLAG: --event-qps="50"
Jul 09 18:04:14.689469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683452    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 09 18:04:14.689469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683455    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 09 18:04:14.689469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683458    2563 flags.go:64] FLAG: --eviction-hard=""
Jul 09 18:04:14.689469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683462    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 09 18:04:14.689469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683465    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 09 18:04:14.689469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683468    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 09 18:04:14.689469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683472    2563 flags.go:64] FLAG: --eviction-soft=""
Jul 09 18:04:14.689469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683475    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 09 18:04:14.689469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683478    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 09 18:04:14.689469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683482    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 09 18:04:14.689469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683488    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 09 18:04:14.689469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683493    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 09 18:04:14.690158 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683497    2563 flags.go:64] FLAG: --fail-swap-on="true"
Jul 09 18:04:14.690158 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683502    2563 flags.go:64] FLAG: --feature-gates=""
Jul 09 18:04:14.690158 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683507    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 09 18:04:14.690158 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683511    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 09 18:04:14.690158 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683517    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 09 18:04:14.690158 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683521    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 09 18:04:14.690158 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683524    2563 flags.go:64] FLAG: --healthz-port="10248"
Jul 09 18:04:14.690158 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683528    2563 flags.go:64] FLAG: --help="false"
Jul 09 18:04:14.690158 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683531    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-129-181.ec2.internal"
Jul 09 18:04:14.690158 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683535    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 09 18:04:14.690158 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683538    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 09 18:04:14.690158 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683541    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 09 18:04:14.690158 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683544    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 09 18:04:14.690158 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683549    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 09 18:04:14.690158 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683552    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 09 18:04:14.690158 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683555    2563 flags.go:64] FLAG: --image-service-endpoint=""
Jul 09 18:04:14.690158 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683558    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 09 18:04:14.690158 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683561    2563 flags.go:64] FLAG: --kube-api-burst="100"
Jul 09 18:04:14.690158 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683564    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 09 18:04:14.690158 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683567    2563 flags.go:64] FLAG: --kube-api-qps="50"
Jul 09 18:04:14.690158 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683570    2563 flags.go:64] FLAG: --kube-reserved=""
Jul 09 18:04:14.690158 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683573    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 09 18:04:14.690158 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683577    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 09 18:04:14.690158 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683580    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 09 18:04:14.690801 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683583    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 09 18:04:14.690801 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683586    2563 flags.go:64] FLAG: --lock-file=""
Jul 09 18:04:14.690801 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683589    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 09 18:04:14.690801 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683592    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 09 18:04:14.690801 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683595    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 09 18:04:14.690801 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683601    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 09 18:04:14.690801 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683607    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 09 18:04:14.690801 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683610    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 09 18:04:14.690801 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683613    2563 flags.go:64] FLAG: --logging-format="text"
Jul 09 18:04:14.690801 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683616    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 09 18:04:14.690801 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683620    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 09 18:04:14.690801 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683623    2563 flags.go:64] FLAG: --manifest-url=""
Jul 09 18:04:14.690801 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683628    2563 flags.go:64] FLAG: --manifest-url-header=""
Jul 09 18:04:14.690801 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683633    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 09 18:04:14.690801 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683636    2563 flags.go:64] FLAG: --max-open-files="1000000"
Jul 09 18:04:14.690801 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683641    2563 flags.go:64] FLAG: --max-pods="110"
Jul 09 18:04:14.690801 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683644    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 09 18:04:14.690801 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683647    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 09 18:04:14.690801 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683651    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 09 18:04:14.690801 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683654    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 09 18:04:14.690801 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683657    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 09 18:04:14.690801 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683660    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 09 18:04:14.690801 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683663    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 09 18:04:14.690801 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683671    2563 flags.go:64] FLAG: --node-status-max-images="50"
Jul 09 18:04:14.690801 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683674    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 09 18:04:14.691474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683677    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 09 18:04:14.691474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683680    2563 flags.go:64] FLAG: --pod-cidr=""
Jul 09 18:04:14.691474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683683    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 09 18:04:14.691474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683689    2563 flags.go:64] FLAG: --pod-manifest-path=""
Jul 09 18:04:14.691474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683692    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 09 18:04:14.691474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683695    2563 flags.go:64] FLAG: --pods-per-core="0"
Jul 09 18:04:14.691474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683698    2563 flags.go:64] FLAG: --port="10250"
Jul 09 18:04:14.691474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683701    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 09 18:04:14.691474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683704    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07696281d377d75ae"
Jul 09 18:04:14.691474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683708    2563 flags.go:64] FLAG: --qos-reserved=""
Jul 09 18:04:14.691474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683711    2563 flags.go:64] FLAG: --read-only-port="10255"
Jul 09 18:04:14.691474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683715    2563 flags.go:64] FLAG: --register-node="true"
Jul 09 18:04:14.691474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683718    2563 flags.go:64] FLAG: --register-with-taints=""
Jul 09 18:04:14.691474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683723    2563 flags.go:64] FLAG: --registry-burst="10"
Jul 09 18:04:14.691474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683727    2563 flags.go:64] FLAG: --registry-qps="5"
Jul 09 18:04:14.691474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683730    2563 flags.go:64] FLAG: --reserved-cpus=""
Jul 09 18:04:14.691474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683733    2563 flags.go:64] FLAG: --reserved-memory=""
Jul 09 18:04:14.691474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683737    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 09 18:04:14.691474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683741    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 09 18:04:14.691474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683744    2563 flags.go:64] FLAG: --rotate-certificates="false"
Jul 09 18:04:14.691474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683749    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 09 18:04:14.691474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683752    2563 flags.go:64] FLAG: --runonce="false"
Jul 09 18:04:14.691474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683755    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 09 18:04:14.691474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683758    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 09 18:04:14.691474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683762    2563 flags.go:64] FLAG: --seccomp-default="false"
Jul 09 18:04:14.692112 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683764    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 09 18:04:14.692112 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683768    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 09 18:04:14.692112 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683771    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 09 18:04:14.692112 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683774    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 09 18:04:14.692112 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683777    2563 flags.go:64] FLAG: --storage-driver-password="root"
Jul 09 18:04:14.692112 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683780    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 09 18:04:14.692112 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683783    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 09 18:04:14.692112 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683786    2563 flags.go:64] FLAG: --storage-driver-user="root"
Jul 09 18:04:14.692112 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683789    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 09 18:04:14.692112 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683793    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 09 18:04:14.692112 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683796    2563 flags.go:64] FLAG: --system-cgroups=""
Jul 09 18:04:14.692112 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683799    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 09 18:04:14.692112 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683805    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 09 18:04:14.692112 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683808    2563 flags.go:64] FLAG: --tls-cert-file=""
Jul 09 18:04:14.692112 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683811    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 09 18:04:14.692112 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683815    2563 flags.go:64] FLAG: --tls-min-version=""
Jul 09 18:04:14.692112 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683818    2563 flags.go:64] FLAG: --tls-private-key-file=""
Jul 09 18:04:14.692112 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683821    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 09 18:04:14.692112 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683824    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 09 18:04:14.692112 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683827    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 09 18:04:14.692112 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683832    2563 flags.go:64] FLAG: --v="2"
Jul 09 18:04:14.692112 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683837    2563 flags.go:64] FLAG: --version="false"
Jul 09 18:04:14.692112 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683841    2563 flags.go:64] FLAG: --vmodule=""
Jul 09 18:04:14.692112 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683846    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 09 18:04:14.692112 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.683850    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 09 18:04:14.692813 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683975    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:14.692813 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683979    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:14.692813 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683984    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:14.692813 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683988    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:14.692813 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683991    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:14.692813 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683994    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:14.692813 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.683997    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:14.692813 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684000    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:14.692813 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684004    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:14.692813 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684006    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:14.692813 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684009    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:14.692813 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684012    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:14.692813 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684015    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:14.692813 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684018    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:14.692813 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684020    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:14.692813 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684023    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:14.692813 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684026    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:14.692813 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684029    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:14.692813 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684032    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:14.692813 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684034    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:14.693327 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684037    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:14.693327 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684040    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:14.693327 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684042    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:14.693327 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684045    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:14.693327 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684048    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:14.693327 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684050    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:14.693327 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684053    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:14.693327 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684057    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:14.693327 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684060    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:14.693327 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684063    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:14.693327 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684066    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:14.693327 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684069    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:14.693327 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684072    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:14.693327 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684075    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:14.693327 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684081    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:14.693327 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684084    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:14.693327 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684087    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:14.693327 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684091    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:14.693327 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684094    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:14.693327 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684097    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:14.693970 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684100    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:14.693970 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684103    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:14.693970 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684106    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:14.693970 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684109    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:14.693970 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684112    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:14.693970 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684115    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:14.693970 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684117    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:14.693970 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684120    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:14.693970 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684123    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:14.693970 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684126    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:14.693970 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684129    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:14.693970 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684131    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:14.693970 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684134    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:14.693970 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684137    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:14.693970 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684140    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:14.693970 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684143    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:14.693970 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684145    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:14.693970 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684148    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:14.693970 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684151    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:14.694479 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684155    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:14.694479 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684158    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:14.694479 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684160    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:14.694479 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684163    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:14.694479 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684166    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:14.694479 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684169    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:14.694479 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684172    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:14.694479 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684176    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:14.694479 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684179    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:14.694479 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684182    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:14.694479 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684184    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:14.694479 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684187    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:14.694479 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684190    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:14.694479 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684193    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:14.694479 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684196    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:14.694479 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684199    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:14.694479 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684202    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:14.694479 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684206    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:14.694479 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684210    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:14.694479 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684213    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:14.694999 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684216    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:14.694999 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684219    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:14.694999 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684221    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:14.694999 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684224    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:14.694999 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684227    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:14.694999 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684230    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:14.694999 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684232    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:14.694999 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684235    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:14.694999 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684239    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:14.694999 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684243    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:14.694999 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684246    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:14.694999 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684249    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:14.694999 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684254    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:14.694999 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684257    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:14.694999 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684261    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:14.694999 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684264    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:14.694999 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684267    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:14.694999 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684270    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:14.694999 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684273    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:14.695526 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.684278    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:14.695526 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.684284    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:04:14.695526 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.691518    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 09 18:04:14.695526 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.691534    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 09 18:04:14.695526 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.691555    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 09 18:04:14.695526 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.691564    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 09 18:04:14.695526 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691625    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:14.695526 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691630    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:14.695526 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691634    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:14.695526 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691637    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:14.695526 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691640    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:14.695526 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691643    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:14.695526 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691646    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:14.695526 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691649    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:14.695893 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691653    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:14.695893 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691656    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:14.695893 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691659    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:14.695893 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691662    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:14.695893 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691665    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:14.695893 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691668    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:14.695893 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691670    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:14.695893 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691673    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:14.695893 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691678    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:14.695893 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691682    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:14.695893 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691686    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:14.695893 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691690    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:14.695893 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691693    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:14.695893 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691696    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:14.695893 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691699    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:14.695893 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691702    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:14.695893 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691706    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:14.695893 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691709    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:14.695893 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691712    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:14.696385 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691714    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:14.696385 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691717    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:14.696385 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691721    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:14.696385 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691724    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:14.696385 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691727    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:14.696385 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691730    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:14.696385 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691733    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:14.696385 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691736    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:14.696385 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691739    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:14.696385 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691742    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:14.696385 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691744    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:14.696385 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691747    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:14.696385 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691750    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:14.696385 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691753    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:14.696385 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691756    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:14.696385 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691759    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:14.696385 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691762    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:14.696385 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691764    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:14.696385 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691767    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:14.696385 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691770    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:14.696900 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691772    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:14.696900 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691775    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:14.696900 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691778    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:14.696900 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691781    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:14.696900 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691784    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:14.696900 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691787    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:14.696900 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691790    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:14.696900 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691795    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:14.696900 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691799    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:14.696900 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691802    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:14.696900 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691806    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:14.696900 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691808    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:14.696900 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691811    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:14.696900 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691814    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:14.696900 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691817    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:14.696900 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691820    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:14.696900 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691823    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:14.696900 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691826    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:14.696900 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691828    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:14.696900 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691831    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:14.697418 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691834    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:14.697418 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691837    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:14.697418 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691840    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:14.697418 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691860    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:14.697418 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691864    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:14.697418 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691867    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:14.697418 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691871    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:14.697418 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691874    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:14.697418 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691877    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:14.697418 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691880    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:14.697418 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691883    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:14.697418 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691886    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:14.697418 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691889    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:14.697418 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691892    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:14.697418 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691895    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:14.697418 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691898    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:14.697418 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691901    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:14.697418 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691905    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:14.697418 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691909    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:14.697903 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691913    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:14.697903 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691916    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:14.697903 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691919    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:14.697903 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691922    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:14.697903 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691925    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:14.697903 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691928    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:14.697903 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691930    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:14.697903 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691933    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:14.697903 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691936    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:14.697903 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691939    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:14.697903 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691943    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:14.697903 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691946    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:14.697903 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.691948    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:14.697903 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.691954    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:04:14.697903 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692074    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:14.698311 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692078    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:14.698311 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692081    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:14.698311 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692086    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:14.698311 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692089    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:14.698311 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692093    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:14.698311 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692096    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:14.698311 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692099    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:14.698311 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692102    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:14.698311 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692105    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:14.698311 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692108    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:14.698311 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692111    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:14.698311 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692114    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:14.698311 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692117    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:14.698311 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692120    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:14.698311 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692122    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:14.698311 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692125    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:14.698311 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692129    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:14.698311 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692132    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:14.698311 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692135    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:14.698803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692137    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:14.698803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692140    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:14.698803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692143    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:14.698803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692147    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:14.698803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692151    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:14.698803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692154    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:14.698803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692156    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:14.698803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692159    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:14.698803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692162    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:14.698803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692165    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:14.698803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692168    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:14.698803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692172    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:14.698803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692175    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:14.698803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692178    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:14.698803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692182    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:14.698803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692185    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:14.698803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692189    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:14.698803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692192    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:14.698803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692194    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:14.699282 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692197    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:14.699282 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692200    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:14.699282 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692203    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:14.699282 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692205    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:14.699282 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692208    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:14.699282 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692211    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:14.699282 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692213    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:14.699282 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692216    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:14.699282 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692219    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:14.699282 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692222    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:14.699282 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692225    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:14.699282 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692228    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:14.699282 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692231    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:14.699282 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692233    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:14.699282 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692236    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:14.699282 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692239    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:14.699282 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692242    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:14.699282 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692245    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:14.699282 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692248    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:14.699282 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692250    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:14.699803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692253    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:14.699803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692256    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:14.699803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692259    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:14.699803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692262    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:14.699803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692264    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:14.699803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692267    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:14.699803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692269    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:14.699803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692272    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:14.699803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692275    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:14.699803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692278    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:14.699803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692280    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:14.699803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692283    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:14.699803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692285    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:14.699803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692306    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:14.699803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692309    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:14.699803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692312    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:14.699803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692315    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:14.699803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692318    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:14.699803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692321    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:14.699803 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692324    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:14.700422 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692327    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:14.700422 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692330    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:14.700422 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692332    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:14.700422 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692335    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:14.700422 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692338    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:14.700422 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692341    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:14.700422 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692345    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:14.700422 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692347    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:14.700422 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692350    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:14.700422 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692352    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:14.700422 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692363    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:14.700422 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692366    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:14.700422 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692369    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:14.700422 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692372    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:14.700422 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692374    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:14.700422 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692377    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:14.700422 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692381    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:14.700422 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692383    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:14.700422 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692386    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:14.700422 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:14.692389    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:14.700916 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.692394    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:04:14.700916 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.692512    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 09 18:04:14.700916 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.694702    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 09 18:04:14.700916 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.696320    2563 server.go:1019] "Starting client certificate rotation"
Jul 09 18:04:14.700916 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.696444    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 18:04:14.700916 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.696492    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 18:04:14.700916 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.700006    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 18:04:14.702972 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.702955    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 18:04:14.713283 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.713262    2563 log.go:25] "Validated CRI v1 runtime API"
Jul 09 18:04:14.717357 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.717341    2563 log.go:25] "Validated CRI v1 image API"
Jul 09 18:04:14.718193 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.718168    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 09 18:04:14.722128 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.722104    2563 fs.go:135] Filesystem UUIDs: map[2fecc5dc-72b2-462e-aa0a-e6cfccdacb0f:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d6b17028-2a8f-420e-af26-57feb1470e9c:/dev/nvme0n1p4]
Jul 09 18:04:14.722212 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.722127    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 09 18:04:14.729404 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.729270    2563 manager.go:216] Machine: {Timestamp:2026-07-09 18:04:14.727936021 +0000 UTC m=+0.397016969 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099014 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2501f4d259996ac60f27f4e88e137d SystemUUID:ec2501f4-d259-996a-c60f-27f4e88e137d BootID:85cbc5d3-e48f-4508-bec7-f2f383ac3184 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0e:f8:0f:1f:5b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0e:f8:0f:1f:5b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3e:7c:09:53:45:ef Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 09 18:04:14.729404 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.729394    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 09 18:04:14.729514 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.729480    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 09 18:04:14.732470 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.732453    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 18:04:14.732985 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.732958    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 09 18:04:14.733128 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.732989    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-181.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 09 18:04:14.733934 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.733923    2563 topology_manager.go:138] "Creating topology manager with none policy"
Jul 09 18:04:14.733968 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.733937    2563 container_manager_linux.go:309] "Creating device plugin manager"
Jul 09 18:04:14.733968 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.733951    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 18:04:14.733968 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.733966    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 18:04:14.734972 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.734961    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 09 18:04:14.738681 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.738670    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 18:04:14.738817 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.738808    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 09 18:04:14.738861 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.738854    2563 kubelet.go:486] "Attempting to sync node with API server"
Jul 09 18:04:14.738894 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.738873    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 09 18:04:14.738894 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.738891    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 09 18:04:14.738949 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.738902    2563 kubelet.go:398] "Adding apiserver pod source"
Jul 09 18:04:14.738949 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.738913    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 09 18:04:14.739345 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.739333    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 18:04:14.739380 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.739356    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 18:04:14.742587 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.742562    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 09 18:04:14.744600 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.744585    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 09 18:04:14.744649 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.744618    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 09 18:04:14.745280 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.745267    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 09 18:04:14.745329 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.745287    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 09 18:04:14.745329 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.745308    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 09 18:04:14.745329 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.745318    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 09 18:04:14.745329 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.745329    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 09 18:04:14.745444 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.745336    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 09 18:04:14.745444 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.745430    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 09 18:04:14.745444 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.745436    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 09 18:04:14.745444 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.745443    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 09 18:04:14.745548 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.745449    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 09 18:04:14.745548 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.745458    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 09 18:04:14.745548 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.745469    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 09 18:04:14.746317 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.746286    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 09 18:04:14.746361 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.746320    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 09 18:04:14.750110 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.750093    2563 server.go:1268] "Started kubelet"
Jul 09 18:04:14.750236 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.750209    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 09 18:04:14.751128 ip-10-0-129-181 systemd[1]: Started Kubernetes Kubelet.
Jul 09 18:04:14.751848 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:14.751307    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 09 18:04:14.751848 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.751338    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-181.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 09 18:04:14.751848 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:14.751490    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-181.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 09 18:04:14.751848 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.751511    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 09 18:04:14.751848 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.751606    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 09 18:04:14.754375 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.754356    2563 server.go:310] "Adding debug handlers to kubelet server"
Jul 09 18:04:14.755260 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:14.755233    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 09 18:04:14.755343 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.755278    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 09 18:04:14.759439 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.759420    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 09 18:04:14.759535 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.759495    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 09 18:04:14.759585 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:14.759531    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-181.ec2.internal\" not found"
Jul 09 18:04:14.760197 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.760163    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 09 18:04:14.760311 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.760221    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 09 18:04:14.760311 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.760250    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 09 18:04:14.760311 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.760280    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 09 18:04:14.760466 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.760319    2563 factory.go:55] Registering systemd factory
Jul 09 18:04:14.760466 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.760331    2563 factory.go:223] Registration of the systemd container factory successfully
Jul 09 18:04:14.760466 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.760380    2563 reconstruct.go:97] "Volume reconstruction finished"
Jul 09 18:04:14.760466 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.760391    2563 reconciler.go:29] "Reconciler: start to sync state"
Jul 09 18:04:14.760846 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.760728    2563 factory.go:153] Registering CRI-O factory
Jul 09 18:04:14.760846 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.760744    2563 factory.go:223] Registration of the crio container factory successfully
Jul 09 18:04:14.760846 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.760768    2563 factory.go:103] Registering Raw factory
Jul 09 18:04:14.760846 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.760783    2563 manager.go:1196] Started watching for new ooms in manager
Jul 09 18:04:14.761381 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:14.760054    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-181.ec2.internal.18c0b12bf291b3b3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-181.ec2.internal,UID:ip-10-0-129-181.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-181.ec2.internal,},FirstTimestamp:2026-07-09 18:04:14.750061491 +0000 UTC m=+0.419142440,LastTimestamp:2026-07-09 18:04:14.750061491 +0000 UTC m=+0.419142440,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-181.ec2.internal,}"
Jul 09 18:04:14.762448 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.762431    2563 manager.go:349] Starting recovery of all containers
Jul 09 18:04:14.769575 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:14.769550    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 09 18:04:14.769769 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:14.769747    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-181.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 09 18:04:14.772528 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.772404    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-v9526"
Jul 09 18:04:14.773382 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.773367    2563 manager.go:354] Recovery completed
Jul 09 18:04:14.777770 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.777757    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:14.780096 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.780081    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-181.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:14.780160 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.780111    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-181.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:14.780160 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.780124    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-181.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:14.780701 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.780687    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 09 18:04:14.780766 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.780702    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 09 18:04:14.780766 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.780718    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:14.780766 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.780721    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 18:04:14.781900 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.781882    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-181.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:14.781954 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.781917    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-181.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:14.781954 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.781931    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-181.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:14.782057 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.782040    2563 policy_none.go:49] "None policy: Start"
Jul 09 18:04:14.782104 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.782058    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 09 18:04:14.782104 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.782073    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 09 18:04:14.783277 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:14.783165    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-181.ec2.internal.18c0b12bf45bfe66  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-181.ec2.internal,UID:ip-10-0-129-181.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-181.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-181.ec2.internal,},FirstTimestamp:2026-07-09 18:04:14.780096102 +0000 UTC m=+0.449177051,LastTimestamp:2026-07-09 18:04:14.780096102 +0000 UTC m=+0.449177051,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-181.ec2.internal,}"
Jul 09 18:04:14.783396 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.783383    2563 policy_none.go:47] "Start"
Jul 09 18:04:14.784070 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.784050    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-v9526"
Jul 09 18:04:14.816856 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.816837    2563 manager.go:337] "Starting Device Plugin manager"
Jul 09 18:04:14.816949 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:14.816877    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 09 18:04:14.816949 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.816893    2563 server.go:87] "Starting device plugin registration server"
Jul 09 18:04:14.817180 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.817168    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 09 18:04:14.817239 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.817185    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 09 18:04:14.817374 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.817350    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 09 18:04:14.817480 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.817437    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 09 18:04:14.817480 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.817446    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 09 18:04:14.817925 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:14.817909    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 09 18:04:14.817999 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:14.817961    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-181.ec2.internal\" not found"
Jul 09 18:04:14.917731 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.917695    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:14.919082 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.918770    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-181.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:14.919082 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.918801    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-181.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:14.919082 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.918812    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-181.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:14.919082 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.918846    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-181.ec2.internal"
Jul 09 18:04:14.929154 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.929099    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-181.ec2.internal"
Jul 09 18:04:14.929154 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:14.929126    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-181.ec2.internal\": node \"ip-10-0-129-181.ec2.internal\" not found"
Jul 09 18:04:14.949325 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:14.949296    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-181.ec2.internal\" not found"
Jul 09 18:04:14.952875 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.952843    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 09 18:04:14.954152 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.954139    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 09 18:04:14.954274 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.954177    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 09 18:04:14.954274 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.954202    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 09 18:04:14.954274 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:14.954239    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 09 18:04:14.956873 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:14.956853    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:15.049985 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:15.049946    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-181.ec2.internal\" not found"
Jul 09 18:04:15.055275 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.055233    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-181.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-181.ec2.internal"]
Jul 09 18:04:15.055403 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.055310    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:15.057178 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.057153    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-181.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:15.057277 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.057187    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-181.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:15.057277 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.057204    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-181.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:15.058639 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.058626    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:15.058762 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.058746    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-181.ec2.internal"
Jul 09 18:04:15.058841 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.058782    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:15.059431 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.059412    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-181.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:15.059431 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.059427    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-181.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:15.059605 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.059449    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-181.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:15.059605 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.059462    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-181.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:15.059605 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.059451    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-181.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:15.059605 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.059550    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-181.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:15.060964 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.060948    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-181.ec2.internal"
Jul 09 18:04:15.061057 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.060981    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:15.061544 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.061526    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aa66b6c771563ae19089762114b13fa1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-181.ec2.internal\" (UID: \"aa66b6c771563ae19089762114b13fa1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-181.ec2.internal"
Jul 09 18:04:15.061632 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.061555    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa66b6c771563ae19089762114b13fa1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-181.ec2.internal\" (UID: \"aa66b6c771563ae19089762114b13fa1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-181.ec2.internal"
Jul 09 18:04:15.061632 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.061581    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72b647e293c446e9af86e2db9bbd3ef8-config\") pod \"kube-apiserver-proxy-ip-10-0-129-181.ec2.internal\" (UID: \"72b647e293c446e9af86e2db9bbd3ef8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-181.ec2.internal"
Jul 09 18:04:15.061744 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.061727    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-181.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:15.061788 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.061757    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-181.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:15.061788 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.061773    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-181.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:15.089287 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:15.089261    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-181.ec2.internal\" not found" node="ip-10-0-129-181.ec2.internal"
Jul 09 18:04:15.094029 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:15.094012    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-181.ec2.internal\" not found" node="ip-10-0-129-181.ec2.internal"
Jul 09 18:04:15.151114 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:15.151081    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-181.ec2.internal\" not found"
Jul 09 18:04:15.161691 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.161661    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72b647e293c446e9af86e2db9bbd3ef8-config\") pod \"kube-apiserver-proxy-ip-10-0-129-181.ec2.internal\" (UID: \"72b647e293c446e9af86e2db9bbd3ef8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-181.ec2.internal"
Jul 09 18:04:15.161691 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.161694    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aa66b6c771563ae19089762114b13fa1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-181.ec2.internal\" (UID: \"aa66b6c771563ae19089762114b13fa1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-181.ec2.internal"
Jul 09 18:04:15.161880 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.161713    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa66b6c771563ae19089762114b13fa1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-181.ec2.internal\" (UID: \"aa66b6c771563ae19089762114b13fa1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-181.ec2.internal"
Jul 09 18:04:15.161880 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.161754    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aa66b6c771563ae19089762114b13fa1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-181.ec2.internal\" (UID: \"aa66b6c771563ae19089762114b13fa1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-181.ec2.internal"
Jul 09 18:04:15.161880 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.161762    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72b647e293c446e9af86e2db9bbd3ef8-config\") pod \"kube-apiserver-proxy-ip-10-0-129-181.ec2.internal\" (UID: \"72b647e293c446e9af86e2db9bbd3ef8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-181.ec2.internal"
Jul 09 18:04:15.161880 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.161811    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa66b6c771563ae19089762114b13fa1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-181.ec2.internal\" (UID: \"aa66b6c771563ae19089762114b13fa1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-181.ec2.internal"
Jul 09 18:04:15.251819 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:15.251726    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-181.ec2.internal\" not found"
Jul 09 18:04:15.358821 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:15.358800    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-181.ec2.internal\" not found"
Jul 09 18:04:15.393826 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.393802    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-181.ec2.internal"
Jul 09 18:04:15.398218 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.397858    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-181.ec2.internal"
Jul 09 18:04:15.459705 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:15.459657    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-181.ec2.internal\" not found"
Jul 09 18:04:15.563787 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:15.563716    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-181.ec2.internal\" not found"
Jul 09 18:04:15.664298 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:15.664262    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-181.ec2.internal\" not found"
Jul 09 18:04:15.695557 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.695529    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 09 18:04:15.696063 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.695706    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 18:04:15.765165 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:15.765140    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-181.ec2.internal\" not found"
Jul 09 18:04:15.788491 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.788432    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 17:59:14 +0000 UTC" deadline="2028-01-14 23:45:57.136451191 +0000 UTC"
Jul 09 18:04:15.788491 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.788487    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13301h41m41.347967431s"
Jul 09 18:04:15.865880 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:15.865847    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-181.ec2.internal\" not found"
Jul 09 18:04:15.948123 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:15.948083    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa66b6c771563ae19089762114b13fa1.slice/crio-4a7564a30256bb8ccea4ce2f2db82a98e2042f2df1ed2a2e92577d2836b60af2 WatchSource:0}: Error finding container 4a7564a30256bb8ccea4ce2f2db82a98e2042f2df1ed2a2e92577d2836b60af2: Status 404 returned error can't find the container with id 4a7564a30256bb8ccea4ce2f2db82a98e2042f2df1ed2a2e92577d2836b60af2
Jul 09 18:04:15.955442 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.955426    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:04:15.957183 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.957143    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-181.ec2.internal" event={"ID":"aa66b6c771563ae19089762114b13fa1","Type":"ContainerStarted","Data":"4a7564a30256bb8ccea4ce2f2db82a98e2042f2df1ed2a2e92577d2836b60af2"}
Jul 09 18:04:15.965958 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:15.965939    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-181.ec2.internal\" not found"
Jul 09 18:04:15.971440 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.971405    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:15.987669 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:15.987648    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:16.047194 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:16.047167    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b647e293c446e9af86e2db9bbd3ef8.slice/crio-a5d60edfb6e26e526986584eb7adda35bdaf6f47e8e4071fe9cfe1d2eb3fe07a WatchSource:0}: Error finding container a5d60edfb6e26e526986584eb7adda35bdaf6f47e8e4071fe9cfe1d2eb3fe07a: Status 404 returned error can't find the container with id a5d60edfb6e26e526986584eb7adda35bdaf6f47e8e4071fe9cfe1d2eb3fe07a
Jul 09 18:04:16.060092 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.060070    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-181.ec2.internal"
Jul 09 18:04:16.070336 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.070314    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 18:04:16.071371 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.071358    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-181.ec2.internal"
Jul 09 18:04:16.087288 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.087267    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 18:04:16.191358 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.191330    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:16.740861 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.740823    2563 apiserver.go:52] "Watching apiserver"
Jul 09 18:04:16.749982 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.749960    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 09 18:04:16.751266 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.751236    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-wqgvh","openshift-network-diagnostics/network-check-target-sf462","openshift-ovn-kubernetes/ovnkube-node-lbftk","kube-system/global-pull-secret-syncer-6ng78","kube-system/konnectivity-agent-tvwn2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r5v4p","openshift-dns/node-resolver-rx2rz","openshift-image-registry/node-ca-9xkxg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-181.ec2.internal","openshift-multus/multus-additional-cni-plugins-lljj8","openshift-network-operator/iptables-alerter-6xthg","kube-system/kube-apiserver-proxy-ip-10-0-129-181.ec2.internal","openshift-cluster-node-tuning-operator/tuned-wd27m","openshift-multus/multus-7xtpt"]
Jul 09 18:04:16.752710 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.752686    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9xkxg"
Jul 09 18:04:16.754499 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.754473    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.756274 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.755999    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 09 18:04:16.756274 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.756059    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 09 18:04:16.756274 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.756130    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-l9h99\""
Jul 09 18:04:16.756602 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.756465    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 09 18:04:16.757311 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.757204    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 09 18:04:16.757461 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.757444    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rx2rz"
Jul 09 18:04:16.758705 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.758536    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 09 18:04:16.758850 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.758821    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:16.758941 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:16.758914    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ng78" podUID="cddd2f06-2abf-4246-ac09-acc38c065830"
Jul 09 18:04:16.759864 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.759844    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 09 18:04:16.760001 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.759986    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tvwn2"
Jul 09 18:04:16.760087 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.760069    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2s8ql\""
Jul 09 18:04:16.761545 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.761522    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 09 18:04:16.761875 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.761665    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 09 18:04:16.761875 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.761776    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-shccz\""
Jul 09 18:04:16.761875 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.761792    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 09 18:04:16.762114 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.761893    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 09 18:04:16.763515 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.763368    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 09 18:04:16.763982 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.763964    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r5v4p"
Jul 09 18:04:16.765467 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.765449    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 09 18:04:16.765558 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.765486    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-sjrm8\""
Jul 09 18:04:16.765729 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.765714    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:04:16.765899 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:16.765878    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqgvh" podUID="d5de2134-56a4-4318-a22f-90937385dc5b"
Jul 09 18:04:16.767039 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.767015    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.768810 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.768784    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lljj8"
Jul 09 18:04:16.769502 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.769482    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 09 18:04:16.769670 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.769655    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 09 18:04:16.769992 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.769740    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 09 18:04:16.769992 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.769832    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 09 18:04:16.769992 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.769835    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 09 18:04:16.770268 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770015    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-host-run-multus-certs\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.770268 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770043    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4c9006a0-0f34-4c16-84e4-ccf1ec96bb79-etc-selinux\") pod \"aws-ebs-csi-driver-node-r5v4p\" (UID: \"4c9006a0-0f34-4c16-84e4-ccf1ec96bb79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r5v4p"
Jul 09 18:04:16.770268 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770069    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbtfs\" (UniqueName: \"kubernetes.io/projected/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-kube-api-access-qbtfs\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.770268 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770093    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4c9006a0-0f34-4c16-84e4-ccf1ec96bb79-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r5v4p\" (UID: \"4c9006a0-0f34-4c16-84e4-ccf1ec96bb79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r5v4p"
Jul 09 18:04:16.770268 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770105    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kvh4t\""
Jul 09 18:04:16.770268 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770118    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-host-run-k8s-cni-cncf-io\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.770268 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770139    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-host-var-lib-cni-multus\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.770268 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770153    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-hostroot\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.770268 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770167    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f7fc15e3-7222-4098-8933-cd0b6cd499bd-agent-certs\") pod \"konnectivity-agent-tvwn2\" (UID: \"f7fc15e3-7222-4098-8933-cd0b6cd499bd\") " pod="kube-system/konnectivity-agent-tvwn2"
Jul 09 18:04:16.770268 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770182    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f7fc15e3-7222-4098-8933-cd0b6cd499bd-konnectivity-ca\") pod \"konnectivity-agent-tvwn2\" (UID: \"f7fc15e3-7222-4098-8933-cd0b6cd499bd\") " pod="kube-system/konnectivity-agent-tvwn2"
Jul 09 18:04:16.770268 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770197    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrprl\" (UniqueName: \"kubernetes.io/projected/4c9006a0-0f34-4c16-84e4-ccf1ec96bb79-kube-api-access-nrprl\") pod \"aws-ebs-csi-driver-node-r5v4p\" (UID: \"4c9006a0-0f34-4c16-84e4-ccf1ec96bb79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r5v4p"
Jul 09 18:04:16.770268 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770228    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/67e30329-228a-455d-886a-626b46e438d4-serviceca\") pod \"node-ca-9xkxg\" (UID: \"67e30329-228a-455d-886a-626b46e438d4\") " pod="openshift-image-registry/node-ca-9xkxg"
Jul 09 18:04:16.770268 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770249    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-multus-socket-dir-parent\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.770268 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770268    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-etc-kubernetes\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.770268 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770288    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cddd2f06-2abf-4246-ac09-acc38c065830-dbus\") pod \"global-pull-secret-syncer-6ng78\" (UID: \"cddd2f06-2abf-4246-ac09-acc38c065830\") " pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:16.771171 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770306    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4c9006a0-0f34-4c16-84e4-ccf1ec96bb79-socket-dir\") pod \"aws-ebs-csi-driver-node-r5v4p\" (UID: \"4c9006a0-0f34-4c16-84e4-ccf1ec96bb79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r5v4p"
Jul 09 18:04:16.771171 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770325    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-system-cni-dir\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.771171 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770349    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-multus-cni-dir\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.771171 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770369    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 09 18:04:16.771171 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770377    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cddd2f06-2abf-4246-ac09-acc38c065830-kubelet-config\") pod \"global-pull-secret-syncer-6ng78\" (UID: \"cddd2f06-2abf-4246-ac09-acc38c065830\") " pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:16.771171 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770384    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 09 18:04:16.771171 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770400    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4c9006a0-0f34-4c16-84e4-ccf1ec96bb79-registration-dir\") pod \"aws-ebs-csi-driver-node-r5v4p\" (UID: \"4c9006a0-0f34-4c16-84e4-ccf1ec96bb79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r5v4p"
Jul 09 18:04:16.771171 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770424    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4c9006a0-0f34-4c16-84e4-ccf1ec96bb79-sys-fs\") pod \"aws-ebs-csi-driver-node-r5v4p\" (UID: \"4c9006a0-0f34-4c16-84e4-ccf1ec96bb79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r5v4p"
Jul 09 18:04:16.771171 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770449    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-host-var-lib-kubelet\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.771171 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770483    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/399b7ee4-7292-4fb5-836c-11acc593fa62-hosts-file\") pod \"node-resolver-rx2rz\" (UID: \"399b7ee4-7292-4fb5-836c-11acc593fa62\") " pod="openshift-dns/node-resolver-rx2rz"
Jul 09 18:04:16.771171 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770506    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnhqd\" (UniqueName: \"kubernetes.io/projected/399b7ee4-7292-4fb5-836c-11acc593fa62-kube-api-access-fnhqd\") pod \"node-resolver-rx2rz\" (UID: \"399b7ee4-7292-4fb5-836c-11acc593fa62\") " pod="openshift-dns/node-resolver-rx2rz"
Jul 09 18:04:16.771171 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770535    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-host-run-netns\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.771171 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770590    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67e30329-228a-455d-886a-626b46e438d4-host\") pod \"node-ca-9xkxg\" (UID: \"67e30329-228a-455d-886a-626b46e438d4\") " pod="openshift-image-registry/node-ca-9xkxg"
Jul 09 18:04:16.771171 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770618    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-host-var-lib-cni-bin\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.771171 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770305    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hklls\""
Jul 09 18:04:16.771171 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770252    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 09 18:04:16.771171 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770657    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-multus-conf-dir\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.771171 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770685    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4c9006a0-0f34-4c16-84e4-ccf1ec96bb79-device-dir\") pod \"aws-ebs-csi-driver-node-r5v4p\" (UID: \"4c9006a0-0f34-4c16-84e4-ccf1ec96bb79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r5v4p"
Jul 09 18:04:16.771171 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770733    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv26c\" (UniqueName: \"kubernetes.io/projected/67e30329-228a-455d-886a-626b46e438d4-kube-api-access-cv26c\") pod \"node-ca-9xkxg\" (UID: \"67e30329-228a-455d-886a-626b46e438d4\") " pod="openshift-image-registry/node-ca-9xkxg"
Jul 09 18:04:16.772043 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/399b7ee4-7292-4fb5-836c-11acc593fa62-tmp-dir\") pod \"node-resolver-rx2rz\" (UID: \"399b7ee4-7292-4fb5-836c-11acc593fa62\") " pod="openshift-dns/node-resolver-rx2rz"
Jul 09 18:04:16.772043 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770805    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cddd2f06-2abf-4246-ac09-acc38c065830-original-pull-secret\") pod \"global-pull-secret-syncer-6ng78\" (UID: \"cddd2f06-2abf-4246-ac09-acc38c065830\") " pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:16.772043 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770832    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-cnibin\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.772043 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770872    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-os-release\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.772043 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770895    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-cni-binary-copy\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.772043 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.770967    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-multus-daemon-config\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.772043 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.771024    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6xthg"
Jul 09 18:04:16.772043 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.771133    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 09 18:04:16.772739 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.772681    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 09 18:04:16.772903 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.772879    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 09 18:04:16.773283 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.773261    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-swqr2\""
Jul 09 18:04:16.773396 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.773379    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.774153 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.774137    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:04:16.774428 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.774412    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 09 18:04:16.775241 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.774854    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fm585\""
Jul 09 18:04:16.775241 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.774994    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 09 18:04:16.775241 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.774856    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:04:16.775241 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:16.775083    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sf462" podUID="52ba7983-15e4-437e-a86f-36f0048b0614"
Jul 09 18:04:16.775643 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.775620    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:04:16.775732 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.775679    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 09 18:04:16.775788 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.775620    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mptt9\""
Jul 09 18:04:16.784182 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.784155    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 18:04:16.810331 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.810310    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4xpw7"
Jul 09 18:04:16.817026 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.817004    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:16.820808 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.820783    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4xpw7"
Jul 09 18:04:16.861342 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.861317    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 09 18:04:16.871194 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.871164    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-cni-binary-copy\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.871360 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.871203    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-multus-daemon-config\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.871360 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.871250    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4c9006a0-0f34-4c16-84e4-ccf1ec96bb79-etc-selinux\") pod \"aws-ebs-csi-driver-node-r5v4p\" (UID: \"4c9006a0-0f34-4c16-84e4-ccf1ec96bb79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r5v4p"
Jul 09 18:04:16.871594 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.871569    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.871715 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.871612    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b10ff914-b92d-4b9f-91ec-99c6fcb1425e-iptables-alerter-script\") pod \"iptables-alerter-6xthg\" (UID: \"b10ff914-b92d-4b9f-91ec-99c6fcb1425e\") " pod="openshift-network-operator/iptables-alerter-6xthg"
Jul 09 18:04:16.871715 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.871636    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4c9006a0-0f34-4c16-84e4-ccf1ec96bb79-etc-selinux\") pod \"aws-ebs-csi-driver-node-r5v4p\" (UID: \"4c9006a0-0f34-4c16-84e4-ccf1ec96bb79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r5v4p"
Jul 09 18:04:16.871715 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.871683    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cnjh\" (UniqueName: \"kubernetes.io/projected/b10ff914-b92d-4b9f-91ec-99c6fcb1425e-kube-api-access-7cnjh\") pod \"iptables-alerter-6xthg\" (UID: \"b10ff914-b92d-4b9f-91ec-99c6fcb1425e\") " pod="openshift-network-operator/iptables-alerter-6xthg"
Jul 09 18:04:16.871896 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.871727    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4c9006a0-0f34-4c16-84e4-ccf1ec96bb79-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r5v4p\" (UID: \"4c9006a0-0f34-4c16-84e4-ccf1ec96bb79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r5v4p"
Jul 09 18:04:16.871896 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.871770    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhmkj\" (UniqueName: \"kubernetes.io/projected/d5de2134-56a4-4318-a22f-90937385dc5b-kube-api-access-xhmkj\") pod \"network-metrics-daemon-wqgvh\" (UID: \"d5de2134-56a4-4318-a22f-90937385dc5b\") " pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:04:16.871896 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.871804    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7f2265cf-194a-43a6-b5d4-4fe400c2f429-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lljj8\" (UID: \"7f2265cf-194a-43a6-b5d4-4fe400c2f429\") " pod="openshift-multus/multus-additional-cni-plugins-lljj8"
Jul 09 18:04:16.871896 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.871833    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1360f0da-56fe-485a-93b9-9ee7a46d81a1-var-lib-kubelet\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.871896 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.871835    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4c9006a0-0f34-4c16-84e4-ccf1ec96bb79-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r5v4p\" (UID: \"4c9006a0-0f34-4c16-84e4-ccf1ec96bb79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r5v4p"
Jul 09 18:04:16.871896 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.871863    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-host-var-lib-cni-multus\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.871896 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.871891    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-multus-daemon-config\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.872235 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.871908    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-host-var-lib-cni-multus\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.872235 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.871943    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f7fc15e3-7222-4098-8933-cd0b6cd499bd-konnectivity-ca\") pod \"konnectivity-agent-tvwn2\" (UID: \"f7fc15e3-7222-4098-8933-cd0b6cd499bd\") " pod="kube-system/konnectivity-agent-tvwn2"
Jul 09 18:04:16.872235 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.871968    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-cni-binary-copy\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.872235 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.871973    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrprl\" (UniqueName: \"kubernetes.io/projected/4c9006a0-0f34-4c16-84e4-ccf1ec96bb79-kube-api-access-nrprl\") pod \"aws-ebs-csi-driver-node-r5v4p\" (UID: \"4c9006a0-0f34-4c16-84e4-ccf1ec96bb79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r5v4p"
Jul 09 18:04:16.872235 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872022    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-host-cni-netd\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.872235 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872045    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/67e30329-228a-455d-886a-626b46e438d4-serviceca\") pod \"node-ca-9xkxg\" (UID: \"67e30329-228a-455d-886a-626b46e438d4\") " pod="openshift-image-registry/node-ca-9xkxg"
Jul 09 18:04:16.872235 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872066    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-multus-socket-dir-parent\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.872235 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872108    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-multus-socket-dir-parent\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.872235 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872132    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-etc-kubernetes\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.872235 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872147    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cddd2f06-2abf-4246-ac09-acc38c065830-dbus\") pod \"global-pull-secret-syncer-6ng78\" (UID: \"cddd2f06-2abf-4246-ac09-acc38c065830\") " pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:16.872235 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872161    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5de2134-56a4-4318-a22f-90937385dc5b-metrics-certs\") pod \"network-metrics-daemon-wqgvh\" (UID: \"d5de2134-56a4-4318-a22f-90937385dc5b\") " pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:04:16.872235 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872176    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1360f0da-56fe-485a-93b9-9ee7a46d81a1-run\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.872235 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872209    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1360f0da-56fe-485a-93b9-9ee7a46d81a1-sys\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.872892 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872249    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b10ff914-b92d-4b9f-91ec-99c6fcb1425e-host-slash\") pod \"iptables-alerter-6xthg\" (UID: \"b10ff914-b92d-4b9f-91ec-99c6fcb1425e\") " pod="openshift-network-operator/iptables-alerter-6xthg"
Jul 09 18:04:16.872892 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872276    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-system-cni-dir\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.872892 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872299    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-multus-cni-dir\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.872892 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872307    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-etc-kubernetes\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.872892 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872320    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cddd2f06-2abf-4246-ac09-acc38c065830-dbus\") pod \"global-pull-secret-syncer-6ng78\" (UID: \"cddd2f06-2abf-4246-ac09-acc38c065830\") " pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:16.872892 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872323    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cddd2f06-2abf-4246-ac09-acc38c065830-kubelet-config\") pod \"global-pull-secret-syncer-6ng78\" (UID: \"cddd2f06-2abf-4246-ac09-acc38c065830\") " pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:16.872892 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872365    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f2265cf-194a-43a6-b5d4-4fe400c2f429-cnibin\") pod \"multus-additional-cni-plugins-lljj8\" (UID: \"7f2265cf-194a-43a6-b5d4-4fe400c2f429\") " pod="openshift-multus/multus-additional-cni-plugins-lljj8"
Jul 09 18:04:16.872892 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872377    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-system-cni-dir\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.872892 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872379    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cddd2f06-2abf-4246-ac09-acc38c065830-kubelet-config\") pod \"global-pull-secret-syncer-6ng78\" (UID: \"cddd2f06-2abf-4246-ac09-acc38c065830\") " pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:16.872892 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872389    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-host-slash\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.872892 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872399    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-multus-cni-dir\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.872892 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872424    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-host-var-lib-kubelet\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.872892 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872450    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/399b7ee4-7292-4fb5-836c-11acc593fa62-hosts-file\") pod \"node-resolver-rx2rz\" (UID: \"399b7ee4-7292-4fb5-836c-11acc593fa62\") " pod="openshift-dns/node-resolver-rx2rz"
Jul 09 18:04:16.872892 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872458    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-host-var-lib-kubelet\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.872892 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872462    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/67e30329-228a-455d-886a-626b46e438d4-serviceca\") pod \"node-ca-9xkxg\" (UID: \"67e30329-228a-455d-886a-626b46e438d4\") " pod="openshift-image-registry/node-ca-9xkxg"
Jul 09 18:04:16.872892 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872473    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnhqd\" (UniqueName: \"kubernetes.io/projected/399b7ee4-7292-4fb5-836c-11acc593fa62-kube-api-access-fnhqd\") pod \"node-resolver-rx2rz\" (UID: \"399b7ee4-7292-4fb5-836c-11acc593fa62\") " pod="openshift-dns/node-resolver-rx2rz"
Jul 09 18:04:16.872892 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872504    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-host-kubelet\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.872892 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872516    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/399b7ee4-7292-4fb5-836c-11acc593fa62-hosts-file\") pod \"node-resolver-rx2rz\" (UID: \"399b7ee4-7292-4fb5-836c-11acc593fa62\") " pod="openshift-dns/node-resolver-rx2rz"
Jul 09 18:04:16.873737 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872529    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-host-run-netns\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.873737 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872556    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77wgq\" (UniqueName: \"kubernetes.io/projected/5d463fb2-45f9-4554-9525-637e1b84351a-kube-api-access-77wgq\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.873737 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872583    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-host-run-netns\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.873737 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872604    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-multus-conf-dir\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.873737 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872627    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4c9006a0-0f34-4c16-84e4-ccf1ec96bb79-device-dir\") pod \"aws-ebs-csi-driver-node-r5v4p\" (UID: \"4c9006a0-0f34-4c16-84e4-ccf1ec96bb79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r5v4p"
Jul 09 18:04:16.873737 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872650    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1360f0da-56fe-485a-93b9-9ee7a46d81a1-etc-kubernetes\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.873737 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872675    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-log-socket\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.873737 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872649    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-host-run-netns\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.873737 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872689    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-multus-conf-dir\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.873737 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872697    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5d463fb2-45f9-4554-9525-637e1b84351a-ovn-node-metrics-cert\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.873737 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872727    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4c9006a0-0f34-4c16-84e4-ccf1ec96bb79-device-dir\") pod \"aws-ebs-csi-driver-node-r5v4p\" (UID: \"4c9006a0-0f34-4c16-84e4-ccf1ec96bb79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r5v4p"
Jul 09 18:04:16.873737 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872735    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv26c\" (UniqueName: \"kubernetes.io/projected/67e30329-228a-455d-886a-626b46e438d4-kube-api-access-cv26c\") pod \"node-ca-9xkxg\" (UID: \"67e30329-228a-455d-886a-626b46e438d4\") " pod="openshift-image-registry/node-ca-9xkxg"
Jul 09 18:04:16.873737 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872764    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cddd2f06-2abf-4246-ac09-acc38c065830-original-pull-secret\") pod \"global-pull-secret-syncer-6ng78\" (UID: \"cddd2f06-2abf-4246-ac09-acc38c065830\") " pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:16.873737 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872790    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7f2265cf-194a-43a6-b5d4-4fe400c2f429-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lljj8\" (UID: \"7f2265cf-194a-43a6-b5d4-4fe400c2f429\") " pod="openshift-multus/multus-additional-cni-plugins-lljj8"
Jul 09 18:04:16.873737 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872815    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1360f0da-56fe-485a-93b9-9ee7a46d81a1-etc-sysctl-d\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.873737 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872837    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1360f0da-56fe-485a-93b9-9ee7a46d81a1-lib-modules\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.873737 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:16.872911    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:16.874469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.872947    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-host-run-ovn-kubernetes\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.874469 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:16.872992    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cddd2f06-2abf-4246-ac09-acc38c065830-original-pull-secret podName:cddd2f06-2abf-4246-ac09-acc38c065830 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:17.372959102 +0000 UTC m=+3.042040052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cddd2f06-2abf-4246-ac09-acc38c065830-original-pull-secret") pod "global-pull-secret-syncer-6ng78" (UID: "cddd2f06-2abf-4246-ac09-acc38c065830") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:16.874469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873010    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-os-release\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.874469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873042    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-host-run-multus-certs\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.874469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873071    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f2265cf-194a-43a6-b5d4-4fe400c2f429-cni-binary-copy\") pod \"multus-additional-cni-plugins-lljj8\" (UID: \"7f2265cf-194a-43a6-b5d4-4fe400c2f429\") " pod="openshift-multus/multus-additional-cni-plugins-lljj8"
Jul 09 18:04:16.874469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873093    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-os-release\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.874469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873097    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1360f0da-56fe-485a-93b9-9ee7a46d81a1-etc-modprobe-d\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.874469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873130    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1360f0da-56fe-485a-93b9-9ee7a46d81a1-etc-sysconfig\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.874469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873143    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f7fc15e3-7222-4098-8933-cd0b6cd499bd-konnectivity-ca\") pod \"konnectivity-agent-tvwn2\" (UID: \"f7fc15e3-7222-4098-8933-cd0b6cd499bd\") " pod="kube-system/konnectivity-agent-tvwn2"
Jul 09 18:04:16.874469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873181    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-host-run-multus-certs\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.874469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873200    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbtfs\" (UniqueName: \"kubernetes.io/projected/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-kube-api-access-qbtfs\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.874469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873250    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1360f0da-56fe-485a-93b9-9ee7a46d81a1-etc-systemd\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.874469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873275    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxzgc\" (UniqueName: \"kubernetes.io/projected/52ba7983-15e4-437e-a86f-36f0048b0614-kube-api-access-lxzgc\") pod \"network-check-target-sf462\" (UID: \"52ba7983-15e4-437e-a86f-36f0048b0614\") " pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:04:16.874469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873306    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-run-systemd\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.874469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873338    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-node-log\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.874469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873376    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-host-run-k8s-cni-cncf-io\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.875289 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873403    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-hostroot\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.875289 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873429    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f7fc15e3-7222-4098-8933-cd0b6cd499bd-agent-certs\") pod \"konnectivity-agent-tvwn2\" (UID: \"f7fc15e3-7222-4098-8933-cd0b6cd499bd\") " pod="kube-system/konnectivity-agent-tvwn2"
Jul 09 18:04:16.875289 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873436    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-host-run-k8s-cni-cncf-io\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.875289 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873481    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-hostroot\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.875289 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873481    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f2265cf-194a-43a6-b5d4-4fe400c2f429-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lljj8\" (UID: \"7f2265cf-194a-43a6-b5d4-4fe400c2f429\") " pod="openshift-multus/multus-additional-cni-plugins-lljj8"
Jul 09 18:04:16.875289 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873569    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1360f0da-56fe-485a-93b9-9ee7a46d81a1-etc-tuned\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.875289 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873606    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5d463fb2-45f9-4554-9525-637e1b84351a-ovnkube-config\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.875289 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873645    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4c9006a0-0f34-4c16-84e4-ccf1ec96bb79-socket-dir\") pod \"aws-ebs-csi-driver-node-r5v4p\" (UID: \"4c9006a0-0f34-4c16-84e4-ccf1ec96bb79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r5v4p"
Jul 09 18:04:16.875289 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873671    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f2265cf-194a-43a6-b5d4-4fe400c2f429-system-cni-dir\") pod \"multus-additional-cni-plugins-lljj8\" (UID: \"7f2265cf-194a-43a6-b5d4-4fe400c2f429\") " pod="openshift-multus/multus-additional-cni-plugins-lljj8"
Jul 09 18:04:16.875289 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873696    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1360f0da-56fe-485a-93b9-9ee7a46d81a1-host\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.875289 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873719    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-systemd-units\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.875289 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873728    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4c9006a0-0f34-4c16-84e4-ccf1ec96bb79-socket-dir\") pod \"aws-ebs-csi-driver-node-r5v4p\" (UID: \"4c9006a0-0f34-4c16-84e4-ccf1ec96bb79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r5v4p"
Jul 09 18:04:16.875289 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873740    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-run-openvswitch\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.875289 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873763    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-run-ovn\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.875289 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873757    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 09 18:04:16.875289 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873792    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4c9006a0-0f34-4c16-84e4-ccf1ec96bb79-registration-dir\") pod \"aws-ebs-csi-driver-node-r5v4p\" (UID: \"4c9006a0-0f34-4c16-84e4-ccf1ec96bb79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r5v4p"
Jul 09 18:04:16.875289 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873815    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4c9006a0-0f34-4c16-84e4-ccf1ec96bb79-sys-fs\") pod \"aws-ebs-csi-driver-node-r5v4p\" (UID: \"4c9006a0-0f34-4c16-84e4-ccf1ec96bb79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r5v4p"
Jul 09 18:04:16.876033 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873842    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7krj\" (UniqueName: \"kubernetes.io/projected/7f2265cf-194a-43a6-b5d4-4fe400c2f429-kube-api-access-x7krj\") pod \"multus-additional-cni-plugins-lljj8\" (UID: \"7f2265cf-194a-43a6-b5d4-4fe400c2f429\") " pod="openshift-multus/multus-additional-cni-plugins-lljj8"
Jul 09 18:04:16.876033 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873856    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4c9006a0-0f34-4c16-84e4-ccf1ec96bb79-sys-fs\") pod \"aws-ebs-csi-driver-node-r5v4p\" (UID: \"4c9006a0-0f34-4c16-84e4-ccf1ec96bb79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r5v4p"
Jul 09 18:04:16.876033 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873864    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1360f0da-56fe-485a-93b9-9ee7a46d81a1-tmp\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.876033 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873877    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4c9006a0-0f34-4c16-84e4-ccf1ec96bb79-registration-dir\") pod \"aws-ebs-csi-driver-node-r5v4p\" (UID: \"4c9006a0-0f34-4c16-84e4-ccf1ec96bb79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r5v4p"
Jul 09 18:04:16.876033 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873889    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-host-cni-bin\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.876033 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873923    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tzcc\" (UniqueName: \"kubernetes.io/projected/1360f0da-56fe-485a-93b9-9ee7a46d81a1-kube-api-access-9tzcc\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.876033 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.873949    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-var-lib-openvswitch\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.876033 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.874006    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-etc-openvswitch\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.876033 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.874034    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67e30329-228a-455d-886a-626b46e438d4-host\") pod \"node-ca-9xkxg\" (UID: \"67e30329-228a-455d-886a-626b46e438d4\") " pod="openshift-image-registry/node-ca-9xkxg"
Jul 09 18:04:16.876033 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.874057    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-host-var-lib-cni-bin\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.876033 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.874079    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67e30329-228a-455d-886a-626b46e438d4-host\") pod \"node-ca-9xkxg\" (UID: \"67e30329-228a-455d-886a-626b46e438d4\") " pod="openshift-image-registry/node-ca-9xkxg"
Jul 09 18:04:16.876033 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.874081    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7f2265cf-194a-43a6-b5d4-4fe400c2f429-os-release\") pod \"multus-additional-cni-plugins-lljj8\" (UID: \"7f2265cf-194a-43a6-b5d4-4fe400c2f429\") " pod="openshift-multus/multus-additional-cni-plugins-lljj8"
Jul 09 18:04:16.876033 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.874125    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-host-var-lib-cni-bin\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.876033 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.874125    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5d463fb2-45f9-4554-9525-637e1b84351a-ovnkube-script-lib\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.876033 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.874165    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/399b7ee4-7292-4fb5-836c-11acc593fa62-tmp-dir\") pod \"node-resolver-rx2rz\" (UID: \"399b7ee4-7292-4fb5-836c-11acc593fa62\") " pod="openshift-dns/node-resolver-rx2rz"
Jul 09 18:04:16.876033 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.874189    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1360f0da-56fe-485a-93b9-9ee7a46d81a1-etc-sysctl-conf\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.876033 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.874228    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5d463fb2-45f9-4554-9525-637e1b84351a-env-overrides\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.876826 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.874262    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-cnibin\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.876826 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.874312    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-cnibin\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.876826 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.874995    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/399b7ee4-7292-4fb5-836c-11acc593fa62-tmp-dir\") pod \"node-resolver-rx2rz\" (UID: \"399b7ee4-7292-4fb5-836c-11acc593fa62\") " pod="openshift-dns/node-resolver-rx2rz"
Jul 09 18:04:16.877678 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.877656    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f7fc15e3-7222-4098-8933-cd0b6cd499bd-agent-certs\") pod \"konnectivity-agent-tvwn2\" (UID: \"f7fc15e3-7222-4098-8933-cd0b6cd499bd\") " pod="kube-system/konnectivity-agent-tvwn2"
Jul 09 18:04:16.881152 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.881102    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrprl\" (UniqueName: \"kubernetes.io/projected/4c9006a0-0f34-4c16-84e4-ccf1ec96bb79-kube-api-access-nrprl\") pod \"aws-ebs-csi-driver-node-r5v4p\" (UID: \"4c9006a0-0f34-4c16-84e4-ccf1ec96bb79\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r5v4p"
Jul 09 18:04:16.884786 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.884762    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv26c\" (UniqueName: \"kubernetes.io/projected/67e30329-228a-455d-886a-626b46e438d4-kube-api-access-cv26c\") pod \"node-ca-9xkxg\" (UID: \"67e30329-228a-455d-886a-626b46e438d4\") " pod="openshift-image-registry/node-ca-9xkxg"
Jul 09 18:04:16.887036 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.887016    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbtfs\" (UniqueName: \"kubernetes.io/projected/30ba50d7-581d-4b1b-9e63-7e1fe0ea9137-kube-api-access-qbtfs\") pod \"multus-7xtpt\" (UID: \"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137\") " pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:16.887417 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.887394    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnhqd\" (UniqueName: \"kubernetes.io/projected/399b7ee4-7292-4fb5-836c-11acc593fa62-kube-api-access-fnhqd\") pod \"node-resolver-rx2rz\" (UID: \"399b7ee4-7292-4fb5-836c-11acc593fa62\") " pod="openshift-dns/node-resolver-rx2rz"
Jul 09 18:04:16.960708 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.960665    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-181.ec2.internal" event={"ID":"72b647e293c446e9af86e2db9bbd3ef8","Type":"ContainerStarted","Data":"a5d60edfb6e26e526986584eb7adda35bdaf6f47e8e4071fe9cfe1d2eb3fe07a"}
Jul 09 18:04:16.975334 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.975304    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f2265cf-194a-43a6-b5d4-4fe400c2f429-cnibin\") pod \"multus-additional-cni-plugins-lljj8\" (UID: \"7f2265cf-194a-43a6-b5d4-4fe400c2f429\") " pod="openshift-multus/multus-additional-cni-plugins-lljj8"
Jul 09 18:04:16.975501 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.975349    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-host-slash\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.975501 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.975380    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-host-kubelet\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.975501 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.975403    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-host-run-netns\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.975501 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.975426    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77wgq\" (UniqueName: \"kubernetes.io/projected/5d463fb2-45f9-4554-9525-637e1b84351a-kube-api-access-77wgq\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.975501 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.975432    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f2265cf-194a-43a6-b5d4-4fe400c2f429-cnibin\") pod \"multus-additional-cni-plugins-lljj8\" (UID: \"7f2265cf-194a-43a6-b5d4-4fe400c2f429\") " pod="openshift-multus/multus-additional-cni-plugins-lljj8"
Jul 09 18:04:16.975501 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.975443    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-host-slash\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.975501 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.975451    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1360f0da-56fe-485a-93b9-9ee7a46d81a1-etc-kubernetes\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.975501 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.975477    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-host-run-netns\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.975501 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.975487    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-log-socket\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.975501 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.975443    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-host-kubelet\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.975949 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.975513    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1360f0da-56fe-485a-93b9-9ee7a46d81a1-etc-kubernetes\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.975949 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.975518    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5d463fb2-45f9-4554-9525-637e1b84351a-ovn-node-metrics-cert\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.975949 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.975529    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-log-socket\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.975949 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.975581    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7f2265cf-194a-43a6-b5d4-4fe400c2f429-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lljj8\" (UID: \"7f2265cf-194a-43a6-b5d4-4fe400c2f429\") " pod="openshift-multus/multus-additional-cni-plugins-lljj8"
Jul 09 18:04:16.975949 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.975612    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1360f0da-56fe-485a-93b9-9ee7a46d81a1-etc-sysctl-d\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.975949 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.975686    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1360f0da-56fe-485a-93b9-9ee7a46d81a1-lib-modules\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.975949 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.975717    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-host-run-ovn-kubernetes\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.975949 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.975745    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f2265cf-194a-43a6-b5d4-4fe400c2f429-cni-binary-copy\") pod \"multus-additional-cni-plugins-lljj8\" (UID: \"7f2265cf-194a-43a6-b5d4-4fe400c2f429\") " pod="openshift-multus/multus-additional-cni-plugins-lljj8"
Jul 09 18:04:16.975949 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.975769    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1360f0da-56fe-485a-93b9-9ee7a46d81a1-etc-modprobe-d\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.975949 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.975773    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1360f0da-56fe-485a-93b9-9ee7a46d81a1-etc-sysctl-d\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.975949 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.975793    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1360f0da-56fe-485a-93b9-9ee7a46d81a1-etc-sysconfig\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.975949 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.975831    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-host-run-ovn-kubernetes\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.975949 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.975844    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1360f0da-56fe-485a-93b9-9ee7a46d81a1-etc-systemd\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.975949 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.975869    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxzgc\" (UniqueName: \"kubernetes.io/projected/52ba7983-15e4-437e-a86f-36f0048b0614-kube-api-access-lxzgc\") pod \"network-check-target-sf462\" (UID: \"52ba7983-15e4-437e-a86f-36f0048b0614\") " pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:04:16.975949 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.975895    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-run-systemd\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.975949 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.975917    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-node-log\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.975949 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.975946    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f2265cf-194a-43a6-b5d4-4fe400c2f429-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lljj8\" (UID: \"7f2265cf-194a-43a6-b5d4-4fe400c2f429\") " pod="openshift-multus/multus-additional-cni-plugins-lljj8"
Jul 09 18:04:16.976751 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.975952    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1360f0da-56fe-485a-93b9-9ee7a46d81a1-lib-modules\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.976751 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.975970    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1360f0da-56fe-485a-93b9-9ee7a46d81a1-etc-tuned\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.976751 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.975999    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5d463fb2-45f9-4554-9525-637e1b84351a-ovnkube-config\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.976751 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976023    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f2265cf-194a-43a6-b5d4-4fe400c2f429-system-cni-dir\") pod \"multus-additional-cni-plugins-lljj8\" (UID: \"7f2265cf-194a-43a6-b5d4-4fe400c2f429\") " pod="openshift-multus/multus-additional-cni-plugins-lljj8"
Jul 09 18:04:16.976751 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976048    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1360f0da-56fe-485a-93b9-9ee7a46d81a1-host\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.976751 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976072    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-systemd-units\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.976751 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976097    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-run-openvswitch\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.976751 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976103    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-run-systemd\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.976751 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976118    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-run-ovn\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.976751 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976146    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7krj\" (UniqueName: \"kubernetes.io/projected/7f2265cf-194a-43a6-b5d4-4fe400c2f429-kube-api-access-x7krj\") pod \"multus-additional-cni-plugins-lljj8\" (UID: \"7f2265cf-194a-43a6-b5d4-4fe400c2f429\") " pod="openshift-multus/multus-additional-cni-plugins-lljj8"
Jul 09 18:04:16.976751 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976187    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1360f0da-56fe-485a-93b9-9ee7a46d81a1-tmp\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.976751 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976203    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1360f0da-56fe-485a-93b9-9ee7a46d81a1-etc-modprobe-d\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.976751 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976210    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-host-cni-bin\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.976751 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976205    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7f2265cf-194a-43a6-b5d4-4fe400c2f429-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lljj8\" (UID: \"7f2265cf-194a-43a6-b5d4-4fe400c2f429\") " pod="openshift-multus/multus-additional-cni-plugins-lljj8"
Jul 09 18:04:16.976751 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976266    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1360f0da-56fe-485a-93b9-9ee7a46d81a1-etc-sysconfig\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.976751 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976277    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-host-cni-bin\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.976751 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976297    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tzcc\" (UniqueName: \"kubernetes.io/projected/1360f0da-56fe-485a-93b9-9ee7a46d81a1-kube-api-access-9tzcc\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.977547 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976319    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-node-log\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.977547 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976325    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-var-lib-openvswitch\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.977547 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976356    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-var-lib-openvswitch\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.977547 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976356    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-etc-openvswitch\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.977547 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976388    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7f2265cf-194a-43a6-b5d4-4fe400c2f429-os-release\") pod \"multus-additional-cni-plugins-lljj8\" (UID: \"7f2265cf-194a-43a6-b5d4-4fe400c2f429\") " pod="openshift-multus/multus-additional-cni-plugins-lljj8"
Jul 09 18:04:16.977547 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976397    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1360f0da-56fe-485a-93b9-9ee7a46d81a1-etc-systemd\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.977547 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976414    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5d463fb2-45f9-4554-9525-637e1b84351a-ovnkube-script-lib\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.977547 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976440    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1360f0da-56fe-485a-93b9-9ee7a46d81a1-etc-sysctl-conf\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.977547 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976473    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5d463fb2-45f9-4554-9525-637e1b84351a-env-overrides\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.977547 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976527    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-run-openvswitch\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.977547 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976565    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.977547 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976594    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b10ff914-b92d-4b9f-91ec-99c6fcb1425e-iptables-alerter-script\") pod \"iptables-alerter-6xthg\" (UID: \"b10ff914-b92d-4b9f-91ec-99c6fcb1425e\") " pod="openshift-network-operator/iptables-alerter-6xthg"
Jul 09 18:04:16.977547 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976621    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cnjh\" (UniqueName: \"kubernetes.io/projected/b10ff914-b92d-4b9f-91ec-99c6fcb1425e-kube-api-access-7cnjh\") pod \"iptables-alerter-6xthg\" (UID: \"b10ff914-b92d-4b9f-91ec-99c6fcb1425e\") " pod="openshift-network-operator/iptables-alerter-6xthg"
Jul 09 18:04:16.977547 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976637    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f2265cf-194a-43a6-b5d4-4fe400c2f429-cni-binary-copy\") pod \"multus-additional-cni-plugins-lljj8\" (UID: \"7f2265cf-194a-43a6-b5d4-4fe400c2f429\") " pod="openshift-multus/multus-additional-cni-plugins-lljj8"
Jul 09 18:04:16.977547 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976649    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhmkj\" (UniqueName: \"kubernetes.io/projected/d5de2134-56a4-4318-a22f-90937385dc5b-kube-api-access-xhmkj\") pod \"network-metrics-daemon-wqgvh\" (UID: \"d5de2134-56a4-4318-a22f-90937385dc5b\") " pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:04:16.977547 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976674    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7f2265cf-194a-43a6-b5d4-4fe400c2f429-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lljj8\" (UID: \"7f2265cf-194a-43a6-b5d4-4fe400c2f429\") " pod="openshift-multus/multus-additional-cni-plugins-lljj8"
Jul 09 18:04:16.977547 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976700    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1360f0da-56fe-485a-93b9-9ee7a46d81a1-var-lib-kubelet\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.978378 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976712    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1360f0da-56fe-485a-93b9-9ee7a46d81a1-host\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.978378 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976733    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-host-cni-netd\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.978378 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976762    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5de2134-56a4-4318-a22f-90937385dc5b-metrics-certs\") pod \"network-metrics-daemon-wqgvh\" (UID: \"d5de2134-56a4-4318-a22f-90937385dc5b\") " pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:04:16.978378 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976778    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-etc-openvswitch\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.978378 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976784    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1360f0da-56fe-485a-93b9-9ee7a46d81a1-run\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.978378 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976823    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1360f0da-56fe-485a-93b9-9ee7a46d81a1-sys\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.978378 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976828    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1360f0da-56fe-485a-93b9-9ee7a46d81a1-run\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.978378 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976838    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f2265cf-194a-43a6-b5d4-4fe400c2f429-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lljj8\" (UID: \"7f2265cf-194a-43a6-b5d4-4fe400c2f429\") " pod="openshift-multus/multus-additional-cni-plugins-lljj8"
Jul 09 18:04:16.978378 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976848    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b10ff914-b92d-4b9f-91ec-99c6fcb1425e-host-slash\") pod \"iptables-alerter-6xthg\" (UID: \"b10ff914-b92d-4b9f-91ec-99c6fcb1425e\") " pod="openshift-network-operator/iptables-alerter-6xthg"
Jul 09 18:04:16.978378 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976874    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-systemd-units\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.978378 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976890    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f2265cf-194a-43a6-b5d4-4fe400c2f429-system-cni-dir\") pod \"multus-additional-cni-plugins-lljj8\" (UID: \"7f2265cf-194a-43a6-b5d4-4fe400c2f429\") " pod="openshift-multus/multus-additional-cni-plugins-lljj8"
Jul 09 18:04:16.978378 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976913    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.978378 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976907    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5d463fb2-45f9-4554-9525-637e1b84351a-env-overrides\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.978378 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.976967    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-host-cni-netd\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.978378 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.977009    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1360f0da-56fe-485a-93b9-9ee7a46d81a1-var-lib-kubelet\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.978378 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.977015    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7f2265cf-194a-43a6-b5d4-4fe400c2f429-os-release\") pod \"multus-additional-cni-plugins-lljj8\" (UID: \"7f2265cf-194a-43a6-b5d4-4fe400c2f429\") " pod="openshift-multus/multus-additional-cni-plugins-lljj8"
Jul 09 18:04:16.978378 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.977416    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1360f0da-56fe-485a-93b9-9ee7a46d81a1-sys\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.978378 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.977421    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5d463fb2-45f9-4554-9525-637e1b84351a-ovnkube-config\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.979123 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:16.977517    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:16.979123 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.977529    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b10ff914-b92d-4b9f-91ec-99c6fcb1425e-iptables-alerter-script\") pod \"iptables-alerter-6xthg\" (UID: \"b10ff914-b92d-4b9f-91ec-99c6fcb1425e\") " pod="openshift-network-operator/iptables-alerter-6xthg"
Jul 09 18:04:16.979123 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.977566    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7f2265cf-194a-43a6-b5d4-4fe400c2f429-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lljj8\" (UID: \"7f2265cf-194a-43a6-b5d4-4fe400c2f429\") " pod="openshift-multus/multus-additional-cni-plugins-lljj8"
Jul 09 18:04:16.979123 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:16.977581    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5de2134-56a4-4318-a22f-90937385dc5b-metrics-certs podName:d5de2134-56a4-4318-a22f-90937385dc5b nodeName:}" failed. No retries permitted until 2026-07-09 18:04:17.477561654 +0000 UTC m=+3.146642588 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5de2134-56a4-4318-a22f-90937385dc5b-metrics-certs") pod "network-metrics-daemon-wqgvh" (UID: "d5de2134-56a4-4318-a22f-90937385dc5b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:16.979123 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.977590    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5d463fb2-45f9-4554-9525-637e1b84351a-ovnkube-script-lib\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.979123 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.977627    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b10ff914-b92d-4b9f-91ec-99c6fcb1425e-host-slash\") pod \"iptables-alerter-6xthg\" (UID: \"b10ff914-b92d-4b9f-91ec-99c6fcb1425e\") " pod="openshift-network-operator/iptables-alerter-6xthg"
Jul 09 18:04:16.979123 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.977641    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5d463fb2-45f9-4554-9525-637e1b84351a-run-ovn\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.979123 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.977682    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1360f0da-56fe-485a-93b9-9ee7a46d81a1-etc-sysctl-conf\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.979123 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.978360    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5d463fb2-45f9-4554-9525-637e1b84351a-ovn-node-metrics-cert\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:16.979504 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.979383    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1360f0da-56fe-485a-93b9-9ee7a46d81a1-tmp\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:16.980322 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:16.980306    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1360f0da-56fe-485a-93b9-9ee7a46d81a1-etc-tuned\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:17.006961 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:17.006856    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7krj\" (UniqueName: \"kubernetes.io/projected/7f2265cf-194a-43a6-b5d4-4fe400c2f429-kube-api-access-x7krj\") pod \"multus-additional-cni-plugins-lljj8\" (UID: \"7f2265cf-194a-43a6-b5d4-4fe400c2f429\") " pod="openshift-multus/multus-additional-cni-plugins-lljj8"
Jul 09 18:04:17.006961 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:17.006881    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:17.006961 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:17.006904    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:17.006961 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:17.006921    2563 projected.go:196] Error preparing data for projected volume kube-api-access-lxzgc for pod openshift-network-diagnostics/network-check-target-sf462: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:17.007231 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:17.006987    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52ba7983-15e4-437e-a86f-36f0048b0614-kube-api-access-lxzgc podName:52ba7983-15e4-437e-a86f-36f0048b0614 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:17.506967707 +0000 UTC m=+3.176048659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lxzgc" (UniqueName: "kubernetes.io/projected/52ba7983-15e4-437e-a86f-36f0048b0614-kube-api-access-lxzgc") pod "network-check-target-sf462" (UID: "52ba7983-15e4-437e-a86f-36f0048b0614") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:17.007231 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:17.007050    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhmkj\" (UniqueName: \"kubernetes.io/projected/d5de2134-56a4-4318-a22f-90937385dc5b-kube-api-access-xhmkj\") pod \"network-metrics-daemon-wqgvh\" (UID: \"d5de2134-56a4-4318-a22f-90937385dc5b\") " pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:04:17.007971 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:17.007946    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tzcc\" (UniqueName: \"kubernetes.io/projected/1360f0da-56fe-485a-93b9-9ee7a46d81a1-kube-api-access-9tzcc\") pod \"tuned-wd27m\" (UID: \"1360f0da-56fe-485a-93b9-9ee7a46d81a1\") " pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:17.009974 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:17.009951    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77wgq\" (UniqueName: \"kubernetes.io/projected/5d463fb2-45f9-4554-9525-637e1b84351a-kube-api-access-77wgq\") pod \"ovnkube-node-lbftk\" (UID: \"5d463fb2-45f9-4554-9525-637e1b84351a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:17.010140 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:17.010121    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cnjh\" (UniqueName: \"kubernetes.io/projected/b10ff914-b92d-4b9f-91ec-99c6fcb1425e-kube-api-access-7cnjh\") pod \"iptables-alerter-6xthg\" (UID: \"b10ff914-b92d-4b9f-91ec-99c6fcb1425e\") " pod="openshift-network-operator/iptables-alerter-6xthg"
Jul 09 18:04:17.066620 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:17.066578    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9xkxg"
Jul 09 18:04:17.076808 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:17.076779    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7xtpt"
Jul 09 18:04:17.086579 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:17.086555    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rx2rz"
Jul 09 18:04:17.092396 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:17.092373    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tvwn2"
Jul 09 18:04:17.100077 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:17.100050    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r5v4p"
Jul 09 18:04:17.107803 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:17.107779    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:17.116410 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:17.116387    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lljj8"
Jul 09 18:04:17.123448 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:17.123430    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6xthg"
Jul 09 18:04:17.132199 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:17.132181    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wd27m"
Jul 09 18:04:17.380346 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:17.380265    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cddd2f06-2abf-4246-ac09-acc38c065830-original-pull-secret\") pod \"global-pull-secret-syncer-6ng78\" (UID: \"cddd2f06-2abf-4246-ac09-acc38c065830\") " pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:17.380496 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:17.380417    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:17.380496 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:17.380479    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cddd2f06-2abf-4246-ac09-acc38c065830-original-pull-secret podName:cddd2f06-2abf-4246-ac09-acc38c065830 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:18.3804628 +0000 UTC m=+4.049543747 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cddd2f06-2abf-4246-ac09-acc38c065830-original-pull-secret") pod "global-pull-secret-syncer-6ng78" (UID: "cddd2f06-2abf-4246-ac09-acc38c065830") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:17.480660 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:17.480624    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5de2134-56a4-4318-a22f-90937385dc5b-metrics-certs\") pod \"network-metrics-daemon-wqgvh\" (UID: \"d5de2134-56a4-4318-a22f-90937385dc5b\") " pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:04:17.480827 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:17.480778    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:17.480880 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:17.480845    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5de2134-56a4-4318-a22f-90937385dc5b-metrics-certs podName:d5de2134-56a4-4318-a22f-90937385dc5b nodeName:}" failed. No retries permitted until 2026-07-09 18:04:18.480826716 +0000 UTC m=+4.149907649 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5de2134-56a4-4318-a22f-90937385dc5b-metrics-certs") pod "network-metrics-daemon-wqgvh" (UID: "d5de2134-56a4-4318-a22f-90937385dc5b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:17.581151 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:17.581117    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxzgc\" (UniqueName: \"kubernetes.io/projected/52ba7983-15e4-437e-a86f-36f0048b0614-kube-api-access-lxzgc\") pod \"network-check-target-sf462\" (UID: \"52ba7983-15e4-437e-a86f-36f0048b0614\") " pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:04:17.581328 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:17.581253    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:17.581328 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:17.581270    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:17.581328 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:17.581278    2563 projected.go:196] Error preparing data for projected volume kube-api-access-lxzgc for pod openshift-network-diagnostics/network-check-target-sf462: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:17.581328 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:17.581328    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52ba7983-15e4-437e-a86f-36f0048b0614-kube-api-access-lxzgc podName:52ba7983-15e4-437e-a86f-36f0048b0614 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:18.581315305 +0000 UTC m=+4.250396242 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lxzgc" (UniqueName: "kubernetes.io/projected/52ba7983-15e4-437e-a86f-36f0048b0614-kube-api-access-lxzgc") pod "network-check-target-sf462" (UID: "52ba7983-15e4-437e-a86f-36f0048b0614") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:17.823555 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:17.823501    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 17:59:16 +0000 UTC" deadline="2027-12-23 15:28:49.07811816 +0000 UTC"
Jul 09 18:04:17.823555 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:17.823550    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12765h24m31.254571043s"
Jul 09 18:04:17.876251 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:17.876141    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7fc15e3_7222_4098_8933_cd0b6cd499bd.slice/crio-1968e39b38eb0efae2414e7ff9187ab20f13442e63b782e8bf9a8135994be35d WatchSource:0}: Error finding container 1968e39b38eb0efae2414e7ff9187ab20f13442e63b782e8bf9a8135994be35d: Status 404 returned error can't find the container with id 1968e39b38eb0efae2414e7ff9187ab20f13442e63b782e8bf9a8135994be35d
Jul 09 18:04:17.878261 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:17.878195    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f2265cf_194a_43a6_b5d4_4fe400c2f429.slice/crio-fc71dafbb6d711adafbdf2398ff7fd831f45cdd8782776631418b1be0281e5bc WatchSource:0}: Error finding container fc71dafbb6d711adafbdf2398ff7fd831f45cdd8782776631418b1be0281e5bc: Status 404 returned error can't find the container with id fc71dafbb6d711adafbdf2398ff7fd831f45cdd8782776631418b1be0281e5bc
Jul 09 18:04:17.881320 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:17.881283    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c9006a0_0f34_4c16_84e4_ccf1ec96bb79.slice/crio-866b4e4a11d72bb21a6b4acd9fdebed517b0cb45fbc59eb0ef2d1f0a01c015ea WatchSource:0}: Error finding container 866b4e4a11d72bb21a6b4acd9fdebed517b0cb45fbc59eb0ef2d1f0a01c015ea: Status 404 returned error can't find the container with id 866b4e4a11d72bb21a6b4acd9fdebed517b0cb45fbc59eb0ef2d1f0a01c015ea
Jul 09 18:04:17.882293 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:17.882206    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1360f0da_56fe_485a_93b9_9ee7a46d81a1.slice/crio-9fbd26ec1461f13c58ed773f6d0ca839b8a4df4ff97a4691ac5cc6b55534d62b WatchSource:0}: Error finding container 9fbd26ec1461f13c58ed773f6d0ca839b8a4df4ff97a4691ac5cc6b55534d62b: Status 404 returned error can't find the container with id 9fbd26ec1461f13c58ed773f6d0ca839b8a4df4ff97a4691ac5cc6b55534d62b
Jul 09 18:04:17.883172 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:17.883150    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67e30329_228a_455d_886a_626b46e438d4.slice/crio-e9b6972f6fc27ec971e17ca441955e6d44f7b4ab8af0bdd1787a639d0a202b62 WatchSource:0}: Error finding container e9b6972f6fc27ec971e17ca441955e6d44f7b4ab8af0bdd1787a639d0a202b62: Status 404 returned error can't find the container with id e9b6972f6fc27ec971e17ca441955e6d44f7b4ab8af0bdd1787a639d0a202b62
Jul 09 18:04:17.883994 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:17.883800    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d463fb2_45f9_4554_9525_637e1b84351a.slice/crio-c5ebef44337e3318a4c49e9bc85e435d45782824b8de1a4566b2abd24d3f5145 WatchSource:0}: Error finding container c5ebef44337e3318a4c49e9bc85e435d45782824b8de1a4566b2abd24d3f5145: Status 404 returned error can't find the container with id c5ebef44337e3318a4c49e9bc85e435d45782824b8de1a4566b2abd24d3f5145
Jul 09 18:04:17.885442 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:04:17.885189    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb10ff914_b92d_4b9f_91ec_99c6fcb1425e.slice/crio-ea6afad0ca8120ee1def6a8f95bab2b3fc984853195347f87880a0f56ca23f92 WatchSource:0}: Error finding container ea6afad0ca8120ee1def6a8f95bab2b3fc984853195347f87880a0f56ca23f92: Status 404 returned error can't find the container with id ea6afad0ca8120ee1def6a8f95bab2b3fc984853195347f87880a0f56ca23f92
Jul 09 18:04:17.955067 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:17.954922    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:17.955184 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:17.955143    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ng78" podUID="cddd2f06-2abf-4246-ac09-acc38c065830"
Jul 09 18:04:17.963136 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:17.963108    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7xtpt" event={"ID":"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137","Type":"ContainerStarted","Data":"b989bc1293049432cfb56f118c5fe364dfc064846e3f57638e29c12dfafc5388"}
Jul 09 18:04:17.964608 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:17.964584    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6xthg" event={"ID":"b10ff914-b92d-4b9f-91ec-99c6fcb1425e","Type":"ContainerStarted","Data":"ea6afad0ca8120ee1def6a8f95bab2b3fc984853195347f87880a0f56ca23f92"}
Jul 09 18:04:17.965744 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:17.965722    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbftk" event={"ID":"5d463fb2-45f9-4554-9525-637e1b84351a","Type":"ContainerStarted","Data":"c5ebef44337e3318a4c49e9bc85e435d45782824b8de1a4566b2abd24d3f5145"}
Jul 09 18:04:17.969324 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:17.969301    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9xkxg" event={"ID":"67e30329-228a-455d-886a-626b46e438d4","Type":"ContainerStarted","Data":"e9b6972f6fc27ec971e17ca441955e6d44f7b4ab8af0bdd1787a639d0a202b62"}
Jul 09 18:04:17.971580 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:17.971559    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lljj8" event={"ID":"7f2265cf-194a-43a6-b5d4-4fe400c2f429","Type":"ContainerStarted","Data":"fc71dafbb6d711adafbdf2398ff7fd831f45cdd8782776631418b1be0281e5bc"}
Jul 09 18:04:17.972476 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:17.972452    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rx2rz" event={"ID":"399b7ee4-7292-4fb5-836c-11acc593fa62","Type":"ContainerStarted","Data":"d77046e86e2e6b84bf2582f4baaf35ebde81d5a037fea1f86b63fb816572de55"}
Jul 09 18:04:17.973489 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:17.973470    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wd27m" event={"ID":"1360f0da-56fe-485a-93b9-9ee7a46d81a1","Type":"ContainerStarted","Data":"9fbd26ec1461f13c58ed773f6d0ca839b8a4df4ff97a4691ac5cc6b55534d62b"}
Jul 09 18:04:17.977339 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:17.977318    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r5v4p" event={"ID":"4c9006a0-0f34-4c16-84e4-ccf1ec96bb79","Type":"ContainerStarted","Data":"866b4e4a11d72bb21a6b4acd9fdebed517b0cb45fbc59eb0ef2d1f0a01c015ea"}
Jul 09 18:04:17.981538 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:17.981519    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tvwn2" event={"ID":"f7fc15e3-7222-4098-8933-cd0b6cd499bd","Type":"ContainerStarted","Data":"1968e39b38eb0efae2414e7ff9187ab20f13442e63b782e8bf9a8135994be35d"}
Jul 09 18:04:18.385841 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:18.385753    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cddd2f06-2abf-4246-ac09-acc38c065830-original-pull-secret\") pod \"global-pull-secret-syncer-6ng78\" (UID: \"cddd2f06-2abf-4246-ac09-acc38c065830\") " pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:18.386003 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:18.385937    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:18.386003 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:18.386000    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cddd2f06-2abf-4246-ac09-acc38c065830-original-pull-secret podName:cddd2f06-2abf-4246-ac09-acc38c065830 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:20.385982286 +0000 UTC m=+6.055063235 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cddd2f06-2abf-4246-ac09-acc38c065830-original-pull-secret") pod "global-pull-secret-syncer-6ng78" (UID: "cddd2f06-2abf-4246-ac09-acc38c065830") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:18.486337 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:18.486300    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5de2134-56a4-4318-a22f-90937385dc5b-metrics-certs\") pod \"network-metrics-daemon-wqgvh\" (UID: \"d5de2134-56a4-4318-a22f-90937385dc5b\") " pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:04:18.486512 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:18.486438    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:18.486512 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:18.486500    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5de2134-56a4-4318-a22f-90937385dc5b-metrics-certs podName:d5de2134-56a4-4318-a22f-90937385dc5b nodeName:}" failed. No retries permitted until 2026-07-09 18:04:20.486482873 +0000 UTC m=+6.155563820 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5de2134-56a4-4318-a22f-90937385dc5b-metrics-certs") pod "network-metrics-daemon-wqgvh" (UID: "d5de2134-56a4-4318-a22f-90937385dc5b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:18.586992 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:18.586943    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxzgc\" (UniqueName: \"kubernetes.io/projected/52ba7983-15e4-437e-a86f-36f0048b0614-kube-api-access-lxzgc\") pod \"network-check-target-sf462\" (UID: \"52ba7983-15e4-437e-a86f-36f0048b0614\") " pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:04:18.587180 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:18.587149    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:18.587180 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:18.587173    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:18.587327 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:18.587186    2563 projected.go:196] Error preparing data for projected volume kube-api-access-lxzgc for pod openshift-network-diagnostics/network-check-target-sf462: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:18.587327 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:18.587256    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52ba7983-15e4-437e-a86f-36f0048b0614-kube-api-access-lxzgc podName:52ba7983-15e4-437e-a86f-36f0048b0614 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:20.587238789 +0000 UTC m=+6.256319725 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lxzgc" (UniqueName: "kubernetes.io/projected/52ba7983-15e4-437e-a86f-36f0048b0614-kube-api-access-lxzgc") pod "network-check-target-sf462" (UID: "52ba7983-15e4-437e-a86f-36f0048b0614") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:18.824002 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:18.823947    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 17:59:16 +0000 UTC" deadline="2028-03-05 00:05:17.342961219 +0000 UTC"
Jul 09 18:04:18.824002 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:18.823980    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14502h0m58.518985751s"
Jul 09 18:04:18.956783 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:18.956330    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:04:18.956783 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:18.956444    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sf462" podUID="52ba7983-15e4-437e-a86f-36f0048b0614"
Jul 09 18:04:18.957208 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:18.957075    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:04:18.957208 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:18.957176    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqgvh" podUID="d5de2134-56a4-4318-a22f-90937385dc5b"
Jul 09 18:04:18.993007 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:18.992968    2563 generic.go:356] "Generic (PLEG): container finished" podID="aa66b6c771563ae19089762114b13fa1" containerID="937d70d061c69655a63e10f74a42221fd358cb4e17b6dad0e4cb5c94127dd8b5" exitCode=0
Jul 09 18:04:18.993246 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:18.993205    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-181.ec2.internal" event={"ID":"aa66b6c771563ae19089762114b13fa1","Type":"ContainerDied","Data":"937d70d061c69655a63e10f74a42221fd358cb4e17b6dad0e4cb5c94127dd8b5"}
Jul 09 18:04:19.006431 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:19.006399    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-181.ec2.internal" event={"ID":"72b647e293c446e9af86e2db9bbd3ef8","Type":"ContainerStarted","Data":"cf104b9fd405317c410db3af4610574e544e28beab90a70a09ddb051dada579e"}
Jul 09 18:04:19.954753 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:19.954710    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:19.955247 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:19.954849    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ng78" podUID="cddd2f06-2abf-4246-ac09-acc38c065830"
Jul 09 18:04:20.028571 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:20.028500    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-181.ec2.internal" event={"ID":"aa66b6c771563ae19089762114b13fa1","Type":"ContainerStarted","Data":"b04c175428c38ddcdff4f8c34572e771c35dc71f7457ef09d0f111d8a9cc3870"}
Jul 09 18:04:20.044244 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:20.044171    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-181.ec2.internal" podStartSLOduration=4.044148407 podStartE2EDuration="4.044148407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:19.047820358 +0000 UTC m=+4.716901313" watchObservedRunningTime="2026-07-09 18:04:20.044148407 +0000 UTC m=+5.713229365"
Jul 09 18:04:20.404174 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:20.403775    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cddd2f06-2abf-4246-ac09-acc38c065830-original-pull-secret\") pod \"global-pull-secret-syncer-6ng78\" (UID: \"cddd2f06-2abf-4246-ac09-acc38c065830\") " pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:20.404174 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:20.403945    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:20.404174 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:20.404016    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cddd2f06-2abf-4246-ac09-acc38c065830-original-pull-secret podName:cddd2f06-2abf-4246-ac09-acc38c065830 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:24.403998256 +0000 UTC m=+10.073079194 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cddd2f06-2abf-4246-ac09-acc38c065830-original-pull-secret") pod "global-pull-secret-syncer-6ng78" (UID: "cddd2f06-2abf-4246-ac09-acc38c065830") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:20.504720 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:20.504672    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5de2134-56a4-4318-a22f-90937385dc5b-metrics-certs\") pod \"network-metrics-daemon-wqgvh\" (UID: \"d5de2134-56a4-4318-a22f-90937385dc5b\") " pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:04:20.504877 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:20.504866    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:20.504935 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:20.504923    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5de2134-56a4-4318-a22f-90937385dc5b-metrics-certs podName:d5de2134-56a4-4318-a22f-90937385dc5b nodeName:}" failed. No retries permitted until 2026-07-09 18:04:24.504905105 +0000 UTC m=+10.173986054 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5de2134-56a4-4318-a22f-90937385dc5b-metrics-certs") pod "network-metrics-daemon-wqgvh" (UID: "d5de2134-56a4-4318-a22f-90937385dc5b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:20.605782 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:20.605740    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxzgc\" (UniqueName: \"kubernetes.io/projected/52ba7983-15e4-437e-a86f-36f0048b0614-kube-api-access-lxzgc\") pod \"network-check-target-sf462\" (UID: \"52ba7983-15e4-437e-a86f-36f0048b0614\") " pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:04:20.605992 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:20.605978    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:20.606082 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:20.606001    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:20.606082 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:20.606014    2563 projected.go:196] Error preparing data for projected volume kube-api-access-lxzgc for pod openshift-network-diagnostics/network-check-target-sf462: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:20.606082 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:20.606074    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52ba7983-15e4-437e-a86f-36f0048b0614-kube-api-access-lxzgc podName:52ba7983-15e4-437e-a86f-36f0048b0614 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:24.606055711 +0000 UTC m=+10.275136659 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lxzgc" (UniqueName: "kubernetes.io/projected/52ba7983-15e4-437e-a86f-36f0048b0614-kube-api-access-lxzgc") pod "network-check-target-sf462" (UID: "52ba7983-15e4-437e-a86f-36f0048b0614") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:20.958005 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:20.957492    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:04:20.958005 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:20.957644    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqgvh" podUID="d5de2134-56a4-4318-a22f-90937385dc5b"
Jul 09 18:04:20.958005 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:20.957729    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:04:20.958005 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:20.957829    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sf462" podUID="52ba7983-15e4-437e-a86f-36f0048b0614"
Jul 09 18:04:21.954647 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:21.954616    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:21.954845 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:21.954769    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ng78" podUID="cddd2f06-2abf-4246-ac09-acc38c065830"
Jul 09 18:04:22.954974 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:22.954940    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:04:22.955428 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:22.954947    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:04:22.955428 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:22.955090    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqgvh" podUID="d5de2134-56a4-4318-a22f-90937385dc5b"
Jul 09 18:04:22.955428 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:22.955176    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sf462" podUID="52ba7983-15e4-437e-a86f-36f0048b0614"
Jul 09 18:04:23.954879 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:23.954449    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:23.954879 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:23.954568    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ng78" podUID="cddd2f06-2abf-4246-ac09-acc38c065830"
Jul 09 18:04:24.436732 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:24.436685    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cddd2f06-2abf-4246-ac09-acc38c065830-original-pull-secret\") pod \"global-pull-secret-syncer-6ng78\" (UID: \"cddd2f06-2abf-4246-ac09-acc38c065830\") " pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:24.436935 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:24.436855    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:24.436935 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:24.436919    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cddd2f06-2abf-4246-ac09-acc38c065830-original-pull-secret podName:cddd2f06-2abf-4246-ac09-acc38c065830 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:32.43690138 +0000 UTC m=+18.105982318 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cddd2f06-2abf-4246-ac09-acc38c065830-original-pull-secret") pod "global-pull-secret-syncer-6ng78" (UID: "cddd2f06-2abf-4246-ac09-acc38c065830") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:24.537081 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:24.537039    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5de2134-56a4-4318-a22f-90937385dc5b-metrics-certs\") pod \"network-metrics-daemon-wqgvh\" (UID: \"d5de2134-56a4-4318-a22f-90937385dc5b\") " pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:04:24.537324 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:24.537255    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:24.537324 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:24.537325    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5de2134-56a4-4318-a22f-90937385dc5b-metrics-certs podName:d5de2134-56a4-4318-a22f-90937385dc5b nodeName:}" failed. No retries permitted until 2026-07-09 18:04:32.537306177 +0000 UTC m=+18.206387112 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5de2134-56a4-4318-a22f-90937385dc5b-metrics-certs") pod "network-metrics-daemon-wqgvh" (UID: "d5de2134-56a4-4318-a22f-90937385dc5b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:24.638367 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:24.638325    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxzgc\" (UniqueName: \"kubernetes.io/projected/52ba7983-15e4-437e-a86f-36f0048b0614-kube-api-access-lxzgc\") pod \"network-check-target-sf462\" (UID: \"52ba7983-15e4-437e-a86f-36f0048b0614\") " pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:04:24.638554 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:24.638523    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:24.638554 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:24.638549    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:24.638655 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:24.638563    2563 projected.go:196] Error preparing data for projected volume kube-api-access-lxzgc for pod openshift-network-diagnostics/network-check-target-sf462: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:24.638655 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:24.638630    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52ba7983-15e4-437e-a86f-36f0048b0614-kube-api-access-lxzgc podName:52ba7983-15e4-437e-a86f-36f0048b0614 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:32.638611236 +0000 UTC m=+18.307692174 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lxzgc" (UniqueName: "kubernetes.io/projected/52ba7983-15e4-437e-a86f-36f0048b0614-kube-api-access-lxzgc") pod "network-check-target-sf462" (UID: "52ba7983-15e4-437e-a86f-36f0048b0614") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:24.957551 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:24.956871    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:04:24.957551 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:24.956982    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sf462" podUID="52ba7983-15e4-437e-a86f-36f0048b0614"
Jul 09 18:04:24.957551 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:24.957385    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:04:24.957551 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:24.957478    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqgvh" podUID="d5de2134-56a4-4318-a22f-90937385dc5b"
Jul 09 18:04:25.955159 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:25.955115    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:25.955405 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:25.955277    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ng78" podUID="cddd2f06-2abf-4246-ac09-acc38c065830"
Jul 09 18:04:26.954644 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:26.954609    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:04:26.954644 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:26.954638    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:04:26.955177 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:26.954746    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sf462" podUID="52ba7983-15e4-437e-a86f-36f0048b0614"
Jul 09 18:04:26.955177 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:26.954872    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqgvh" podUID="d5de2134-56a4-4318-a22f-90937385dc5b"
Jul 09 18:04:27.954429 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:27.954395    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:27.954594 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:27.954521    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ng78" podUID="cddd2f06-2abf-4246-ac09-acc38c065830"
Jul 09 18:04:28.954978 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:28.954941    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:04:28.954978 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:28.954972    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:04:28.955498 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:28.955091    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqgvh" podUID="d5de2134-56a4-4318-a22f-90937385dc5b"
Jul 09 18:04:28.955498 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:28.955210    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sf462" podUID="52ba7983-15e4-437e-a86f-36f0048b0614"
Jul 09 18:04:29.955390 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:29.955352    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:29.955828 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:29.955488    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ng78" podUID="cddd2f06-2abf-4246-ac09-acc38c065830"
Jul 09 18:04:30.955178 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:30.955139    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:04:30.955398 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:30.955139    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:04:30.955398 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:30.955303    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sf462" podUID="52ba7983-15e4-437e-a86f-36f0048b0614"
Jul 09 18:04:30.955398 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:30.955362    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqgvh" podUID="d5de2134-56a4-4318-a22f-90937385dc5b"
Jul 09 18:04:31.954706 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:31.954662    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:31.954893 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:31.954784    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ng78" podUID="cddd2f06-2abf-4246-ac09-acc38c065830"
Jul 09 18:04:32.497795 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:32.497762    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cddd2f06-2abf-4246-ac09-acc38c065830-original-pull-secret\") pod \"global-pull-secret-syncer-6ng78\" (UID: \"cddd2f06-2abf-4246-ac09-acc38c065830\") " pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:32.498261 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:32.497970    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:32.498261 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:32.498052    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cddd2f06-2abf-4246-ac09-acc38c065830-original-pull-secret podName:cddd2f06-2abf-4246-ac09-acc38c065830 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:48.49803016 +0000 UTC m=+34.167111114 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cddd2f06-2abf-4246-ac09-acc38c065830-original-pull-secret") pod "global-pull-secret-syncer-6ng78" (UID: "cddd2f06-2abf-4246-ac09-acc38c065830") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:32.598801 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:32.598758    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5de2134-56a4-4318-a22f-90937385dc5b-metrics-certs\") pod \"network-metrics-daemon-wqgvh\" (UID: \"d5de2134-56a4-4318-a22f-90937385dc5b\") " pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:04:32.598983 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:32.598963    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:32.599055 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:32.599043    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5de2134-56a4-4318-a22f-90937385dc5b-metrics-certs podName:d5de2134-56a4-4318-a22f-90937385dc5b nodeName:}" failed. No retries permitted until 2026-07-09 18:04:48.599022416 +0000 UTC m=+34.268103349 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5de2134-56a4-4318-a22f-90937385dc5b-metrics-certs") pod "network-metrics-daemon-wqgvh" (UID: "d5de2134-56a4-4318-a22f-90937385dc5b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:32.699237 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:32.699141    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxzgc\" (UniqueName: \"kubernetes.io/projected/52ba7983-15e4-437e-a86f-36f0048b0614-kube-api-access-lxzgc\") pod \"network-check-target-sf462\" (UID: \"52ba7983-15e4-437e-a86f-36f0048b0614\") " pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:04:32.699383 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:32.699339    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:32.699383 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:32.699366    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:32.699383 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:32.699380    2563 projected.go:196] Error preparing data for projected volume kube-api-access-lxzgc for pod openshift-network-diagnostics/network-check-target-sf462: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:32.699519 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:32.699446    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52ba7983-15e4-437e-a86f-36f0048b0614-kube-api-access-lxzgc podName:52ba7983-15e4-437e-a86f-36f0048b0614 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:48.699427462 +0000 UTC m=+34.368508395 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lxzgc" (UniqueName: "kubernetes.io/projected/52ba7983-15e4-437e-a86f-36f0048b0614-kube-api-access-lxzgc") pod "network-check-target-sf462" (UID: "52ba7983-15e4-437e-a86f-36f0048b0614") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:32.955525 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:32.955438    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:04:32.955525 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:32.955481    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:04:32.955747 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:32.955572    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqgvh" podUID="d5de2134-56a4-4318-a22f-90937385dc5b"
Jul 09 18:04:32.955747 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:32.955711    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sf462" podUID="52ba7983-15e4-437e-a86f-36f0048b0614"
Jul 09 18:04:33.955089 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:33.955061    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:33.955535 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:33.955178    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ng78" podUID="cddd2f06-2abf-4246-ac09-acc38c065830"
Jul 09 18:04:34.955786 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:34.955752    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:04:34.956210 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:34.955868    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqgvh" podUID="d5de2134-56a4-4318-a22f-90937385dc5b"
Jul 09 18:04:34.956210 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:34.955918    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:04:34.956210 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:34.955998    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sf462" podUID="52ba7983-15e4-437e-a86f-36f0048b0614"
Jul 09 18:04:35.954835 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:35.954792    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:35.955001 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:35.954918    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ng78" podUID="cddd2f06-2abf-4246-ac09-acc38c065830"
Jul 09 18:04:36.954876 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:36.954550    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:04:36.955572 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:36.954602    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:04:36.955572 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:36.954982    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqgvh" podUID="d5de2134-56a4-4318-a22f-90937385dc5b"
Jul 09 18:04:36.955572 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:36.955039    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sf462" podUID="52ba7983-15e4-437e-a86f-36f0048b0614"
Jul 09 18:04:37.064814 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:37.064691    2563 generic.go:356] "Generic (PLEG): container finished" podID="7f2265cf-194a-43a6-b5d4-4fe400c2f429" containerID="8506bd826b324ea092340b667253bb3c0a329406c6920c6ea722a32e83089cbc" exitCode=0
Jul 09 18:04:37.064814 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:37.064782    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lljj8" event={"ID":"7f2265cf-194a-43a6-b5d4-4fe400c2f429","Type":"ContainerDied","Data":"8506bd826b324ea092340b667253bb3c0a329406c6920c6ea722a32e83089cbc"}
Jul 09 18:04:37.066813 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:37.066378    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rx2rz" event={"ID":"399b7ee4-7292-4fb5-836c-11acc593fa62","Type":"ContainerStarted","Data":"2f9b155a90173c97a4e2b6c85bd5440ffcafd60c3a6e09e9654b329048efcb1b"}
Jul 09 18:04:37.071527 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:37.071500    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wd27m" event={"ID":"1360f0da-56fe-485a-93b9-9ee7a46d81a1","Type":"ContainerStarted","Data":"af72526fb203f892c30b31c9d27c19a32159d6e9725815858f2f60df614ea620"}
Jul 09 18:04:37.073358 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:37.073308    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r5v4p" event={"ID":"4c9006a0-0f34-4c16-84e4-ccf1ec96bb79","Type":"ContainerStarted","Data":"1b89767765d10c72253230ff75445c55639a58a96b5b793fa48ed324d2cc3a51"}
Jul 09 18:04:37.074839 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:37.074811    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tvwn2" event={"ID":"f7fc15e3-7222-4098-8933-cd0b6cd499bd","Type":"ContainerStarted","Data":"91465aaa6af167476936e45a3419a96f2cbc9dfdbe66a06eae2b44c5b2836699"}
Jul 09 18:04:37.076317 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:37.076296    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7xtpt" event={"ID":"30ba50d7-581d-4b1b-9e63-7e1fe0ea9137","Type":"ContainerStarted","Data":"cfe76ef895c1d6098c533dfc19a30d39f7a766175525f5057f0a6ac54a308467"}
Jul 09 18:04:37.079309 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:37.079287    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbftk_5d463fb2-45f9-4554-9525-637e1b84351a/ovn-acl-logging/0.log"
Jul 09 18:04:37.079719 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:37.079696    2563 generic.go:356] "Generic (PLEG): container finished" podID="5d463fb2-45f9-4554-9525-637e1b84351a" containerID="5699e3a63e5e78a320fb25bf66616ca2c980109deeb61995a9c7fd3dc616970b" exitCode=1
Jul 09 18:04:37.079797 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:37.079774    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbftk" event={"ID":"5d463fb2-45f9-4554-9525-637e1b84351a","Type":"ContainerStarted","Data":"25feb267adf9d10f52d2f35208c0387347f46fdcdda3b4168d436d286ac98dce"}
Jul 09 18:04:37.079856 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:37.079847    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbftk" event={"ID":"5d463fb2-45f9-4554-9525-637e1b84351a","Type":"ContainerStarted","Data":"e634bb3dc8e998f2514a74b01b2ec8a4b65fef5e2d701eb120006b4b50ae3810"}
Jul 09 18:04:37.079917 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:37.079898    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbftk" event={"ID":"5d463fb2-45f9-4554-9525-637e1b84351a","Type":"ContainerStarted","Data":"df4996e1989340a0b371d2cfebb8f833d61e14a8cf4d12cca9fb675687dfa242"}
Jul 09 18:04:37.079973 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:37.079939    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbftk" event={"ID":"5d463fb2-45f9-4554-9525-637e1b84351a","Type":"ContainerStarted","Data":"d928b0e8cadd8d4fd95fae70a83bcd3666d88642228e808b2198742932a3a236"}
Jul 09 18:04:37.080014 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:37.079977    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbftk" event={"ID":"5d463fb2-45f9-4554-9525-637e1b84351a","Type":"ContainerDied","Data":"5699e3a63e5e78a320fb25bf66616ca2c980109deeb61995a9c7fd3dc616970b"}
Jul 09 18:04:37.080014 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:37.080009    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbftk" event={"ID":"5d463fb2-45f9-4554-9525-637e1b84351a","Type":"ContainerStarted","Data":"1627f2049599c594f91e6b493e0767e2f004443d04c85a2e8e94ac84bbd05de6"}
Jul 09 18:04:37.081360 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:37.081341    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9xkxg" event={"ID":"67e30329-228a-455d-886a-626b46e438d4","Type":"ContainerStarted","Data":"4429d04f883fc25b6e058dabd54e71edf4c7dde42823a46cba46f5d5e8b52554"}
Jul 09 18:04:37.094553 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:37.094499    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-181.ec2.internal" podStartSLOduration=21.094483107 podStartE2EDuration="21.094483107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:20.044716334 +0000 UTC m=+5.713797291" watchObservedRunningTime="2026-07-09 18:04:37.094483107 +0000 UTC m=+22.763564062"
Jul 09 18:04:37.159384 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:37.159333    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wd27m" podStartSLOduration=3.759125566 podStartE2EDuration="22.159317729s" totalImagesPullingTime="18.400192163s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:37.126644844 +0000 UTC m=+22.795725800" watchObservedRunningTime="2026-07-09 18:04:37.159317729 +0000 UTC m=+22.828398683"
Jul 09 18:04:37.159564 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:37.159446    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7xtpt" podStartSLOduration=4.705838387 podStartE2EDuration="23.159439419s" totalImagesPullingTime="18.453601032s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:37.154024944 +0000 UTC m=+22.823105899" watchObservedRunningTime="2026-07-09 18:04:37.159439419 +0000 UTC m=+22.828520374"
Jul 09 18:04:37.205532 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:37.205482    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tvwn2" podStartSLOduration=7.641759165 podStartE2EDuration="22.205461242s" totalImagesPullingTime="14.563702077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:37.17471514 +0000 UTC m=+22.843796119" watchObservedRunningTime="2026-07-09 18:04:37.205461242 +0000 UTC m=+22.874542198"
Jul 09 18:04:37.205728 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:37.205664    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9xkxg" podStartSLOduration=4.82735395 podStartE2EDuration="23.205659411s" totalImagesPullingTime="18.378305461s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:37.205170252 +0000 UTC m=+22.874251206" watchObservedRunningTime="2026-07-09 18:04:37.205659411 +0000 UTC m=+22.874740413"
Jul 09 18:04:37.827647 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:37.827624    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 09 18:04:37.829702 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:37.829620    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T18:04:37.827641239Z","UUID":"1e0c2a91-d442-4e7a-89d9-23fadc744625","Handler":null,"Name":"","Endpoint":""}
Jul 09 18:04:37.830995 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:37.830980    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 09 18:04:37.831087 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:37.831010    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 09 18:04:37.954931 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:37.954899    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:37.955407 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:37.955035    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ng78" podUID="cddd2f06-2abf-4246-ac09-acc38c065830"
Jul 09 18:04:38.087185 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:38.087147    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r5v4p" event={"ID":"4c9006a0-0f34-4c16-84e4-ccf1ec96bb79","Type":"ContainerStarted","Data":"bddcccbce01ba3c0d866a0c043a56140b28913ac0d81112825f90f1a9ff4297d"}
Jul 09 18:04:38.089233 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:38.089185    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6xthg" event={"ID":"b10ff914-b92d-4b9f-91ec-99c6fcb1425e","Type":"ContainerStarted","Data":"88afe36f816aa498e109cca12788e8b1d0de244a4074ea7d00d483815b30da8d"}
Jul 09 18:04:38.122144 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:38.122082    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rx2rz" podStartSLOduration=4.7464675100000004 podStartE2EDuration="23.122062411s" totalImagesPullingTime="18.375594901s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:37.225351573 +0000 UTC m=+22.894432527" watchObservedRunningTime="2026-07-09 18:04:38.122062411 +0000 UTC m=+23.791143361"
Jul 09 18:04:38.122355 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:38.122325    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6xthg" podStartSLOduration=4.748422483 podStartE2EDuration="23.122318178s" totalImagesPullingTime="18.373895695s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:38.122150094 +0000 UTC m=+23.791231050" watchObservedRunningTime="2026-07-09 18:04:38.122318178 +0000 UTC m=+23.791399133"
Jul 09 18:04:38.777883 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:38.777847    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tvwn2"
Jul 09 18:04:38.778435 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:38.778413    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tvwn2"
Jul 09 18:04:38.955378 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:38.955336    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:04:38.955812 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:38.955475    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqgvh" podUID="d5de2134-56a4-4318-a22f-90937385dc5b"
Jul 09 18:04:38.955812 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:38.955492    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:04:38.955812 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:38.955599    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sf462" podUID="52ba7983-15e4-437e-a86f-36f0048b0614"
Jul 09 18:04:39.093865 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:39.093771    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r5v4p" event={"ID":"4c9006a0-0f34-4c16-84e4-ccf1ec96bb79","Type":"ContainerStarted","Data":"c50a5339ba609fbc4b8acb1f76e63626be945d8971b7904bdebd1d59b475ea2d"}
Jul 09 18:04:39.094154 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:39.094137    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tvwn2"
Jul 09 18:04:39.094631 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:39.094614    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tvwn2"
Jul 09 18:04:39.124504 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:39.124347    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r5v4p" podStartSLOduration=3.467910346 podStartE2EDuration="24.124331775s" totalImagesPullingTime="20.656421429s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:39.124104645 +0000 UTC m=+24.793185602" watchObservedRunningTime="2026-07-09 18:04:39.124331775 +0000 UTC m=+24.793412732"
Jul 09 18:04:39.187808 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:39.187778    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rx2rz_399b7ee4-7292-4fb5-836c-11acc593fa62/dns-node-resolver/0.log"
Jul 09 18:04:39.954418 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:39.954390    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:39.954716 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:39.954510    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ng78" podUID="cddd2f06-2abf-4246-ac09-acc38c065830"
Jul 09 18:04:39.954716 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:39.954655    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9xkxg_67e30329-228a-455d-886a-626b46e438d4/node-ca/0.log"
Jul 09 18:04:40.099583 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:40.099360    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbftk_5d463fb2-45f9-4554-9525-637e1b84351a/ovn-acl-logging/0.log"
Jul 09 18:04:40.100028 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:40.099956    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbftk" event={"ID":"5d463fb2-45f9-4554-9525-637e1b84351a","Type":"ContainerStarted","Data":"abbf735d9efd71b0b8f510c611f8e9335ed26783908e33fd144a057fd25e5de6"}
Jul 09 18:04:40.954596 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:40.954558    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:04:40.954798 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:40.954709    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqgvh" podUID="d5de2134-56a4-4318-a22f-90937385dc5b"
Jul 09 18:04:40.954798 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:40.954759    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:04:40.954890 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:40.954866    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sf462" podUID="52ba7983-15e4-437e-a86f-36f0048b0614"
Jul 09 18:04:41.955159 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:41.954997    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:41.955816 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:41.955258    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ng78" podUID="cddd2f06-2abf-4246-ac09-acc38c065830"
Jul 09 18:04:42.107253 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:42.107229    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbftk_5d463fb2-45f9-4554-9525-637e1b84351a/ovn-acl-logging/0.log"
Jul 09 18:04:42.107593 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:42.107565    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbftk" event={"ID":"5d463fb2-45f9-4554-9525-637e1b84351a","Type":"ContainerStarted","Data":"898e5bb4eb0a12dd164c618b90979b0d081aa19853ca15fc5d9ed8dba33537a9"}
Jul 09 18:04:42.107848 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:42.107822    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:42.108122 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:42.108102    2563 scope.go:117] "RemoveContainer" containerID="5699e3a63e5e78a320fb25bf66616ca2c980109deeb61995a9c7fd3dc616970b"
Jul 09 18:04:42.109340 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:42.109305    2563 generic.go:356] "Generic (PLEG): container finished" podID="7f2265cf-194a-43a6-b5d4-4fe400c2f429" containerID="6859256ce7961ca292703fadd7735f9ed3ce0e52efae489dd91d861de3e81a54" exitCode=0
Jul 09 18:04:42.109432 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:42.109333    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lljj8" event={"ID":"7f2265cf-194a-43a6-b5d4-4fe400c2f429","Type":"ContainerDied","Data":"6859256ce7961ca292703fadd7735f9ed3ce0e52efae489dd91d861de3e81a54"}
Jul 09 18:04:42.124046 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:42.124021    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:42.954487 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:42.954447    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:04:42.954694 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:42.954574    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sf462" podUID="52ba7983-15e4-437e-a86f-36f0048b0614"
Jul 09 18:04:42.954694 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:42.954447    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:04:42.954800 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:42.954726    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqgvh" podUID="d5de2134-56a4-4318-a22f-90937385dc5b"
Jul 09 18:04:43.115854 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:43.115827    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbftk_5d463fb2-45f9-4554-9525-637e1b84351a/ovn-acl-logging/0.log"
Jul 09 18:04:43.116385 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:43.116306    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbftk" event={"ID":"5d463fb2-45f9-4554-9525-637e1b84351a","Type":"ContainerStarted","Data":"c4601f7cb76204a7736a75e1190f2a771ac9a253ef2cb4afec835ee42c6c8927"}
Jul 09 18:04:43.116593 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:43.116566    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:43.116711 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:43.116636    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:43.133743 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:43.133717    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:04:43.320307 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:43.320197    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lbftk" podStartSLOduration=9.870005571 podStartE2EDuration="28.320180992s" totalImagesPullingTime="18.450175421s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:43.318846682 +0000 UTC m=+28.987927636" watchObservedRunningTime="2026-07-09 18:04:43.320180992 +0000 UTC m=+28.989261947"
Jul 09 18:04:43.954813 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:43.954779    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:43.955006 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:43.954909    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ng78" podUID="cddd2f06-2abf-4246-ac09-acc38c065830"
Jul 09 18:04:44.121104 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:44.121073    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wqgvh"]
Jul 09 18:04:44.121647 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:44.121204    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:04:44.121647 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:44.121276    2563 generic.go:356] "Generic (PLEG): container finished" podID="7f2265cf-194a-43a6-b5d4-4fe400c2f429" containerID="cd83ecae6df4d230f8c62e69617ece4a99dfa5d749cd12ae79c4fe75e19e51ab" exitCode=0
Jul 09 18:04:44.121647 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:44.121320    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqgvh" podUID="d5de2134-56a4-4318-a22f-90937385dc5b"
Jul 09 18:04:44.121647 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:44.121370    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lljj8" event={"ID":"7f2265cf-194a-43a6-b5d4-4fe400c2f429","Type":"ContainerDied","Data":"cd83ecae6df4d230f8c62e69617ece4a99dfa5d749cd12ae79c4fe75e19e51ab"}
Jul 09 18:04:44.123161 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:44.123145    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6ng78"]
Jul 09 18:04:44.123261 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:44.123238    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:44.123346 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:44.123328    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ng78" podUID="cddd2f06-2abf-4246-ac09-acc38c065830"
Jul 09 18:04:44.124039 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:44.124021    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sf462"]
Jul 09 18:04:44.124118 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:44.124105    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:04:44.124224 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:44.124188    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sf462" podUID="52ba7983-15e4-437e-a86f-36f0048b0614"
Jul 09 18:04:45.955533 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:45.955360    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:45.956034 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:45.955360    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:04:45.956034 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:45.955599    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ng78" podUID="cddd2f06-2abf-4246-ac09-acc38c065830"
Jul 09 18:04:45.956034 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:45.955713    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqgvh" podUID="d5de2134-56a4-4318-a22f-90937385dc5b"
Jul 09 18:04:45.956034 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:45.955360    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:04:45.956034 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:45.955803    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sf462" podUID="52ba7983-15e4-437e-a86f-36f0048b0614"
Jul 09 18:04:46.127989 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:46.127959    2563 generic.go:356] "Generic (PLEG): container finished" podID="7f2265cf-194a-43a6-b5d4-4fe400c2f429" containerID="802d944698325bb7245734ba52e431d430f08e73d17c91be3814ba96b8bc7691" exitCode=0
Jul 09 18:04:46.128139 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:46.128004    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lljj8" event={"ID":"7f2265cf-194a-43a6-b5d4-4fe400c2f429","Type":"ContainerDied","Data":"802d944698325bb7245734ba52e431d430f08e73d17c91be3814ba96b8bc7691"}
Jul 09 18:04:47.954726 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:47.954695    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:04:47.954726 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:47.954714    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:47.955252 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:47.954828    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sf462" podUID="52ba7983-15e4-437e-a86f-36f0048b0614"
Jul 09 18:04:47.955252 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:47.954911    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ng78" podUID="cddd2f06-2abf-4246-ac09-acc38c065830"
Jul 09 18:04:47.955252 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:47.954935    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:04:47.955252 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:47.955038    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqgvh" podUID="d5de2134-56a4-4318-a22f-90937385dc5b"
Jul 09 18:04:48.527700 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:48.527666    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cddd2f06-2abf-4246-ac09-acc38c065830-original-pull-secret\") pod \"global-pull-secret-syncer-6ng78\" (UID: \"cddd2f06-2abf-4246-ac09-acc38c065830\") " pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:48.527876 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:48.527805    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:48.527876 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:48.527872    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cddd2f06-2abf-4246-ac09-acc38c065830-original-pull-secret podName:cddd2f06-2abf-4246-ac09-acc38c065830 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:20.527853439 +0000 UTC m=+66.196934388 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cddd2f06-2abf-4246-ac09-acc38c065830-original-pull-secret") pod "global-pull-secret-syncer-6ng78" (UID: "cddd2f06-2abf-4246-ac09-acc38c065830") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:48.628959 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:48.628919    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5de2134-56a4-4318-a22f-90937385dc5b-metrics-certs\") pod \"network-metrics-daemon-wqgvh\" (UID: \"d5de2134-56a4-4318-a22f-90937385dc5b\") " pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:04:48.629139 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:48.629085    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:48.629203 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:48.629180    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5de2134-56a4-4318-a22f-90937385dc5b-metrics-certs podName:d5de2134-56a4-4318-a22f-90937385dc5b nodeName:}" failed. No retries permitted until 2026-07-09 18:05:20.629159877 +0000 UTC m=+66.298240823 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5de2134-56a4-4318-a22f-90937385dc5b-metrics-certs") pod "network-metrics-daemon-wqgvh" (UID: "d5de2134-56a4-4318-a22f-90937385dc5b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:48.729668 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:48.729629    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxzgc\" (UniqueName: \"kubernetes.io/projected/52ba7983-15e4-437e-a86f-36f0048b0614-kube-api-access-lxzgc\") pod \"network-check-target-sf462\" (UID: \"52ba7983-15e4-437e-a86f-36f0048b0614\") " pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:04:48.729846 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:48.729793    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:48.729846 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:48.729817    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:48.729846 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:48.729830    2563 projected.go:196] Error preparing data for projected volume kube-api-access-lxzgc for pod openshift-network-diagnostics/network-check-target-sf462: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:48.729996 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:48.729899    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52ba7983-15e4-437e-a86f-36f0048b0614-kube-api-access-lxzgc podName:52ba7983-15e4-437e-a86f-36f0048b0614 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:20.729879898 +0000 UTC m=+66.398960851 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lxzgc" (UniqueName: "kubernetes.io/projected/52ba7983-15e4-437e-a86f-36f0048b0614-kube-api-access-lxzgc") pod "network-check-target-sf462" (UID: "52ba7983-15e4-437e-a86f-36f0048b0614") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:49.954468 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:49.954433    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:49.954468 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:49.954449    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:04:49.955104 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:49.954431    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:04:49.955104 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:49.954589    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqgvh" podUID="d5de2134-56a4-4318-a22f-90937385dc5b"
Jul 09 18:04:49.955104 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:49.954630    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sf462" podUID="52ba7983-15e4-437e-a86f-36f0048b0614"
Jul 09 18:04:49.955104 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:49.954727    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ng78" podUID="cddd2f06-2abf-4246-ac09-acc38c065830"
Jul 09 18:04:51.142327 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:51.142296    2563 generic.go:356] "Generic (PLEG): container finished" podID="7f2265cf-194a-43a6-b5d4-4fe400c2f429" containerID="65547dc53a3ce548d73e2fd7a2b9b46591540ad8a9cc6e13bf66772d093ef4de" exitCode=0
Jul 09 18:04:51.142881 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:51.142361    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lljj8" event={"ID":"7f2265cf-194a-43a6-b5d4-4fe400c2f429","Type":"ContainerDied","Data":"65547dc53a3ce548d73e2fd7a2b9b46591540ad8a9cc6e13bf66772d093ef4de"}
Jul 09 18:04:51.954868 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:51.954838    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:04:51.954868 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:51.954861    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:51.955063 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:51.954940    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:04:51.955063 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:51.954938    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqgvh" podUID="d5de2134-56a4-4318-a22f-90937385dc5b"
Jul 09 18:04:51.955063 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:51.955015    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ng78" podUID="cddd2f06-2abf-4246-ac09-acc38c065830"
Jul 09 18:04:51.955160 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:51.955091    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sf462" podUID="52ba7983-15e4-437e-a86f-36f0048b0614"
Jul 09 18:04:52.147869 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:52.147836    2563 generic.go:356] "Generic (PLEG): container finished" podID="7f2265cf-194a-43a6-b5d4-4fe400c2f429" containerID="1bcf426d3b172cff9590515d7ba4dbe115a952abb8c0c05e1907d0e9d0a4246c" exitCode=0
Jul 09 18:04:52.148245 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:52.147893    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lljj8" event={"ID":"7f2265cf-194a-43a6-b5d4-4fe400c2f429","Type":"ContainerDied","Data":"1bcf426d3b172cff9590515d7ba4dbe115a952abb8c0c05e1907d0e9d0a4246c"}
Jul 09 18:04:53.152982 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:53.152945    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lljj8" event={"ID":"7f2265cf-194a-43a6-b5d4-4fe400c2f429","Type":"ContainerStarted","Data":"7ca7a66f87f3a6fdac2c57238dff928b03476fa7e1032f86ce25128c4b9a686f"}
Jul 09 18:04:53.183546 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:53.183489    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lljj8" podStartSLOduration=8.353777182 podStartE2EDuration="38.183471156s" totalImagesPullingTime="29.829693974s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:53.178544479 +0000 UTC m=+38.847625434" watchObservedRunningTime="2026-07-09 18:04:53.183471156 +0000 UTC m=+38.852552114"
Jul 09 18:04:53.954589 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:53.954561    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:04:53.954748 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:53.954561    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:04:53.954748 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:53.954685    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqgvh" podUID="d5de2134-56a4-4318-a22f-90937385dc5b"
Jul 09 18:04:53.954748 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:53.954570    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:53.954871 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:53.954769    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sf462" podUID="52ba7983-15e4-437e-a86f-36f0048b0614"
Jul 09 18:04:53.954871 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:53.954809    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ng78" podUID="cddd2f06-2abf-4246-ac09-acc38c065830"
Jul 09 18:04:55.955172 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:55.955128    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:55.955172 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:55.955172    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:04:55.955657 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:55.955172    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:04:55.955657 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:55.955303    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ng78" podUID="cddd2f06-2abf-4246-ac09-acc38c065830"
Jul 09 18:04:55.955657 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:55.955358    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sf462" podUID="52ba7983-15e4-437e-a86f-36f0048b0614"
Jul 09 18:04:55.955657 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:55.955462    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqgvh" podUID="d5de2134-56a4-4318-a22f-90937385dc5b"
Jul 09 18:04:57.954496 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:57.954453    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:57.954868 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:57.954502    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:04:57.954868 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:57.954506    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:04:57.954868 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:57.954593    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ng78" podUID="cddd2f06-2abf-4246-ac09-acc38c065830"
Jul 09 18:04:57.954868 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:57.954654    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sf462" podUID="52ba7983-15e4-437e-a86f-36f0048b0614"
Jul 09 18:04:57.954868 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:57.954761    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqgvh" podUID="d5de2134-56a4-4318-a22f-90937385dc5b"
Jul 09 18:04:59.954944 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:59.954908    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:04:59.955508 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:59.954908    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:04:59.955508 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:59.955041    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqgvh" podUID="d5de2134-56a4-4318-a22f-90937385dc5b"
Jul 09 18:04:59.955508 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:04:59.954908    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:04:59.955508 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:59.955089    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sf462" podUID="52ba7983-15e4-437e-a86f-36f0048b0614"
Jul 09 18:04:59.955508 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:04:59.955160    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ng78" podUID="cddd2f06-2abf-4246-ac09-acc38c065830"
Jul 09 18:05:01.954390 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:01.954355    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:05:01.954866 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:01.954356    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:05:01.954866 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:05:01.954488    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqgvh" podUID="d5de2134-56a4-4318-a22f-90937385dc5b"
Jul 09 18:05:01.954866 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:01.954362    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:05:01.954866 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:05:01.954538    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sf462" podUID="52ba7983-15e4-437e-a86f-36f0048b0614"
Jul 09 18:05:01.954866 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:05:01.954612    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ng78" podUID="cddd2f06-2abf-4246-ac09-acc38c065830"
Jul 09 18:05:03.955297 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:03.955263    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:05:03.955678 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:03.955266    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:05:03.955678 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:05:03.955371    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqgvh" podUID="d5de2134-56a4-4318-a22f-90937385dc5b"
Jul 09 18:05:03.955678 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:03.955266    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:05:03.955678 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:05:03.955441    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sf462" podUID="52ba7983-15e4-437e-a86f-36f0048b0614"
Jul 09 18:05:03.955678 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:05:03.955512    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ng78" podUID="cddd2f06-2abf-4246-ac09-acc38c065830"
Jul 09 18:05:05.955152 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:05.955114    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:05:05.955614 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:05.955206    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:05:05.955614 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:05.955245    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:05:05.955614 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:05:05.955210    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sf462" podUID="52ba7983-15e4-437e-a86f-36f0048b0614"
Jul 09 18:05:05.955614 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:05:05.955323    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wqgvh" podUID="d5de2134-56a4-4318-a22f-90937385dc5b"
Jul 09 18:05:05.955614 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:05:05.955402    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ng78" podUID="cddd2f06-2abf-4246-ac09-acc38c065830"
Jul 09 18:05:07.161977 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.161883    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-181.ec2.internal" event="NodeReady"
Jul 09 18:05:07.162442 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.162044    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 09 18:05:07.332490 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.332458    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pnvtm"]
Jul 09 18:05:07.356410 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.356374    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-q6mvh"]
Jul 09 18:05:07.356575 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.356418    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pnvtm"
Jul 09 18:05:07.359778 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.359756    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h78nz\""
Jul 09 18:05:07.359903 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.359805    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 09 18:05:07.362587 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.362569    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 09 18:05:07.365756 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.365733    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/996150c6-fdce-44a0-963e-d607d34fdc1a-metrics-tls\") pod \"dns-default-pnvtm\" (UID: \"996150c6-fdce-44a0-963e-d607d34fdc1a\") " pod="openshift-dns/dns-default-pnvtm"
Jul 09 18:05:07.365850 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.365761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/996150c6-fdce-44a0-963e-d607d34fdc1a-tmp-dir\") pod \"dns-default-pnvtm\" (UID: \"996150c6-fdce-44a0-963e-d607d34fdc1a\") " pod="openshift-dns/dns-default-pnvtm"
Jul 09 18:05:07.365850 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.365837    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr226\" (UniqueName: \"kubernetes.io/projected/996150c6-fdce-44a0-963e-d607d34fdc1a-kube-api-access-vr226\") pod \"dns-default-pnvtm\" (UID: \"996150c6-fdce-44a0-963e-d607d34fdc1a\") " pod="openshift-dns/dns-default-pnvtm"
Jul 09 18:05:07.365918 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.365886    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/996150c6-fdce-44a0-963e-d607d34fdc1a-config-volume\") pod \"dns-default-pnvtm\" (UID: \"996150c6-fdce-44a0-963e-d607d34fdc1a\") " pod="openshift-dns/dns-default-pnvtm"
Jul 09 18:05:07.376395 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.376365    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pnvtm"]
Jul 09 18:05:07.376502 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.376431    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q6mvh"]
Jul 09 18:05:07.376546 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.376521    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q6mvh"
Jul 09 18:05:07.379005 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.378984    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 09 18:05:07.379115 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.379022    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sqrc5\""
Jul 09 18:05:07.379235 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.379209    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 09 18:05:07.379621 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.379601    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 09 18:05:07.379706 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.379601    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 09 18:05:07.440269 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.440237    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5nkqz"]
Jul 09 18:05:07.466341 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.466316    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5nkqz"
Jul 09 18:05:07.466474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.466380    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d8df3e9b-66dd-45af-9829-41d518142d58-crio-socket\") pod \"insights-runtime-extractor-q6mvh\" (UID: \"d8df3e9b-66dd-45af-9829-41d518142d58\") " pod="openshift-insights/insights-runtime-extractor-q6mvh"
Jul 09 18:05:07.466474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.466407    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/996150c6-fdce-44a0-963e-d607d34fdc1a-config-volume\") pod \"dns-default-pnvtm\" (UID: \"996150c6-fdce-44a0-963e-d607d34fdc1a\") " pod="openshift-dns/dns-default-pnvtm"
Jul 09 18:05:07.466474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.466427    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95525\" (UniqueName: \"kubernetes.io/projected/d8df3e9b-66dd-45af-9829-41d518142d58-kube-api-access-95525\") pod \"insights-runtime-extractor-q6mvh\" (UID: \"d8df3e9b-66dd-45af-9829-41d518142d58\") " pod="openshift-insights/insights-runtime-extractor-q6mvh"
Jul 09 18:05:07.466474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.466445    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8df3e9b-66dd-45af-9829-41d518142d58-data-volume\") pod \"insights-runtime-extractor-q6mvh\" (UID: \"d8df3e9b-66dd-45af-9829-41d518142d58\") " pod="openshift-insights/insights-runtime-extractor-q6mvh"
Jul 09 18:05:07.466474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.466467    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/996150c6-fdce-44a0-963e-d607d34fdc1a-metrics-tls\") pod \"dns-default-pnvtm\" (UID: \"996150c6-fdce-44a0-963e-d607d34fdc1a\") " pod="openshift-dns/dns-default-pnvtm"
Jul 09 18:05:07.466744 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.466484    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/996150c6-fdce-44a0-963e-d607d34fdc1a-tmp-dir\") pod \"dns-default-pnvtm\" (UID: \"996150c6-fdce-44a0-963e-d607d34fdc1a\") " pod="openshift-dns/dns-default-pnvtm"
Jul 09 18:05:07.466744 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.466544    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8df3e9b-66dd-45af-9829-41d518142d58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6mvh\" (UID: \"d8df3e9b-66dd-45af-9829-41d518142d58\") " pod="openshift-insights/insights-runtime-extractor-q6mvh"
Jul 09 18:05:07.466744 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.466576    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr226\" (UniqueName: \"kubernetes.io/projected/996150c6-fdce-44a0-963e-d607d34fdc1a-kube-api-access-vr226\") pod \"dns-default-pnvtm\" (UID: \"996150c6-fdce-44a0-963e-d607d34fdc1a\") " pod="openshift-dns/dns-default-pnvtm"
Jul 09 18:05:07.466744 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.466602    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8df3e9b-66dd-45af-9829-41d518142d58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q6mvh\" (UID: \"d8df3e9b-66dd-45af-9829-41d518142d58\") " pod="openshift-insights/insights-runtime-extractor-q6mvh"
Jul 09 18:05:07.466912 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.466878    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/996150c6-fdce-44a0-963e-d607d34fdc1a-tmp-dir\") pod \"dns-default-pnvtm\" (UID: \"996150c6-fdce-44a0-963e-d607d34fdc1a\") " pod="openshift-dns/dns-default-pnvtm"
Jul 09 18:05:07.467089 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.467072    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/996150c6-fdce-44a0-963e-d607d34fdc1a-config-volume\") pod \"dns-default-pnvtm\" (UID: \"996150c6-fdce-44a0-963e-d607d34fdc1a\") " pod="openshift-dns/dns-default-pnvtm"
Jul 09 18:05:07.470917 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.470879    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/996150c6-fdce-44a0-963e-d607d34fdc1a-metrics-tls\") pod \"dns-default-pnvtm\" (UID: \"996150c6-fdce-44a0-963e-d607d34fdc1a\") " pod="openshift-dns/dns-default-pnvtm"
Jul 09 18:05:07.471061 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.470992    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 09 18:05:07.471245 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.471204    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gzs4d\""
Jul 09 18:05:07.471290 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.471243    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 09 18:05:07.472375 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.472352    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 09 18:05:07.473101 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.473084    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5nkqz"]
Jul 09 18:05:07.490005 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.489981    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr226\" (UniqueName: \"kubernetes.io/projected/996150c6-fdce-44a0-963e-d607d34fdc1a-kube-api-access-vr226\") pod \"dns-default-pnvtm\" (UID: \"996150c6-fdce-44a0-963e-d607d34fdc1a\") " pod="openshift-dns/dns-default-pnvtm"
Jul 09 18:05:07.566975 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.566936    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95525\" (UniqueName: \"kubernetes.io/projected/d8df3e9b-66dd-45af-9829-41d518142d58-kube-api-access-95525\") pod \"insights-runtime-extractor-q6mvh\" (UID: \"d8df3e9b-66dd-45af-9829-41d518142d58\") " pod="openshift-insights/insights-runtime-extractor-q6mvh"
Jul 09 18:05:07.566975 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.566971    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8df3e9b-66dd-45af-9829-41d518142d58-data-volume\") pod \"insights-runtime-extractor-q6mvh\" (UID: \"d8df3e9b-66dd-45af-9829-41d518142d58\") " pod="openshift-insights/insights-runtime-extractor-q6mvh"
Jul 09 18:05:07.567185 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.567130    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8df3e9b-66dd-45af-9829-41d518142d58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6mvh\" (UID: \"d8df3e9b-66dd-45af-9829-41d518142d58\") " pod="openshift-insights/insights-runtime-extractor-q6mvh"
Jul 09 18:05:07.567185 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.567162    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8df3e9b-66dd-45af-9829-41d518142d58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q6mvh\" (UID: \"d8df3e9b-66dd-45af-9829-41d518142d58\") " pod="openshift-insights/insights-runtime-extractor-q6mvh"
Jul 09 18:05:07.567319 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.567192    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf55x\" (UniqueName: \"kubernetes.io/projected/51ac9338-1662-42ce-8b4b-069afc8fcd55-kube-api-access-sf55x\") pod \"ingress-canary-5nkqz\" (UID: \"51ac9338-1662-42ce-8b4b-069afc8fcd55\") " pod="openshift-ingress-canary/ingress-canary-5nkqz"
Jul 09 18:05:07.567319 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.567271    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8df3e9b-66dd-45af-9829-41d518142d58-data-volume\") pod \"insights-runtime-extractor-q6mvh\" (UID: \"d8df3e9b-66dd-45af-9829-41d518142d58\") " pod="openshift-insights/insights-runtime-extractor-q6mvh"
Jul 09 18:05:07.567319 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.567307    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d8df3e9b-66dd-45af-9829-41d518142d58-crio-socket\") pod \"insights-runtime-extractor-q6mvh\" (UID: \"d8df3e9b-66dd-45af-9829-41d518142d58\") " pod="openshift-insights/insights-runtime-extractor-q6mvh"
Jul 09 18:05:07.567455 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.567350    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51ac9338-1662-42ce-8b4b-069afc8fcd55-cert\") pod \"ingress-canary-5nkqz\" (UID: \"51ac9338-1662-42ce-8b4b-069afc8fcd55\") " pod="openshift-ingress-canary/ingress-canary-5nkqz"
Jul 09 18:05:07.567496 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.567458    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d8df3e9b-66dd-45af-9829-41d518142d58-crio-socket\") pod \"insights-runtime-extractor-q6mvh\" (UID: \"d8df3e9b-66dd-45af-9829-41d518142d58\") " pod="openshift-insights/insights-runtime-extractor-q6mvh"
Jul 09 18:05:07.567687 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.567665    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8df3e9b-66dd-45af-9829-41d518142d58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q6mvh\" (UID: \"d8df3e9b-66dd-45af-9829-41d518142d58\") " pod="openshift-insights/insights-runtime-extractor-q6mvh"
Jul 09 18:05:07.569402 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.569382    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8df3e9b-66dd-45af-9829-41d518142d58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6mvh\" (UID: \"d8df3e9b-66dd-45af-9829-41d518142d58\") " pod="openshift-insights/insights-runtime-extractor-q6mvh"
Jul 09 18:05:07.576513 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.576492    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95525\" (UniqueName: \"kubernetes.io/projected/d8df3e9b-66dd-45af-9829-41d518142d58-kube-api-access-95525\") pod \"insights-runtime-extractor-q6mvh\" (UID: \"d8df3e9b-66dd-45af-9829-41d518142d58\") " pod="openshift-insights/insights-runtime-extractor-q6mvh"
Jul 09 18:05:07.666495 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.666458    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pnvtm"
Jul 09 18:05:07.667632 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.667611    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf55x\" (UniqueName: \"kubernetes.io/projected/51ac9338-1662-42ce-8b4b-069afc8fcd55-kube-api-access-sf55x\") pod \"ingress-canary-5nkqz\" (UID: \"51ac9338-1662-42ce-8b4b-069afc8fcd55\") " pod="openshift-ingress-canary/ingress-canary-5nkqz"
Jul 09 18:05:07.667687 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.667653    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51ac9338-1662-42ce-8b4b-069afc8fcd55-cert\") pod \"ingress-canary-5nkqz\" (UID: \"51ac9338-1662-42ce-8b4b-069afc8fcd55\") " pod="openshift-ingress-canary/ingress-canary-5nkqz"
Jul 09 18:05:07.678524 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.678497    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf55x\" (UniqueName: \"kubernetes.io/projected/51ac9338-1662-42ce-8b4b-069afc8fcd55-kube-api-access-sf55x\") pod \"ingress-canary-5nkqz\" (UID: \"51ac9338-1662-42ce-8b4b-069afc8fcd55\") " pod="openshift-ingress-canary/ingress-canary-5nkqz"
Jul 09 18:05:07.678846 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.678831    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51ac9338-1662-42ce-8b4b-069afc8fcd55-cert\") pod \"ingress-canary-5nkqz\" (UID: \"51ac9338-1662-42ce-8b4b-069afc8fcd55\") " pod="openshift-ingress-canary/ingress-canary-5nkqz"
Jul 09 18:05:07.684305 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.684286    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q6mvh"
Jul 09 18:05:07.780540 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.780380    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5nkqz"
Jul 09 18:05:07.868154 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.865193    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q6mvh"]
Jul 09 18:05:07.871866 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:05:07.871843    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8df3e9b_66dd_45af_9829_41d518142d58.slice/crio-c2e728efdc912a2f8607f52cdefc34860bee1705e543e4911e4e8e35172a38f0 WatchSource:0}: Error finding container c2e728efdc912a2f8607f52cdefc34860bee1705e543e4911e4e8e35172a38f0: Status 404 returned error can't find the container with id c2e728efdc912a2f8607f52cdefc34860bee1705e543e4911e4e8e35172a38f0
Jul 09 18:05:07.875841 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.875814    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pnvtm"]
Jul 09 18:05:07.940242 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.940194    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5nkqz"]
Jul 09 18:05:07.943135 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:05:07.943104    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51ac9338_1662_42ce_8b4b_069afc8fcd55.slice/crio-245f2f637f8c885d25658fd93d5c64a1ac3fb2798d3e22f3705322c53250445d WatchSource:0}: Error finding container 245f2f637f8c885d25658fd93d5c64a1ac3fb2798d3e22f3705322c53250445d: Status 404 returned error can't find the container with id 245f2f637f8c885d25658fd93d5c64a1ac3fb2798d3e22f3705322c53250445d
Jul 09 18:05:07.954665 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.954615    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:05:07.954752 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.954615    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:05:07.954842 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.954614    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:05:07.957187 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.957171    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pdjws\""
Jul 09 18:05:07.957311 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.957296    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 18:05:07.957608 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.957596    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 18:05:07.958259 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.958244    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 18:05:07.958497 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.958481    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 18:05:07.958553 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:07.958523    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ws4d4\""
Jul 09 18:05:08.186705 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:08.186663    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q6mvh" event={"ID":"d8df3e9b-66dd-45af-9829-41d518142d58","Type":"ContainerStarted","Data":"c2e728efdc912a2f8607f52cdefc34860bee1705e543e4911e4e8e35172a38f0"}
Jul 09 18:05:08.187726 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:08.187703    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5nkqz" event={"ID":"51ac9338-1662-42ce-8b4b-069afc8fcd55","Type":"ContainerStarted","Data":"245f2f637f8c885d25658fd93d5c64a1ac3fb2798d3e22f3705322c53250445d"}
Jul 09 18:05:08.188695 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:08.188674    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pnvtm" event={"ID":"996150c6-fdce-44a0-963e-d607d34fdc1a","Type":"ContainerStarted","Data":"02f97952c685adf71d9f3caaf1ecf12bb505d50fe94a878a0c04ebecd3491759"}
Jul 09 18:05:09.296005 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:09.295971    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vnbxz"]
Jul 09 18:05:09.313572 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:09.313383    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vnbxz"
Jul 09 18:05:09.317638 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:09.317433    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 09 18:05:09.319303 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:09.319280    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-wr7vc\""
Jul 09 18:05:09.320281 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:09.320260    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vnbxz"]
Jul 09 18:05:09.383116 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:09.382814    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/44fd6ea3-1676-4fd0-8b6b-94ecf269733e-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-vnbxz\" (UID: \"44fd6ea3-1676-4fd0-8b6b-94ecf269733e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vnbxz"
Jul 09 18:05:09.484026 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:09.483964    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/44fd6ea3-1676-4fd0-8b6b-94ecf269733e-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-vnbxz\" (UID: \"44fd6ea3-1676-4fd0-8b6b-94ecf269733e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vnbxz"
Jul 09 18:05:09.488689 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:09.488623    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/44fd6ea3-1676-4fd0-8b6b-94ecf269733e-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-vnbxz\" (UID: \"44fd6ea3-1676-4fd0-8b6b-94ecf269733e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vnbxz"
Jul 09 18:05:09.628135 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:09.628055    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vnbxz"
Jul 09 18:05:09.799090 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:09.799061    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vnbxz"]
Jul 09 18:05:10.229401 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:05:10.229371    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44fd6ea3_1676_4fd0_8b6b_94ecf269733e.slice/crio-9389e916ec9e410952d045664f51e6f69e51bdbb0fd89ae6bd6d561f34133c4d WatchSource:0}: Error finding container 9389e916ec9e410952d045664f51e6f69e51bdbb0fd89ae6bd6d561f34133c4d: Status 404 returned error can't find the container with id 9389e916ec9e410952d045664f51e6f69e51bdbb0fd89ae6bd6d561f34133c4d
Jul 09 18:05:11.198565 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:11.198516    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vnbxz" event={"ID":"44fd6ea3-1676-4fd0-8b6b-94ecf269733e","Type":"ContainerStarted","Data":"9389e916ec9e410952d045664f51e6f69e51bdbb0fd89ae6bd6d561f34133c4d"}
Jul 09 18:05:14.207835 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:14.207793    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q6mvh" event={"ID":"d8df3e9b-66dd-45af-9829-41d518142d58","Type":"ContainerStarted","Data":"96634ce72ded045349582272fa8d8ec72b9ec947600ae259089325e48fd710a8"}
Jul 09 18:05:14.209497 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:14.209471    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pnvtm" event={"ID":"996150c6-fdce-44a0-963e-d607d34fdc1a","Type":"ContainerStarted","Data":"4a4e50119a79e33d44757047ca2b2b3ff691669a851ac4395a83e9ab7c3ad1bd"}
Jul 09 18:05:14.209611 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:14.209515    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pnvtm" event={"ID":"996150c6-fdce-44a0-963e-d607d34fdc1a","Type":"ContainerStarted","Data":"0e6b365241c8e81ab603e98a2a34b5ddba44956fceee11f7617feb177bc71343"}
Jul 09 18:05:14.209611 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:14.209576    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pnvtm"
Jul 09 18:05:14.210863 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:14.210841    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5nkqz" event={"ID":"51ac9338-1662-42ce-8b4b-069afc8fcd55","Type":"ContainerStarted","Data":"aa40589b9b46e96a64c243d68c09585dcfa07e71b669ee1ec0260d3888c846aa"}
Jul 09 18:05:14.212100 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:14.212079    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vnbxz" event={"ID":"44fd6ea3-1676-4fd0-8b6b-94ecf269733e","Type":"ContainerStarted","Data":"b27429ed253049a6340a3780e8b5f93f27b634c3ef80311106524677530bf8e6"}
Jul 09 18:05:14.212267 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:14.212252    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vnbxz"
Jul 09 18:05:14.217276 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:14.217258    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vnbxz"
Jul 09 18:05:14.234393 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:14.234356    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pnvtm" podStartSLOduration=1.58148238 podStartE2EDuration="7.234342989s" totalImagesPullingTime="5.652860609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:14.233015864 +0000 UTC m=+59.902096818" watchObservedRunningTime="2026-07-09 18:05:14.234342989 +0000 UTC m=+59.903423944"
Jul 09 18:05:14.259183 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:14.259142    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vnbxz" podStartSLOduration=1.68571349 podStartE2EDuration="5.259128762s" totalImagesPullingTime="3.573415272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:14.258336979 +0000 UTC m=+59.927417935" watchObservedRunningTime="2026-07-09 18:05:14.259128762 +0000 UTC m=+59.928209695"
Jul 09 18:05:14.290879 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:14.290829    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5nkqz" podStartSLOduration=1.437368394 podStartE2EDuration="7.2908102s" totalImagesPullingTime="5.853441806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:14.29029581 +0000 UTC m=+59.959376766" watchObservedRunningTime="2026-07-09 18:05:14.2908102 +0000 UTC m=+59.959891168"
Jul 09 18:05:15.135614 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:15.135581    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lbftk"
Jul 09 18:05:15.217556 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:15.217515    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q6mvh" event={"ID":"d8df3e9b-66dd-45af-9829-41d518142d58","Type":"ContainerStarted","Data":"536519d67975b81f7aec6ead23e0c0d00c4e1e2efc9103c648de4319ac5c88ba"}
Jul 09 18:05:15.504737 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:15.504705    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-ddwdk"]
Jul 09 18:05:15.506805 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:15.506772    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-ddwdk"
Jul 09 18:05:15.525147 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:15.525118    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 09 18:05:15.525297 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:15.525127    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 09 18:05:15.525361 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:15.525329    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 09 18:05:15.526653 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:15.526639    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-zx658\""
Jul 09 18:05:15.528884 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:15.528861    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 09 18:05:15.529061 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:15.529044    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 09 18:05:15.534383 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:15.534360    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f1d9734-a399-4397-94e9-7e40770d1b0a-metrics-client-ca\") pod \"prometheus-operator-64785b6675-ddwdk\" (UID: \"9f1d9734-a399-4397-94e9-7e40770d1b0a\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ddwdk"
Jul 09 18:05:15.534492 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:15.534398    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9f1d9734-a399-4397-94e9-7e40770d1b0a-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-ddwdk\" (UID: \"9f1d9734-a399-4397-94e9-7e40770d1b0a\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ddwdk"
Jul 09 18:05:15.534492 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:15.534449    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzt2t\" (UniqueName: \"kubernetes.io/projected/9f1d9734-a399-4397-94e9-7e40770d1b0a-kube-api-access-zzt2t\") pod \"prometheus-operator-64785b6675-ddwdk\" (UID: \"9f1d9734-a399-4397-94e9-7e40770d1b0a\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ddwdk"
Jul 09 18:05:15.534606 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:15.534553    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f1d9734-a399-4397-94e9-7e40770d1b0a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-ddwdk\" (UID: \"9f1d9734-a399-4397-94e9-7e40770d1b0a\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ddwdk"
Jul 09 18:05:15.543205 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:15.543181    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-ddwdk"]
Jul 09 18:05:15.635320 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:15.635288    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzt2t\" (UniqueName: \"kubernetes.io/projected/9f1d9734-a399-4397-94e9-7e40770d1b0a-kube-api-access-zzt2t\") pod \"prometheus-operator-64785b6675-ddwdk\" (UID: \"9f1d9734-a399-4397-94e9-7e40770d1b0a\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ddwdk"
Jul 09 18:05:15.635490 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:15.635360    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f1d9734-a399-4397-94e9-7e40770d1b0a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-ddwdk\" (UID: \"9f1d9734-a399-4397-94e9-7e40770d1b0a\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ddwdk"
Jul 09 18:05:15.635490 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:15.635403    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f1d9734-a399-4397-94e9-7e40770d1b0a-metrics-client-ca\") pod \"prometheus-operator-64785b6675-ddwdk\" (UID: \"9f1d9734-a399-4397-94e9-7e40770d1b0a\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ddwdk"
Jul 09 18:05:15.635490 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:15.635429    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9f1d9734-a399-4397-94e9-7e40770d1b0a-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-ddwdk\" (UID: \"9f1d9734-a399-4397-94e9-7e40770d1b0a\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ddwdk"
Jul 09 18:05:15.636126 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:15.636092    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f1d9734-a399-4397-94e9-7e40770d1b0a-metrics-client-ca\") pod \"prometheus-operator-64785b6675-ddwdk\" (UID: \"9f1d9734-a399-4397-94e9-7e40770d1b0a\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ddwdk"
Jul 09 18:05:15.638081 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:15.638056    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9f1d9734-a399-4397-94e9-7e40770d1b0a-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-ddwdk\" (UID: \"9f1d9734-a399-4397-94e9-7e40770d1b0a\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ddwdk"
Jul 09 18:05:15.638231 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:15.638068    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f1d9734-a399-4397-94e9-7e40770d1b0a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-ddwdk\" (UID: \"9f1d9734-a399-4397-94e9-7e40770d1b0a\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ddwdk"
Jul 09 18:05:15.654178 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:15.654157    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzt2t\" (UniqueName: \"kubernetes.io/projected/9f1d9734-a399-4397-94e9-7e40770d1b0a-kube-api-access-zzt2t\") pod \"prometheus-operator-64785b6675-ddwdk\" (UID: \"9f1d9734-a399-4397-94e9-7e40770d1b0a\") " pod="openshift-monitoring/prometheus-operator-64785b6675-ddwdk"
Jul 09 18:05:15.817222 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:15.817121    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-ddwdk"
Jul 09 18:05:15.974488 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:15.974456    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-ddwdk"]
Jul 09 18:05:16.991579 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:05:16.991543    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f1d9734_a399_4397_94e9_7e40770d1b0a.slice/crio-a7972b168ef45aa81b195e15e20d48767029390353914958ab9901e8f3040abe WatchSource:0}: Error finding container a7972b168ef45aa81b195e15e20d48767029390353914958ab9901e8f3040abe: Status 404 returned error can't find the container with id a7972b168ef45aa81b195e15e20d48767029390353914958ab9901e8f3040abe
Jul 09 18:05:17.225175 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:17.225137    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q6mvh" event={"ID":"d8df3e9b-66dd-45af-9829-41d518142d58","Type":"ContainerStarted","Data":"cb2bbd3368f1ac2485d3424790d863cd6dd4ac6be08df38bf1e763ba457daf25"}
Jul 09 18:05:17.226277 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:17.226250    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-ddwdk" event={"ID":"9f1d9734-a399-4397-94e9-7e40770d1b0a","Type":"ContainerStarted","Data":"a7972b168ef45aa81b195e15e20d48767029390353914958ab9901e8f3040abe"}
Jul 09 18:05:17.248389 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:17.248309    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-q6mvh" podStartSLOduration=1.254086274 podStartE2EDuration="10.248296389s" totalImagesPullingTime="8.994210115s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:05:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:17.246867599 +0000 UTC m=+62.915948554" watchObservedRunningTime="2026-07-09 18:05:17.248296389 +0000 UTC m=+62.917377342"
Jul 09 18:05:19.234812 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:19.234778    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-ddwdk" event={"ID":"9f1d9734-a399-4397-94e9-7e40770d1b0a","Type":"ContainerStarted","Data":"920293764d99a1857ec6d722783cd94c4cefb6e5138f5b7e84368c5a328a4d28"}
Jul 09 18:05:19.235183 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:19.234832    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-ddwdk" event={"ID":"9f1d9734-a399-4397-94e9-7e40770d1b0a","Type":"ContainerStarted","Data":"8062b13a1ae1095970932274368d8086e7f5eff67820957d9f9e3000e6fbae4f"}
Jul 09 18:05:19.255325 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:19.255278    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-ddwdk" podStartSLOduration=2.949449204 podStartE2EDuration="4.255261843s" totalImagesPullingTime="1.305812639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:19.253580028 +0000 UTC m=+64.922660981" watchObservedRunningTime="2026-07-09 18:05:19.255261843 +0000 UTC m=+64.924342797"
Jul 09 18:05:20.572298 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.572259    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cddd2f06-2abf-4246-ac09-acc38c065830-original-pull-secret\") pod \"global-pull-secret-syncer-6ng78\" (UID: \"cddd2f06-2abf-4246-ac09-acc38c065830\") " pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:05:20.575441 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.575426    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 18:05:20.585375 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.585357    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cddd2f06-2abf-4246-ac09-acc38c065830-original-pull-secret\") pod \"global-pull-secret-syncer-6ng78\" (UID: \"cddd2f06-2abf-4246-ac09-acc38c065830\") " pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:05:20.672747 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.672710    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5de2134-56a4-4318-a22f-90937385dc5b-metrics-certs\") pod \"network-metrics-daemon-wqgvh\" (UID: \"d5de2134-56a4-4318-a22f-90937385dc5b\") " pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:05:20.682317 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.682287    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 18:05:20.685387 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.685367    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5de2134-56a4-4318-a22f-90937385dc5b-metrics-certs\") pod \"network-metrics-daemon-wqgvh\" (UID: \"d5de2134-56a4-4318-a22f-90937385dc5b\") " pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:05:20.774096 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.774059    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxzgc\" (UniqueName: \"kubernetes.io/projected/52ba7983-15e4-437e-a86f-36f0048b0614-kube-api-access-lxzgc\") pod \"network-check-target-sf462\" (UID: \"52ba7983-15e4-437e-a86f-36f0048b0614\") " pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:05:20.778566 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.778549    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 18:05:20.787332 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.787318    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 18:05:20.797546 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.797526    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxzgc\" (UniqueName: \"kubernetes.io/projected/52ba7983-15e4-437e-a86f-36f0048b0614-kube-api-access-lxzgc\") pod \"network-check-target-sf462\" (UID: \"52ba7983-15e4-437e-a86f-36f0048b0614\") " pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:05:20.868201 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.868120    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pdjws\""
Jul 09 18:05:20.870734 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.870717    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ng78"
Jul 09 18:05:20.875400 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.873918    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-4t62b"]
Jul 09 18:05:20.875400 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.875370    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:05:20.877316 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.877297    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-4t62b"
Jul 09 18:05:20.882408 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.882388    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 09 18:05:20.882570 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.882549    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 09 18:05:20.882685 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.882666    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ws4d4\""
Jul 09 18:05:20.882685 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.882677    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-wxrmd\""
Jul 09 18:05:20.885520 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.885500    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wqgvh"
Jul 09 18:05:20.910920 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.910822    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-4t62b"]
Jul 09 18:05:20.949413 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.948061    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-wrs2c"]
Jul 09 18:05:20.953279 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.951824    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zfclm"]
Jul 09 18:05:20.953279 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.952538    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-wrs2c"
Jul 09 18:05:20.956312 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.954245    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zfclm"
Jul 09 18:05:20.957924 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.957614    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 09 18:05:20.959851 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.958475    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 09 18:05:20.959851 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.958678    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 09 18:05:20.959851 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.958890    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 09 18:05:20.959851 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.959179    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-xq4gd\""
Jul 09 18:05:20.959851 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.959407    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 09 18:05:20.959851 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.959663    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 09 18:05:20.960236 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.960207    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sdp9m\""
Jul 09 18:05:20.977285 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.977260    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c61025f3-9d66-42c4-8f78-c9b828e113dc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-4t62b\" (UID: \"c61025f3-9d66-42c4-8f78-c9b828e113dc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-4t62b"
Jul 09 18:05:20.977879 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.977700    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c61025f3-9d66-42c4-8f78-c9b828e113dc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-4t62b\" (UID: \"c61025f3-9d66-42c4-8f78-c9b828e113dc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-4t62b"
Jul 09 18:05:20.977879 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.977745    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkx4l\" (UniqueName: \"kubernetes.io/projected/c61025f3-9d66-42c4-8f78-c9b828e113dc-kube-api-access-nkx4l\") pod \"openshift-state-metrics-84bd68b896-4t62b\" (UID: \"c61025f3-9d66-42c4-8f78-c9b828e113dc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-4t62b"
Jul 09 18:05:20.977879 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.977793    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c61025f3-9d66-42c4-8f78-c9b828e113dc-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-4t62b\" (UID: \"c61025f3-9d66-42c4-8f78-c9b828e113dc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-4t62b"
Jul 09 18:05:20.978708 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:20.978638    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-wrs2c"]
Jul 09 18:05:21.036784 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.036748    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sf462"]
Jul 09 18:05:21.040740 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:05:21.040711    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ba7983_15e4_437e_a86f_36f0048b0614.slice/crio-efcf9c4beeea131e704618f3893c0b0dbac94f5916faa4cb297386190eea7f3b WatchSource:0}: Error finding container efcf9c4beeea131e704618f3893c0b0dbac94f5916faa4cb297386190eea7f3b: Status 404 returned error can't find the container with id efcf9c4beeea131e704618f3893c0b0dbac94f5916faa4cb297386190eea7f3b
Jul 09 18:05:21.078267 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.078230    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b1dcaaf4-af90-4ca5-9351-33c1e2c612cf-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-wrs2c\" (UID: \"b1dcaaf4-af90-4ca5-9351-33c1e2c612cf\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrs2c"
Jul 09 18:05:21.078267 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.078264    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md6d2\" (UniqueName: \"kubernetes.io/projected/b1dcaaf4-af90-4ca5-9351-33c1e2c612cf-kube-api-access-md6d2\") pod \"kube-state-metrics-785778ff74-wrs2c\" (UID: \"b1dcaaf4-af90-4ca5-9351-33c1e2c612cf\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrs2c"
Jul 09 18:05:21.078486 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.078295    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c61025f3-9d66-42c4-8f78-c9b828e113dc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-4t62b\" (UID: \"c61025f3-9d66-42c4-8f78-c9b828e113dc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-4t62b"
Jul 09 18:05:21.078486 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.078315    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6feeb09a-7eb5-4c46-a687-202ddba40804-root\") pod \"node-exporter-zfclm\" (UID: \"6feeb09a-7eb5-4c46-a687-202ddba40804\") " pod="openshift-monitoring/node-exporter-zfclm"
Jul 09 18:05:21.078486 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.078334    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6feeb09a-7eb5-4c46-a687-202ddba40804-node-exporter-textfile\") pod \"node-exporter-zfclm\" (UID: \"6feeb09a-7eb5-4c46-a687-202ddba40804\") " pod="openshift-monitoring/node-exporter-zfclm"
Jul 09 18:05:21.078486 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.078352    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1dcaaf4-af90-4ca5-9351-33c1e2c612cf-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-wrs2c\" (UID: \"b1dcaaf4-af90-4ca5-9351-33c1e2c612cf\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrs2c"
Jul 09 18:05:21.078486 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.078366    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1dcaaf4-af90-4ca5-9351-33c1e2c612cf-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-wrs2c\" (UID: \"b1dcaaf4-af90-4ca5-9351-33c1e2c612cf\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrs2c"
Jul 09 18:05:21.078486 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.078387    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6feeb09a-7eb5-4c46-a687-202ddba40804-node-exporter-wtmp\") pod \"node-exporter-zfclm\" (UID: \"6feeb09a-7eb5-4c46-a687-202ddba40804\") " pod="openshift-monitoring/node-exporter-zfclm"
Jul 09 18:05:21.078486 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.078434    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b1dcaaf4-af90-4ca5-9351-33c1e2c612cf-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-wrs2c\" (UID: \"b1dcaaf4-af90-4ca5-9351-33c1e2c612cf\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrs2c"
Jul 09 18:05:21.078486 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.078477    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6feeb09a-7eb5-4c46-a687-202ddba40804-node-exporter-tls\") pod \"node-exporter-zfclm\" (UID: \"6feeb09a-7eb5-4c46-a687-202ddba40804\") " pod="openshift-monitoring/node-exporter-zfclm"
Jul 09 18:05:21.078870 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.078522    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkx4l\" (UniqueName: \"kubernetes.io/projected/c61025f3-9d66-42c4-8f78-c9b828e113dc-kube-api-access-nkx4l\") pod \"openshift-state-metrics-84bd68b896-4t62b\" (UID: \"c61025f3-9d66-42c4-8f78-c9b828e113dc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-4t62b"
Jul 09 18:05:21.078870 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.078539    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6feeb09a-7eb5-4c46-a687-202ddba40804-metrics-client-ca\") pod \"node-exporter-zfclm\" (UID: \"6feeb09a-7eb5-4c46-a687-202ddba40804\") " pod="openshift-monitoring/node-exporter-zfclm"
Jul 09 18:05:21.078870 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.078558    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6feeb09a-7eb5-4c46-a687-202ddba40804-node-exporter-accelerators-collector-config\") pod \"node-exporter-zfclm\" (UID: \"6feeb09a-7eb5-4c46-a687-202ddba40804\") " pod="openshift-monitoring/node-exporter-zfclm"
Jul 09 18:05:21.078870 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.078574    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c61025f3-9d66-42c4-8f78-c9b828e113dc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-4t62b\" (UID: \"c61025f3-9d66-42c4-8f78-c9b828e113dc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-4t62b"
Jul 09 18:05:21.078870 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.078591    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfmr2\" (UniqueName: \"kubernetes.io/projected/6feeb09a-7eb5-4c46-a687-202ddba40804-kube-api-access-zfmr2\") pod \"node-exporter-zfclm\" (UID: \"6feeb09a-7eb5-4c46-a687-202ddba40804\") " pod="openshift-monitoring/node-exporter-zfclm"
Jul 09 18:05:21.078870 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.078607    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1dcaaf4-af90-4ca5-9351-33c1e2c612cf-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-wrs2c\" (UID: \"b1dcaaf4-af90-4ca5-9351-33c1e2c612cf\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrs2c"
Jul 09 18:05:21.078870 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.078627    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c61025f3-9d66-42c4-8f78-c9b828e113dc-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-4t62b\" (UID: \"c61025f3-9d66-42c4-8f78-c9b828e113dc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-4t62b"
Jul 09 18:05:21.078870 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.078645    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6feeb09a-7eb5-4c46-a687-202ddba40804-sys\") pod \"node-exporter-zfclm\" (UID: \"6feeb09a-7eb5-4c46-a687-202ddba40804\") " pod="openshift-monitoring/node-exporter-zfclm"
Jul 09 18:05:21.078870 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:05:21.078716    2563 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 09 18:05:21.078870 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.078737    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6feeb09a-7eb5-4c46-a687-202ddba40804-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zfclm\" (UID: \"6feeb09a-7eb5-4c46-a687-202ddba40804\") " pod="openshift-monitoring/node-exporter-zfclm"
Jul 09 18:05:21.078870 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:05:21.078796    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c61025f3-9d66-42c4-8f78-c9b828e113dc-openshift-state-metrics-tls podName:c61025f3-9d66-42c4-8f78-c9b828e113dc nodeName:}" failed. No retries permitted until 2026-07-09 18:05:21.578776901 +0000 UTC m=+67.247857835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/c61025f3-9d66-42c4-8f78-c9b828e113dc-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-4t62b" (UID: "c61025f3-9d66-42c4-8f78-c9b828e113dc") : secret "openshift-state-metrics-tls" not found
Jul 09 18:05:21.079425 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.079406    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c61025f3-9d66-42c4-8f78-c9b828e113dc-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-4t62b\" (UID: \"c61025f3-9d66-42c4-8f78-c9b828e113dc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-4t62b"
Jul 09 18:05:21.080848 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.080827    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c61025f3-9d66-42c4-8f78-c9b828e113dc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-4t62b\" (UID: \"c61025f3-9d66-42c4-8f78-c9b828e113dc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-4t62b"
Jul 09 18:05:21.090053 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.090028    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkx4l\" (UniqueName: \"kubernetes.io/projected/c61025f3-9d66-42c4-8f78-c9b828e113dc-kube-api-access-nkx4l\") pod \"openshift-state-metrics-84bd68b896-4t62b\" (UID: \"c61025f3-9d66-42c4-8f78-c9b828e113dc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-4t62b"
Jul 09 18:05:21.179286 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.179247    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6feeb09a-7eb5-4c46-a687-202ddba40804-metrics-client-ca\") pod \"node-exporter-zfclm\" (UID: \"6feeb09a-7eb5-4c46-a687-202ddba40804\") " pod="openshift-monitoring/node-exporter-zfclm"
Jul 09 18:05:21.179468 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.179296    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6feeb09a-7eb5-4c46-a687-202ddba40804-node-exporter-accelerators-collector-config\") pod \"node-exporter-zfclm\" (UID: \"6feeb09a-7eb5-4c46-a687-202ddba40804\") " pod="openshift-monitoring/node-exporter-zfclm"
Jul 09 18:05:21.179468 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.179427    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfmr2\" (UniqueName: \"kubernetes.io/projected/6feeb09a-7eb5-4c46-a687-202ddba40804-kube-api-access-zfmr2\") pod \"node-exporter-zfclm\" (UID: \"6feeb09a-7eb5-4c46-a687-202ddba40804\") " pod="openshift-monitoring/node-exporter-zfclm"
Jul 09 18:05:21.179570 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.179463    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1dcaaf4-af90-4ca5-9351-33c1e2c612cf-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-wrs2c\" (UID: \"b1dcaaf4-af90-4ca5-9351-33c1e2c612cf\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrs2c"
Jul 09 18:05:21.179570 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.179503    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6feeb09a-7eb5-4c46-a687-202ddba40804-sys\") pod \"node-exporter-zfclm\" (UID: \"6feeb09a-7eb5-4c46-a687-202ddba40804\") " pod="openshift-monitoring/node-exporter-zfclm"
Jul 09 18:05:21.179570 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.179552    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6feeb09a-7eb5-4c46-a687-202ddba40804-sys\") pod \"node-exporter-zfclm\" (UID: \"6feeb09a-7eb5-4c46-a687-202ddba40804\") " pod="openshift-monitoring/node-exporter-zfclm"
Jul 09 18:05:21.179570 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.179552    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6feeb09a-7eb5-4c46-a687-202ddba40804-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zfclm\" (UID: \"6feeb09a-7eb5-4c46-a687-202ddba40804\") " pod="openshift-monitoring/node-exporter-zfclm"
Jul 09 18:05:21.179762 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.179601    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b1dcaaf4-af90-4ca5-9351-33c1e2c612cf-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-wrs2c\" (UID: \"b1dcaaf4-af90-4ca5-9351-33c1e2c612cf\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrs2c"
Jul 09 18:05:21.179762 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.179628    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md6d2\" (UniqueName: \"kubernetes.io/projected/b1dcaaf4-af90-4ca5-9351-33c1e2c612cf-kube-api-access-md6d2\") pod \"kube-state-metrics-785778ff74-wrs2c\" (UID: \"b1dcaaf4-af90-4ca5-9351-33c1e2c612cf\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrs2c"
Jul 09 18:05:21.179762 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.179678    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6feeb09a-7eb5-4c46-a687-202ddba40804-root\") pod \"node-exporter-zfclm\" (UID: \"6feeb09a-7eb5-4c46-a687-202ddba40804\") " pod="openshift-monitoring/node-exporter-zfclm"
Jul 09 18:05:21.179762 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.179708    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6feeb09a-7eb5-4c46-a687-202ddba40804-node-exporter-textfile\") pod \"node-exporter-zfclm\" (UID: \"6feeb09a-7eb5-4c46-a687-202ddba40804\") " pod="openshift-monitoring/node-exporter-zfclm"
Jul 09 18:05:21.179762 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.179737    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1dcaaf4-af90-4ca5-9351-33c1e2c612cf-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-wrs2c\" (UID: \"b1dcaaf4-af90-4ca5-9351-33c1e2c612cf\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrs2c"
Jul 09 18:05:21.179762 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.179761    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1dcaaf4-af90-4ca5-9351-33c1e2c612cf-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-wrs2c\" (UID: \"b1dcaaf4-af90-4ca5-9351-33c1e2c612cf\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrs2c"
Jul 09 18:05:21.180078 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.179790    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6feeb09a-7eb5-4c46-a687-202ddba40804-node-exporter-wtmp\") pod \"node-exporter-zfclm\" (UID: \"6feeb09a-7eb5-4c46-a687-202ddba40804\") " pod="openshift-monitoring/node-exporter-zfclm"
Jul 09 18:05:21.180078 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.179823    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b1dcaaf4-af90-4ca5-9351-33c1e2c612cf-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-wrs2c\" (UID: \"b1dcaaf4-af90-4ca5-9351-33c1e2c612cf\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrs2c"
Jul 09 18:05:21.180078 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.179844    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6feeb09a-7eb5-4c46-a687-202ddba40804-node-exporter-tls\") pod \"node-exporter-zfclm\" (UID: \"6feeb09a-7eb5-4c46-a687-202ddba40804\") " pod="openshift-monitoring/node-exporter-zfclm"
Jul 09 18:05:21.180078 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.179924    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6feeb09a-7eb5-4c46-a687-202ddba40804-metrics-client-ca\") pod \"node-exporter-zfclm\" (UID: \"6feeb09a-7eb5-4c46-a687-202ddba40804\") " pod="openshift-monitoring/node-exporter-zfclm"
Jul 09 18:05:21.180078 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.179955    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6feeb09a-7eb5-4c46-a687-202ddba40804-node-exporter-accelerators-collector-config\") pod \"node-exporter-zfclm\" (UID: \"6feeb09a-7eb5-4c46-a687-202ddba40804\") " pod="openshift-monitoring/node-exporter-zfclm"
Jul 09 18:05:21.180310 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.180080    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6feeb09a-7eb5-4c46-a687-202ddba40804-node-exporter-wtmp\") pod \"node-exporter-zfclm\" (UID: \"6feeb09a-7eb5-4c46-a687-202ddba40804\") " pod="openshift-monitoring/node-exporter-zfclm"
Jul 09 18:05:21.180310 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.180138    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6feeb09a-7eb5-4c46-a687-202ddba40804-root\") pod \"node-exporter-zfclm\" (UID: \"6feeb09a-7eb5-4c46-a687-202ddba40804\") " pod="openshift-monitoring/node-exporter-zfclm"
Jul 09 18:05:21.180416 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.180381    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6feeb09a-7eb5-4c46-a687-202ddba40804-node-exporter-textfile\") pod \"node-exporter-zfclm\" (UID: \"6feeb09a-7eb5-4c46-a687-202ddba40804\") " pod="openshift-monitoring/node-exporter-zfclm"
Jul 09 18:05:21.180615 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.180591    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b1dcaaf4-af90-4ca5-9351-33c1e2c612cf-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-wrs2c\" (UID: \"b1dcaaf4-af90-4ca5-9351-33c1e2c612cf\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrs2c"
Jul 09 18:05:21.180615 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.180600    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b1dcaaf4-af90-4ca5-9351-33c1e2c612cf-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-wrs2c\" (UID: \"b1dcaaf4-af90-4ca5-9351-33c1e2c612cf\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrs2c"
Jul 09 18:05:21.180870 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.180851    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1dcaaf4-af90-4ca5-9351-33c1e2c612cf-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-wrs2c\" (UID: \"b1dcaaf4-af90-4ca5-9351-33c1e2c612cf\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrs2c"
Jul 09 18:05:21.182122 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.182101    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1dcaaf4-af90-4ca5-9351-33c1e2c612cf-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-wrs2c\" (UID: \"b1dcaaf4-af90-4ca5-9351-33c1e2c612cf\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrs2c"
Jul 09 18:05:21.182307 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.182287    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1dcaaf4-af90-4ca5-9351-33c1e2c612cf-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-wrs2c\" (UID: \"b1dcaaf4-af90-4ca5-9351-33c1e2c612cf\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrs2c"
Jul 09 18:05:21.182357 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.182295    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6feeb09a-7eb5-4c46-a687-202ddba40804-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zfclm\" (UID: \"6feeb09a-7eb5-4c46-a687-202ddba40804\") " pod="openshift-monitoring/node-exporter-zfclm"
Jul 09 18:05:21.182357 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.182342    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6feeb09a-7eb5-4c46-a687-202ddba40804-node-exporter-tls\") pod \"node-exporter-zfclm\" (UID: \"6feeb09a-7eb5-4c46-a687-202ddba40804\") " pod="openshift-monitoring/node-exporter-zfclm"
Jul 09 18:05:21.199905 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.199887    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md6d2\" (UniqueName: \"kubernetes.io/projected/b1dcaaf4-af90-4ca5-9351-33c1e2c612cf-kube-api-access-md6d2\") pod \"kube-state-metrics-785778ff74-wrs2c\" (UID: \"b1dcaaf4-af90-4ca5-9351-33c1e2c612cf\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-wrs2c"
Jul 09 18:05:21.202810 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.202792    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfmr2\" (UniqueName: \"kubernetes.io/projected/6feeb09a-7eb5-4c46-a687-202ddba40804-kube-api-access-zfmr2\") pod \"node-exporter-zfclm\" (UID: \"6feeb09a-7eb5-4c46-a687-202ddba40804\") " pod="openshift-monitoring/node-exporter-zfclm"
Jul 09 18:05:21.241945 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.241913    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sf462" event={"ID":"52ba7983-15e4-437e-a86f-36f0048b0614","Type":"ContainerStarted","Data":"efcf9c4beeea131e704618f3893c0b0dbac94f5916faa4cb297386190eea7f3b"}
Jul 09 18:05:21.255528 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.255506    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6ng78"]
Jul 09 18:05:21.257733 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:05:21.257708    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcddd2f06_2abf_4246_ac09_acc38c065830.slice/crio-891526188cb035406fc554b09caffe63cbe55112baf64fe1d110d787758d66d4 WatchSource:0}: Error finding container 891526188cb035406fc554b09caffe63cbe55112baf64fe1d110d787758d66d4: Status 404 returned error can't find the container with id 891526188cb035406fc554b09caffe63cbe55112baf64fe1d110d787758d66d4
Jul 09 18:05:21.258072 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.258053    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wqgvh"]
Jul 09 18:05:21.262839 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:05:21.262815    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5de2134_56a4_4318_a22f_90937385dc5b.slice/crio-c70e621dd03dc4e4f7430541ffcff97d94c38d970caa43fd6e8f49321978f1d1 WatchSource:0}: Error finding container c70e621dd03dc4e4f7430541ffcff97d94c38d970caa43fd6e8f49321978f1d1: Status 404 returned error can't find the container with id c70e621dd03dc4e4f7430541ffcff97d94c38d970caa43fd6e8f49321978f1d1
Jul 09 18:05:21.271974 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.271951    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-wrs2c"
Jul 09 18:05:21.280712 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.280692    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zfclm"
Jul 09 18:05:21.290178 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:05:21.290129    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6feeb09a_7eb5_4c46_a687_202ddba40804.slice/crio-f3117ce14d2ddd56f66facb8d06e52e91988f5c6d54b1766b466b84fa7b46852 WatchSource:0}: Error finding container f3117ce14d2ddd56f66facb8d06e52e91988f5c6d54b1766b466b84fa7b46852: Status 404 returned error can't find the container with id f3117ce14d2ddd56f66facb8d06e52e91988f5c6d54b1766b466b84fa7b46852
Jul 09 18:05:21.431856 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.431764    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-wrs2c"]
Jul 09 18:05:21.438379 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:05:21.438351    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1dcaaf4_af90_4ca5_9351_33c1e2c612cf.slice/crio-2c5a95ed7a75d15a8d47b5960e4a7a4e6e7436d1ccec82fb1743e91ad62579a7 WatchSource:0}: Error finding container 2c5a95ed7a75d15a8d47b5960e4a7a4e6e7436d1ccec82fb1743e91ad62579a7: Status 404 returned error can't find the container with id 2c5a95ed7a75d15a8d47b5960e4a7a4e6e7436d1ccec82fb1743e91ad62579a7
Jul 09 18:05:21.583207 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.583169    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c61025f3-9d66-42c4-8f78-c9b828e113dc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-4t62b\" (UID: \"c61025f3-9d66-42c4-8f78-c9b828e113dc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-4t62b"
Jul 09 18:05:21.586393 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.586363    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c61025f3-9d66-42c4-8f78-c9b828e113dc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-4t62b\" (UID: \"c61025f3-9d66-42c4-8f78-c9b828e113dc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-4t62b"
Jul 09 18:05:21.801586 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.801302    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-4t62b"
Jul 09 18:05:21.983058 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:21.983019    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-4t62b"]
Jul 09 18:05:22.185402 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.184884    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 18:05:22.188758 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.188727    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.195088 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.195067    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 09 18:05:22.195918 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.195105    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 09 18:05:22.196240 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.195253    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 09 18:05:22.196513 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.195370    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-c8nvb\""
Jul 09 18:05:22.197393 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.195415    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 09 18:05:22.197393 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.195488    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 09 18:05:22.197393 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.195538    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 09 18:05:22.197393 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.195573    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 09 18:05:22.197393 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.195593    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 09 18:05:22.197393 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.195627    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 09 18:05:22.223117 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.220459    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 18:05:22.256996 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.256798    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zfclm" event={"ID":"6feeb09a-7eb5-4c46-a687-202ddba40804","Type":"ContainerStarted","Data":"f3117ce14d2ddd56f66facb8d06e52e91988f5c6d54b1766b466b84fa7b46852"}
Jul 09 18:05:22.261981 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.261953    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-wrs2c" event={"ID":"b1dcaaf4-af90-4ca5-9351-33c1e2c612cf","Type":"ContainerStarted","Data":"2c5a95ed7a75d15a8d47b5960e4a7a4e6e7436d1ccec82fb1743e91ad62579a7"}
Jul 09 18:05:22.263521 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.263496    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6ng78" event={"ID":"cddd2f06-2abf-4246-ac09-acc38c065830","Type":"ContainerStarted","Data":"891526188cb035406fc554b09caffe63cbe55112baf64fe1d110d787758d66d4"}
Jul 09 18:05:22.266544 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.266476    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wqgvh" event={"ID":"d5de2134-56a4-4318-a22f-90937385dc5b","Type":"ContainerStarted","Data":"c70e621dd03dc4e4f7430541ffcff97d94c38d970caa43fd6e8f49321978f1d1"}
Jul 09 18:05:22.288128 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.287586    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.288128 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.287634    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a243ec7-6930-4bff-a320-0ddb7037d639-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.288128 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.287681    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.288128 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.287707    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p48b\" (UniqueName: \"kubernetes.io/projected/4a243ec7-6930-4bff-a320-0ddb7037d639-kube-api-access-7p48b\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.288128 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.287741    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-config-volume\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.288128 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.287765    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4a243ec7-6930-4bff-a320-0ddb7037d639-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.288128 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.287795    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.288128 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.287840    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a243ec7-6930-4bff-a320-0ddb7037d639-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.288128 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.287879    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a243ec7-6930-4bff-a320-0ddb7037d639-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.288128 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.287912    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.288128 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.287949    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.288128 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.287978    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-web-config\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.288128 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.288006    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a243ec7-6930-4bff-a320-0ddb7037d639-config-out\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.308847 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:05:22.308809    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc61025f3_9d66_42c4_8f78_c9b828e113dc.slice/crio-2c7d41d65e4580243c48718b2952d72573063d896503e10d152e813c9dc751c0 WatchSource:0}: Error finding container 2c7d41d65e4580243c48718b2952d72573063d896503e10d152e813c9dc751c0: Status 404 returned error can't find the container with id 2c7d41d65e4580243c48718b2952d72573063d896503e10d152e813c9dc751c0
Jul 09 18:05:22.388674 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.388638    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.388816 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.388686    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.388816 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.388716    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-web-config\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.388816 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.388745    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a243ec7-6930-4bff-a320-0ddb7037d639-config-out\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.388816 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.388772    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.388816 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.388795    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a243ec7-6930-4bff-a320-0ddb7037d639-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.389057 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.388831    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.389057 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.388855    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p48b\" (UniqueName: \"kubernetes.io/projected/4a243ec7-6930-4bff-a320-0ddb7037d639-kube-api-access-7p48b\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.389057 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.388885    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-config-volume\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.389057 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.388905    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4a243ec7-6930-4bff-a320-0ddb7037d639-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.389057 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.388931    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.389057 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.388976    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a243ec7-6930-4bff-a320-0ddb7037d639-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.389057 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.389015    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a243ec7-6930-4bff-a320-0ddb7037d639-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.389780 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.389753    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a243ec7-6930-4bff-a320-0ddb7037d639-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.392287 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.390687    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4a243ec7-6930-4bff-a320-0ddb7037d639-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.394208 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.394158    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a243ec7-6930-4bff-a320-0ddb7037d639-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.395424 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.394697    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.397029 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.396997    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.398891 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.398773    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a243ec7-6930-4bff-a320-0ddb7037d639-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.399849 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.399762    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.400448 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.400423    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a243ec7-6930-4bff-a320-0ddb7037d639-config-out\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.400524 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.400459    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-config-volume\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.401715 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.401689    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.402309 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.402265    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.403746 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.403703    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-web-config\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.432122 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.432081    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p48b\" (UniqueName: \"kubernetes.io/projected/4a243ec7-6930-4bff-a320-0ddb7037d639-kube-api-access-7p48b\") pod \"alertmanager-main-0\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.510425 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.510390    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:05:22.720805 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:22.720726    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 18:05:23.274533 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:23.274363    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-4t62b" event={"ID":"c61025f3-9d66-42c4-8f78-c9b828e113dc","Type":"ContainerStarted","Data":"c3d470b87cc45fbc89ffa25dc651e22de70a3eea004ee0a3617090fff7cacd6d"}
Jul 09 18:05:23.274533 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:23.274442    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-4t62b" event={"ID":"c61025f3-9d66-42c4-8f78-c9b828e113dc","Type":"ContainerStarted","Data":"6f904b653a7afeebe5129e93c689d2e2a0f1f21adeaaccead2018ddeda55dab2"}
Jul 09 18:05:23.274533 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:23.274472    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-4t62b" event={"ID":"c61025f3-9d66-42c4-8f78-c9b828e113dc","Type":"ContainerStarted","Data":"2c7d41d65e4580243c48718b2952d72573063d896503e10d152e813c9dc751c0"}
Jul 09 18:05:23.276619 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:23.276591    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wqgvh" event={"ID":"d5de2134-56a4-4318-a22f-90937385dc5b","Type":"ContainerStarted","Data":"52f5cc71b455a6d6676b5583ea7e35260a08e6c51a8619e8edd0d70182f24ae5"}
Jul 09 18:05:23.276749 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:23.276646    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wqgvh" event={"ID":"d5de2134-56a4-4318-a22f-90937385dc5b","Type":"ContainerStarted","Data":"b2c72a1129de9266dad34832cca9992a9fadbee2afc9a36a86e2bf6090d19a2a"}
Jul 09 18:05:23.278554 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:23.278485    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a243ec7-6930-4bff-a320-0ddb7037d639","Type":"ContainerStarted","Data":"29e7d43696af8ee5df39eae37209a826af734531b01b3d47c2749ed0772fa54e"}
Jul 09 18:05:23.301341 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:23.301116    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wqgvh" podStartSLOduration=67.217066201 podStartE2EDuration="1m8.301094994s" totalImagesPullingTime="1.084028793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:23.300098329 +0000 UTC m=+68.969179306" watchObservedRunningTime="2026-07-09 18:05:23.301094994 +0000 UTC m=+68.970175940"
Jul 09 18:05:24.006491 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.006422    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6f8c49d449-vp9xh"]
Jul 09 18:05:24.031260 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.031080    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh"
Jul 09 18:05:24.036819 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.036556    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 09 18:05:24.036819 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.036576    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 09 18:05:24.036819 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.036608    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 09 18:05:24.036819 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.036789    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-8clqx\""
Jul 09 18:05:24.037132 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.036874    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 09 18:05:24.037455 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.037433    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-1ce84ovpkm7l9\""
Jul 09 18:05:24.037634 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.037613    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 09 18:05:24.052809 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.052781    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6f8c49d449-vp9xh"]
Jul 09 18:05:24.104374 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.104202    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06ca3e54-f63b-4b0b-9901-76c96741055d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f8c49d449-vp9xh\" (UID: \"06ca3e54-f63b-4b0b-9901-76c96741055d\") " pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh"
Jul 09 18:05:24.104374 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.104269    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/06ca3e54-f63b-4b0b-9901-76c96741055d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f8c49d449-vp9xh\" (UID: \"06ca3e54-f63b-4b0b-9901-76c96741055d\") " pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh"
Jul 09 18:05:24.104374 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.104306    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06ca3e54-f63b-4b0b-9901-76c96741055d-metrics-client-ca\") pod \"thanos-querier-6f8c49d449-vp9xh\" (UID: \"06ca3e54-f63b-4b0b-9901-76c96741055d\") " pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh"
Jul 09 18:05:24.104374 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.104339    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06ca3e54-f63b-4b0b-9901-76c96741055d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f8c49d449-vp9xh\" (UID: \"06ca3e54-f63b-4b0b-9901-76c96741055d\") " pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh"
Jul 09 18:05:24.104706 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.104495    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8kb6\" (UniqueName: \"kubernetes.io/projected/06ca3e54-f63b-4b0b-9901-76c96741055d-kube-api-access-n8kb6\") pod \"thanos-querier-6f8c49d449-vp9xh\" (UID: \"06ca3e54-f63b-4b0b-9901-76c96741055d\") " pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh"
Jul 09 18:05:24.104706 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.104539    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/06ca3e54-f63b-4b0b-9901-76c96741055d-secret-grpc-tls\") pod \"thanos-querier-6f8c49d449-vp9xh\" (UID: \"06ca3e54-f63b-4b0b-9901-76c96741055d\") " pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh"
Jul 09 18:05:24.104706 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.104581    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/06ca3e54-f63b-4b0b-9901-76c96741055d-secret-thanos-querier-tls\") pod \"thanos-querier-6f8c49d449-vp9xh\" (UID: \"06ca3e54-f63b-4b0b-9901-76c96741055d\") " pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh"
Jul 09 18:05:24.104706 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.104616    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/06ca3e54-f63b-4b0b-9901-76c96741055d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f8c49d449-vp9xh\" (UID: \"06ca3e54-f63b-4b0b-9901-76c96741055d\") " pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh"
Jul 09 18:05:24.205994 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.205479    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06ca3e54-f63b-4b0b-9901-76c96741055d-metrics-client-ca\") pod \"thanos-querier-6f8c49d449-vp9xh\" (UID: \"06ca3e54-f63b-4b0b-9901-76c96741055d\") " pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh"
Jul 09 18:05:24.205994 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.205563    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06ca3e54-f63b-4b0b-9901-76c96741055d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f8c49d449-vp9xh\" (UID: \"06ca3e54-f63b-4b0b-9901-76c96741055d\") " pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh"
Jul 09 18:05:24.205994 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.205598    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8kb6\" (UniqueName: \"kubernetes.io/projected/06ca3e54-f63b-4b0b-9901-76c96741055d-kube-api-access-n8kb6\") pod \"thanos-querier-6f8c49d449-vp9xh\" (UID: \"06ca3e54-f63b-4b0b-9901-76c96741055d\") " pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh"
Jul 09 18:05:24.205994 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.205630    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/06ca3e54-f63b-4b0b-9901-76c96741055d-secret-grpc-tls\") pod \"thanos-querier-6f8c49d449-vp9xh\" (UID: \"06ca3e54-f63b-4b0b-9901-76c96741055d\") " pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh"
Jul 09 18:05:24.205994 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.205671    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/06ca3e54-f63b-4b0b-9901-76c96741055d-secret-thanos-querier-tls\") pod \"thanos-querier-6f8c49d449-vp9xh\" (UID: \"06ca3e54-f63b-4b0b-9901-76c96741055d\") " pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh"
Jul 09 18:05:24.205994 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.205719    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/06ca3e54-f63b-4b0b-9901-76c96741055d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f8c49d449-vp9xh\" (UID: \"06ca3e54-f63b-4b0b-9901-76c96741055d\") " pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh"
Jul 09 18:05:24.205994 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.205771    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06ca3e54-f63b-4b0b-9901-76c96741055d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f8c49d449-vp9xh\" (UID: \"06ca3e54-f63b-4b0b-9901-76c96741055d\") " pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh"
Jul 09 18:05:24.205994 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.205799    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/06ca3e54-f63b-4b0b-9901-76c96741055d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f8c49d449-vp9xh\" (UID: \"06ca3e54-f63b-4b0b-9901-76c96741055d\") " pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh"
Jul 09 18:05:24.208121 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.208070    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06ca3e54-f63b-4b0b-9901-76c96741055d-metrics-client-ca\") pod \"thanos-querier-6f8c49d449-vp9xh\" (UID: \"06ca3e54-f63b-4b0b-9901-76c96741055d\") " pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh"
Jul 09 18:05:24.210258 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.210201    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/06ca3e54-f63b-4b0b-9901-76c96741055d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f8c49d449-vp9xh\" (UID: \"06ca3e54-f63b-4b0b-9901-76c96741055d\") " pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh"
Jul 09 18:05:24.210490 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.210463    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06ca3e54-f63b-4b0b-9901-76c96741055d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f8c49d449-vp9xh\" (UID: \"06ca3e54-f63b-4b0b-9901-76c96741055d\") " pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh"
Jul 09 18:05:24.211241 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.211181    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/06ca3e54-f63b-4b0b-9901-76c96741055d-secret-grpc-tls\") pod \"thanos-querier-6f8c49d449-vp9xh\" (UID: \"06ca3e54-f63b-4b0b-9901-76c96741055d\") " pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh"
Jul 09 18:05:24.212761 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.212682    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/06ca3e54-f63b-4b0b-9901-76c96741055d-secret-thanos-querier-tls\") pod \"thanos-querier-6f8c49d449-vp9xh\" (UID: \"06ca3e54-f63b-4b0b-9901-76c96741055d\") " pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh"
Jul 09 18:05:24.213368 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.213326    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/06ca3e54-f63b-4b0b-9901-76c96741055d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f8c49d449-vp9xh\" (UID: \"06ca3e54-f63b-4b0b-9901-76c96741055d\") " pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh"
Jul 09 18:05:24.213644 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.213617    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06ca3e54-f63b-4b0b-9901-76c96741055d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f8c49d449-vp9xh\" (UID: \"06ca3e54-f63b-4b0b-9901-76c96741055d\") " pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh"
Jul 09 18:05:24.220891 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.219719    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pnvtm"
Jul 09 18:05:24.228350 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.228305    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8kb6\" (UniqueName: \"kubernetes.io/projected/06ca3e54-f63b-4b0b-9901-76c96741055d-kube-api-access-n8kb6\") pod \"thanos-querier-6f8c49d449-vp9xh\" (UID: \"06ca3e54-f63b-4b0b-9901-76c96741055d\") " pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh"
Jul 09 18:05:24.344252 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:24.344155    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh"
Jul 09 18:05:25.435436 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.434099    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-578687bf9d-r7prf"]
Jul 09 18:05:25.436933 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.436909    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-578687bf9d-r7prf"
Jul 09 18:05:25.439689 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.439667    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 09 18:05:25.439689 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.439686    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 09 18:05:25.440695 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.440666    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 09 18:05:25.440695 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.440673    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dd8o9j6q55dtt\""
Jul 09 18:05:25.440862 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.440697    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-bt8fn\""
Jul 09 18:05:25.440972 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.440957    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 09 18:05:25.455274 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.455249    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-578687bf9d-r7prf"]
Jul 09 18:05:25.519044 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.519004    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cee30c90-1586-4847-a689-7c71a4fb2a5e-metrics-server-audit-profiles\") pod \"metrics-server-578687bf9d-r7prf\" (UID: \"cee30c90-1586-4847-a689-7c71a4fb2a5e\") " pod="openshift-monitoring/metrics-server-578687bf9d-r7prf"
Jul 09 18:05:25.519247 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.519059    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cee30c90-1586-4847-a689-7c71a4fb2a5e-audit-log\") pod \"metrics-server-578687bf9d-r7prf\" (UID: \"cee30c90-1586-4847-a689-7c71a4fb2a5e\") " pod="openshift-monitoring/metrics-server-578687bf9d-r7prf"
Jul 09 18:05:25.519247 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.519083    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cee30c90-1586-4847-a689-7c71a4fb2a5e-client-ca-bundle\") pod \"metrics-server-578687bf9d-r7prf\" (UID: \"cee30c90-1586-4847-a689-7c71a4fb2a5e\") " pod="openshift-monitoring/metrics-server-578687bf9d-r7prf"
Jul 09 18:05:25.519247 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.519112    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hzj4\" (UniqueName: \"kubernetes.io/projected/cee30c90-1586-4847-a689-7c71a4fb2a5e-kube-api-access-4hzj4\") pod \"metrics-server-578687bf9d-r7prf\" (UID: \"cee30c90-1586-4847-a689-7c71a4fb2a5e\") " pod="openshift-monitoring/metrics-server-578687bf9d-r7prf"
Jul 09 18:05:25.519247 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.519142    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cee30c90-1586-4847-a689-7c71a4fb2a5e-secret-metrics-server-tls\") pod \"metrics-server-578687bf9d-r7prf\" (UID: \"cee30c90-1586-4847-a689-7c71a4fb2a5e\") " pod="openshift-monitoring/metrics-server-578687bf9d-r7prf"
Jul 09 18:05:25.519247 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.519169    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cee30c90-1586-4847-a689-7c71a4fb2a5e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-578687bf9d-r7prf\" (UID: \"cee30c90-1586-4847-a689-7c71a4fb2a5e\") " pod="openshift-monitoring/metrics-server-578687bf9d-r7prf"
Jul 09 18:05:25.519247 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.519244    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cee30c90-1586-4847-a689-7c71a4fb2a5e-secret-metrics-server-client-certs\") pod \"metrics-server-578687bf9d-r7prf\" (UID: \"cee30c90-1586-4847-a689-7c71a4fb2a5e\") " pod="openshift-monitoring/metrics-server-578687bf9d-r7prf"
Jul 09 18:05:25.575526 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.575488    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-98rsk"]
Jul 09 18:05:25.577791 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.577769    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-98rsk"
Jul 09 18:05:25.580714 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.580689    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-lbnbf\""
Jul 09 18:05:25.580955 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.580938    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 09 18:05:25.592260 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.592182    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-98rsk"]
Jul 09 18:05:25.620655 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.620620    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cld4\" (UniqueName: \"kubernetes.io/projected/0049e43b-54aa-410c-9971-d89ad2d50563-kube-api-access-4cld4\") pod \"monitoring-plugin-5f68df7f58-98rsk\" (UID: \"0049e43b-54aa-410c-9971-d89ad2d50563\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-98rsk"
Jul 09 18:05:25.620804 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.620672    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cee30c90-1586-4847-a689-7c71a4fb2a5e-secret-metrics-server-client-certs\") pod \"metrics-server-578687bf9d-r7prf\" (UID: \"cee30c90-1586-4847-a689-7c71a4fb2a5e\") " pod="openshift-monitoring/metrics-server-578687bf9d-r7prf"
Jul 09 18:05:25.620804 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.620771    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cee30c90-1586-4847-a689-7c71a4fb2a5e-metrics-server-audit-profiles\") pod \"metrics-server-578687bf9d-r7prf\" (UID: \"cee30c90-1586-4847-a689-7c71a4fb2a5e\") " pod="openshift-monitoring/metrics-server-578687bf9d-r7prf"
Jul 09 18:05:25.620901 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.620824    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cee30c90-1586-4847-a689-7c71a4fb2a5e-audit-log\") pod \"metrics-server-578687bf9d-r7prf\" (UID: \"cee30c90-1586-4847-a689-7c71a4fb2a5e\") " pod="openshift-monitoring/metrics-server-578687bf9d-r7prf"
Jul 09 18:05:25.620901 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.620851    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cee30c90-1586-4847-a689-7c71a4fb2a5e-client-ca-bundle\") pod \"metrics-server-578687bf9d-r7prf\" (UID: \"cee30c90-1586-4847-a689-7c71a4fb2a5e\") " pod="openshift-monitoring/metrics-server-578687bf9d-r7prf"
Jul 09 18:05:25.620991 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.620897    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hzj4\" (UniqueName: \"kubernetes.io/projected/cee30c90-1586-4847-a689-7c71a4fb2a5e-kube-api-access-4hzj4\") pod \"metrics-server-578687bf9d-r7prf\" (UID: \"cee30c90-1586-4847-a689-7c71a4fb2a5e\") " pod="openshift-monitoring/metrics-server-578687bf9d-r7prf"
Jul 09 18:05:25.620991 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.620931    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0049e43b-54aa-410c-9971-d89ad2d50563-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-98rsk\" (UID: \"0049e43b-54aa-410c-9971-d89ad2d50563\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-98rsk"
Jul 09 18:05:25.620991 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.620962    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cee30c90-1586-4847-a689-7c71a4fb2a5e-secret-metrics-server-tls\") pod \"metrics-server-578687bf9d-r7prf\" (UID: \"cee30c90-1586-4847-a689-7c71a4fb2a5e\") " pod="openshift-monitoring/metrics-server-578687bf9d-r7prf"
Jul 09 18:05:25.621140 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.620991    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cee30c90-1586-4847-a689-7c71a4fb2a5e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-578687bf9d-r7prf\" (UID: \"cee30c90-1586-4847-a689-7c71a4fb2a5e\") " pod="openshift-monitoring/metrics-server-578687bf9d-r7prf"
Jul 09 18:05:25.621653 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.621630    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cee30c90-1586-4847-a689-7c71a4fb2a5e-audit-log\") pod \"metrics-server-578687bf9d-r7prf\" (UID: \"cee30c90-1586-4847-a689-7c71a4fb2a5e\") " pod="openshift-monitoring/metrics-server-578687bf9d-r7prf"
Jul 09 18:05:25.621751 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.621658    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cee30c90-1586-4847-a689-7c71a4fb2a5e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-578687bf9d-r7prf\" (UID: \"cee30c90-1586-4847-a689-7c71a4fb2a5e\") " pod="openshift-monitoring/metrics-server-578687bf9d-r7prf"
Jul 09 18:05:25.622086 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.622060    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cee30c90-1586-4847-a689-7c71a4fb2a5e-metrics-server-audit-profiles\") pod \"metrics-server-578687bf9d-r7prf\" (UID: \"cee30c90-1586-4847-a689-7c71a4fb2a5e\") " pod="openshift-monitoring/metrics-server-578687bf9d-r7prf"
Jul 09 18:05:25.623527 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.623503    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cee30c90-1586-4847-a689-7c71a4fb2a5e-secret-metrics-server-client-certs\") pod \"metrics-server-578687bf9d-r7prf\" (UID: \"cee30c90-1586-4847-a689-7c71a4fb2a5e\") " pod="openshift-monitoring/metrics-server-578687bf9d-r7prf"
Jul 09 18:05:25.624136 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.624116    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cee30c90-1586-4847-a689-7c71a4fb2a5e-secret-metrics-server-tls\") pod \"metrics-server-578687bf9d-r7prf\" (UID: \"cee30c90-1586-4847-a689-7c71a4fb2a5e\") " pod="openshift-monitoring/metrics-server-578687bf9d-r7prf"
Jul 09 18:05:25.624326 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.624297    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cee30c90-1586-4847-a689-7c71a4fb2a5e-client-ca-bundle\") pod \"metrics-server-578687bf9d-r7prf\" (UID: \"cee30c90-1586-4847-a689-7c71a4fb2a5e\") " pod="openshift-monitoring/metrics-server-578687bf9d-r7prf"
Jul 09 18:05:25.629873 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.629852    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hzj4\" (UniqueName: \"kubernetes.io/projected/cee30c90-1586-4847-a689-7c71a4fb2a5e-kube-api-access-4hzj4\") pod \"metrics-server-578687bf9d-r7prf\" (UID: \"cee30c90-1586-4847-a689-7c71a4fb2a5e\") " pod="openshift-monitoring/metrics-server-578687bf9d-r7prf"
Jul 09 18:05:25.722263 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.722163    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0049e43b-54aa-410c-9971-d89ad2d50563-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-98rsk\" (UID: \"0049e43b-54aa-410c-9971-d89ad2d50563\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-98rsk"
Jul 09 18:05:25.722263 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.722240    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cld4\" (UniqueName: \"kubernetes.io/projected/0049e43b-54aa-410c-9971-d89ad2d50563-kube-api-access-4cld4\") pod \"monitoring-plugin-5f68df7f58-98rsk\" (UID: \"0049e43b-54aa-410c-9971-d89ad2d50563\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-98rsk"
Jul 09 18:05:25.722473 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:05:25.722330    2563 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 09 18:05:25.722473 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:05:25.722405    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0049e43b-54aa-410c-9971-d89ad2d50563-monitoring-plugin-cert podName:0049e43b-54aa-410c-9971-d89ad2d50563 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:26.222384669 +0000 UTC m=+71.891465602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/0049e43b-54aa-410c-9971-d89ad2d50563-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-98rsk" (UID: "0049e43b-54aa-410c-9971-d89ad2d50563") : secret "monitoring-plugin-cert" not found
Jul 09 18:05:25.731951 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.731921    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cld4\" (UniqueName: \"kubernetes.io/projected/0049e43b-54aa-410c-9971-d89ad2d50563-kube-api-access-4cld4\") pod \"monitoring-plugin-5f68df7f58-98rsk\" (UID: \"0049e43b-54aa-410c-9971-d89ad2d50563\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-98rsk"
Jul 09 18:05:25.748819 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:25.748777    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-578687bf9d-r7prf"
Jul 09 18:05:26.226196 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:26.226149    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0049e43b-54aa-410c-9971-d89ad2d50563-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-98rsk\" (UID: \"0049e43b-54aa-410c-9971-d89ad2d50563\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-98rsk"
Jul 09 18:05:26.228974 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:26.228930    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0049e43b-54aa-410c-9971-d89ad2d50563-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-98rsk\" (UID: \"0049e43b-54aa-410c-9971-d89ad2d50563\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-98rsk"
Jul 09 18:05:26.487612 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:26.487526    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6f8c49d449-vp9xh"]
Jul 09 18:05:26.488045 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:26.487636    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-98rsk"
Jul 09 18:05:26.500763 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:26.500741    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-578687bf9d-r7prf"]
Jul 09 18:05:26.595904 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:05:26.595867    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ca3e54_f63b_4b0b_9901_76c96741055d.slice/crio-0b2c940f86a7a5a81d07178e52990e2e94dbf4da22c30da4fd71ba6ee89be891 WatchSource:0}: Error finding container 0b2c940f86a7a5a81d07178e52990e2e94dbf4da22c30da4fd71ba6ee89be891: Status 404 returned error can't find the container with id 0b2c940f86a7a5a81d07178e52990e2e94dbf4da22c30da4fd71ba6ee89be891
Jul 09 18:05:26.596590 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:05:26.596475    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee30c90_1586_4847_a689_7c71a4fb2a5e.slice/crio-da1727a548500f5006e6109bdc8aa5a29ce03e14818236cbcb8169a60a4d9e1f WatchSource:0}: Error finding container da1727a548500f5006e6109bdc8aa5a29ce03e14818236cbcb8169a60a4d9e1f: Status 404 returned error can't find the container with id da1727a548500f5006e6109bdc8aa5a29ce03e14818236cbcb8169a60a4d9e1f
Jul 09 18:05:27.158926 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.158892    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-98rsk"]
Jul 09 18:05:27.168477 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:05:27.168446    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0049e43b_54aa_410c_9971_d89ad2d50563.slice/crio-2f92cd59d932fb30cfc2cde62144a9538111e176cef4cd56f093a44cba8ffcdd WatchSource:0}: Error finding container 2f92cd59d932fb30cfc2cde62144a9538111e176cef4cd56f093a44cba8ffcdd: Status 404 returned error can't find the container with id 2f92cd59d932fb30cfc2cde62144a9538111e176cef4cd56f093a44cba8ffcdd
Jul 09 18:05:27.298037 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.297998    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-98rsk" event={"ID":"0049e43b-54aa-410c-9971-d89ad2d50563","Type":"ContainerStarted","Data":"2f92cd59d932fb30cfc2cde62144a9538111e176cef4cd56f093a44cba8ffcdd"}
Jul 09 18:05:27.300889 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.300827    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sf462" event={"ID":"52ba7983-15e4-437e-a86f-36f0048b0614","Type":"ContainerStarted","Data":"18b8338ddef888ea9fb7eace94f48ec1b2d7b4f006f7239e65caf18d469afd20"}
Jul 09 18:05:27.301039 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.300938    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:05:27.303550 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.303524    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-wrs2c" event={"ID":"b1dcaaf4-af90-4ca5-9351-33c1e2c612cf","Type":"ContainerStarted","Data":"d1fbfe1f9788b86f618996cc2301a87b3af574178a4e88398d3d48957565f857"}
Jul 09 18:05:27.303668 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.303574    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-wrs2c" event={"ID":"b1dcaaf4-af90-4ca5-9351-33c1e2c612cf","Type":"ContainerStarted","Data":"793d0251e2529a83b0703e92fea4d57a7e030e73a7ecbb93bd1a3c7213edcf8f"}
Jul 09 18:05:27.303668 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.303604    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-wrs2c" event={"ID":"b1dcaaf4-af90-4ca5-9351-33c1e2c612cf","Type":"ContainerStarted","Data":"98dfe94b3ea9191e6d4cf5c744c329c3c2a3839c381703ee7dbab4ecb46a1cda"}
Jul 09 18:05:27.305292 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.305178    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6ng78" event={"ID":"cddd2f06-2abf-4246-ac09-acc38c065830","Type":"ContainerStarted","Data":"7848138532a7926509075c661e9b4588b5423cd7c34f2223f360a1448e7c5eb6"}
Jul 09 18:05:27.307648 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.307627    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-4t62b" event={"ID":"c61025f3-9d66-42c4-8f78-c9b828e113dc","Type":"ContainerStarted","Data":"f836619beabe9df7cdd336f0e808a97959789e5a867cf5dfb1ae0ba674095dad"}
Jul 09 18:05:27.308981 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.308927    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh" event={"ID":"06ca3e54-f63b-4b0b-9901-76c96741055d","Type":"ContainerStarted","Data":"0b2c940f86a7a5a81d07178e52990e2e94dbf4da22c30da4fd71ba6ee89be891"}
Jul 09 18:05:27.310524 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.310501    2563 generic.go:356] "Generic (PLEG): container finished" podID="6feeb09a-7eb5-4c46-a687-202ddba40804" containerID="2c24abdd7b1901a324592439b8b599cdf7458e4e96a2568c8507563c1e30cb76" exitCode=0
Jul 09 18:05:27.310749 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.310718    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zfclm" event={"ID":"6feeb09a-7eb5-4c46-a687-202ddba40804","Type":"ContainerDied","Data":"2c24abdd7b1901a324592439b8b599cdf7458e4e96a2568c8507563c1e30cb76"}
Jul 09 18:05:27.312504 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.312483    2563 generic.go:356] "Generic (PLEG): container finished" podID="4a243ec7-6930-4bff-a320-0ddb7037d639" containerID="789d0472b35001148c9bf4da083d3897990a241db9534e1f5cf765feefda9a65" exitCode=0
Jul 09 18:05:27.312608 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.312562    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a243ec7-6930-4bff-a320-0ddb7037d639","Type":"ContainerDied","Data":"789d0472b35001148c9bf4da083d3897990a241db9534e1f5cf765feefda9a65"}
Jul 09 18:05:27.314513 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.314481    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-578687bf9d-r7prf" event={"ID":"cee30c90-1586-4847-a689-7c71a4fb2a5e","Type":"ContainerStarted","Data":"da1727a548500f5006e6109bdc8aa5a29ce03e14818236cbcb8169a60a4d9e1f"}
Jul 09 18:05:27.364337 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.364252    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-sf462" podStartSLOduration=66.52212647 podStartE2EDuration="1m12.364232147s" totalImagesPullingTime="5.842105677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:27.363042531 +0000 UTC m=+73.032123488" watchObservedRunningTime="2026-07-09 18:05:27.364232147 +0000 UTC m=+73.033313093"
Jul 09 18:05:27.481525 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.481473    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6ng78" podStartSLOduration=66.856199179 podStartE2EDuration="1m12.481447157s" totalImagesPullingTime="5.625247978s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:27.440252275 +0000 UTC m=+73.109333220" watchObservedRunningTime="2026-07-09 18:05:27.481447157 +0000 UTC m=+73.150528089"
Jul 09 18:05:27.524302 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.524247    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-wrs2c" podStartSLOduration=2.629937144 podStartE2EDuration="7.524204467s" totalImagesPullingTime="4.894267323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:27.523743275 +0000 UTC m=+73.192824230" watchObservedRunningTime="2026-07-09 18:05:27.524204467 +0000 UTC m=+73.193285415"
Jul 09 18:05:27.646753 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.646649    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-4t62b" podStartSLOduration=3.285606001 podStartE2EDuration="7.646633217s" totalImagesPullingTime="4.361027216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:27.594567363 +0000 UTC m=+73.263648316" watchObservedRunningTime="2026-07-09 18:05:27.646633217 +0000 UTC m=+73.315714170"
Jul 09 18:05:27.656348 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.656319    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 18:05:27.661059 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.661018    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.670020 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.669996    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 09 18:05:27.670123 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.670108    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 09 18:05:27.671007 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.670335    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 09 18:05:27.671007 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.670480    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 09 18:05:27.671007 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.670514    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 09 18:05:27.671007 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.670692    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 09 18:05:27.671007 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.670869    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 09 18:05:27.671007 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.670966    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 09 18:05:27.671440 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.671377    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 09 18:05:27.671755 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.671444    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9hj8rkhi5mk78\""
Jul 09 18:05:27.672001 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.671491    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 09 18:05:27.672332 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.671535    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 09 18:05:27.672439 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.672422    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 09 18:05:27.672509 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.672444    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-g5ggg\""
Jul 09 18:05:27.673606 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.673588    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 09 18:05:27.681563 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.681540    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 09 18:05:27.704031 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.703990    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 18:05:27.739577 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.739535    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a823a0e1-a248-468c-8bbf-027a46435ceb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.739763 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.739595    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.739763 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.739622    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.739763 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.739674    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-web-config\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.739763 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.739706    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.739763 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.739737    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.739763 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.739760    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.740051 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.739787    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.740051 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.739815    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a823a0e1-a248-468c-8bbf-027a46435ceb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.740051 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.739844    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.740051 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.739868    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7cb9\" (UniqueName: \"kubernetes.io/projected/a823a0e1-a248-468c-8bbf-027a46435ceb-kube-api-access-c7cb9\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.740051 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.739970    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a823a0e1-a248-468c-8bbf-027a46435ceb-config-out\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.740051 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.740017    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.740355 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.740072    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.740355 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.740105    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.740355 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.740160    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-config\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.740355 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.740196    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.740355 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.740258    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.740355 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.740288    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.740355 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.740303    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.843164 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.841814    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.843164 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.841866    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.843164 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.841905    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-config\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.843164 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.841933    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.843164 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.841958    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.843164 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.841984    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.843164 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.842006    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.843164 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.842034    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a823a0e1-a248-468c-8bbf-027a46435ceb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.843164 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.842068    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.843164 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.842109    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.843164 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.842132    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-web-config\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.843164 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.842163    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.843164 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.842190    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.843164 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.842229    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.843164 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.842259    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.843164 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.842282    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a823a0e1-a248-468c-8bbf-027a46435ceb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.843164 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.842308    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.844129 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.842338    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7cb9\" (UniqueName: \"kubernetes.io/projected/a823a0e1-a248-468c-8bbf-027a46435ceb-kube-api-access-c7cb9\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.844129 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.842370    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a823a0e1-a248-468c-8bbf-027a46435ceb-config-out\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.844129 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.842407    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.844129 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.842896    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.844129 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.843681    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.844129 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.843760    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.845818 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.845190    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.847063 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.847037    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.848243 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.847307    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a823a0e1-a248-468c-8bbf-027a46435ceb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.848243 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.847954    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.848653 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.848493    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.849333 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.848875    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-config\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.854528 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.854489    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.859014 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.858969    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a823a0e1-a248-468c-8bbf-027a46435ceb-config-out\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.860756 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.860736    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.868816 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.863986    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-web-config\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.868816 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.864175    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.868816 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.864697    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.868816 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.865031    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a823a0e1-a248-468c-8bbf-027a46435ceb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.868816 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.865783    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.868816 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.866866    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7cb9\" (UniqueName: \"kubernetes.io/projected/a823a0e1-a248-468c-8bbf-027a46435ceb-kube-api-access-c7cb9\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.869627 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.869585    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.870618 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.870580    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:27.974796 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:27.974503    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:28.202380 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:28.202244    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 18:05:28.207414 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:05:28.207377    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda823a0e1_a248_468c_8bbf_027a46435ceb.slice/crio-1f684464b0d961bb1b187b572ac473f65f5f9808877abbf5a454fabd5484dd67 WatchSource:0}: Error finding container 1f684464b0d961bb1b187b572ac473f65f5f9808877abbf5a454fabd5484dd67: Status 404 returned error can't find the container with id 1f684464b0d961bb1b187b572ac473f65f5f9808877abbf5a454fabd5484dd67
Jul 09 18:05:28.324754 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:28.324604    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zfclm" event={"ID":"6feeb09a-7eb5-4c46-a687-202ddba40804","Type":"ContainerStarted","Data":"b18e7646ddf180335a5e85a7457ff1e417f395db88574e3d3be86148398a2de5"}
Jul 09 18:05:28.324754 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:28.324669    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zfclm" event={"ID":"6feeb09a-7eb5-4c46-a687-202ddba40804","Type":"ContainerStarted","Data":"5d96b67a99bc29fb4192e120485f5788e54e53c205f31e5711663f65f66a78e7"}
Jul 09 18:05:28.329474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:28.329315    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a823a0e1-a248-468c-8bbf-027a46435ceb","Type":"ContainerStarted","Data":"9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233"}
Jul 09 18:05:28.329474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:28.329410    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a823a0e1-a248-468c-8bbf-027a46435ceb","Type":"ContainerStarted","Data":"1f684464b0d961bb1b187b572ac473f65f5f9808877abbf5a454fabd5484dd67"}
Jul 09 18:05:28.412073 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:28.411919    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zfclm" podStartSLOduration=3.283478458 podStartE2EDuration="8.411894997s" totalImagesPullingTime="5.047760032s" totalInitContainerRuntime="80.656507ms" isStatefulPod=true podCreationTimestamp="2026-07-09 18:05:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:28.364815477 +0000 UTC m=+74.033896429" watchObservedRunningTime="2026-07-09 18:05:28.411894997 +0000 UTC m=+74.080975944"
Jul 09 18:05:29.335002 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:29.334972    2563 generic.go:356] "Generic (PLEG): container finished" podID="a823a0e1-a248-468c-8bbf-027a46435ceb" containerID="9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233" exitCode=0
Jul 09 18:05:29.335395 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:29.335028    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a823a0e1-a248-468c-8bbf-027a46435ceb","Type":"ContainerDied","Data":"9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233"}
Jul 09 18:05:31.346686 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:31.346655    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh" event={"ID":"06ca3e54-f63b-4b0b-9901-76c96741055d","Type":"ContainerStarted","Data":"bac4c5c8e7a0c9a429145be0c2e1843a8d34e81b6ab22049666e4099f6907ab2"}
Jul 09 18:05:31.347145 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:31.346723    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh" event={"ID":"06ca3e54-f63b-4b0b-9901-76c96741055d","Type":"ContainerStarted","Data":"9aaab9d19e19c50294232cd612c81a809f31d1b777335e54f843ad1e4ec1de71"}
Jul 09 18:05:31.347145 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:31.346795    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh" event={"ID":"06ca3e54-f63b-4b0b-9901-76c96741055d","Type":"ContainerStarted","Data":"914cea6d416990abab31bf0d702f9577cc4659a9fa3b5d7bd62decf0ba2a227b"}
Jul 09 18:05:31.349605 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:31.349580    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a243ec7-6930-4bff-a320-0ddb7037d639","Type":"ContainerStarted","Data":"7f19b2d0090355bce6dad6f393e102b54e48e02af168f1dd82238fd6430e29df"}
Jul 09 18:05:31.349741 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:31.349642    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a243ec7-6930-4bff-a320-0ddb7037d639","Type":"ContainerStarted","Data":"0b9cd0fdc2448d85def1ed3cb18d3cfdff5f57152bc4edd0db489c170a733c6c"}
Jul 09 18:05:31.349741 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:31.349680    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a243ec7-6930-4bff-a320-0ddb7037d639","Type":"ContainerStarted","Data":"a0e440970b4dacaa6d7c7bf28bb41c4a89f94a313f637e1a14dfd361e5de6aa4"}
Jul 09 18:05:31.349741 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:31.349718    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a243ec7-6930-4bff-a320-0ddb7037d639","Type":"ContainerStarted","Data":"db73e791e536291d320e60782162f055bcaa6c3e56f228c65667e8a3c5ae866b"}
Jul 09 18:05:31.349741 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:31.349741    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a243ec7-6930-4bff-a320-0ddb7037d639","Type":"ContainerStarted","Data":"54f34503cf00c5679dd7d5e3214c503ca125c25299f90223856231e85b243b88"}
Jul 09 18:05:31.351209 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:31.351181    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-578687bf9d-r7prf" event={"ID":"cee30c90-1586-4847-a689-7c71a4fb2a5e","Type":"ContainerStarted","Data":"2c7d7db8a13b0eb5fc393065e4f4f93da7a6efdea7b35d1e54aa4a399e24fae7"}
Jul 09 18:05:31.352567 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:31.352546    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-98rsk" event={"ID":"0049e43b-54aa-410c-9971-d89ad2d50563","Type":"ContainerStarted","Data":"27e2670cf368de2b0f89cbdd332ee509bf20542d8fa3feb52c411d7eac3d9a4b"}
Jul 09 18:05:31.352750 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:31.352734    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-98rsk"
Jul 09 18:05:31.357477 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:31.357458    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-98rsk"
Jul 09 18:05:31.371287 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:31.371240    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-578687bf9d-r7prf" podStartSLOduration=2.727083225 podStartE2EDuration="6.371202963s" totalImagesPullingTime="3.644119738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:31.371137035 +0000 UTC m=+77.040217991" watchObservedRunningTime="2026-07-09 18:05:31.371202963 +0000 UTC m=+77.040283918"
Jul 09 18:05:31.391989 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:31.391917    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-98rsk" podStartSLOduration=3.316327739 podStartE2EDuration="6.391895574s" totalImagesPullingTime="3.075567835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:31.389792465 +0000 UTC m=+77.058873419" watchObservedRunningTime="2026-07-09 18:05:31.391895574 +0000 UTC m=+77.060976529"
Jul 09 18:05:32.360454 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:32.360365    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh" event={"ID":"06ca3e54-f63b-4b0b-9901-76c96741055d","Type":"ContainerStarted","Data":"aa44b63704a479acce0ed86f9981f0f0d4020a68ccbb504472403bcc56a82961"}
Jul 09 18:05:32.360861 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:32.360478    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh" event={"ID":"06ca3e54-f63b-4b0b-9901-76c96741055d","Type":"ContainerStarted","Data":"1c336ce831f14654d1594be53d96606691c6d474a980ea2f4671cc1422aba68b"}
Jul 09 18:05:32.360861 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:32.360518    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh" event={"ID":"06ca3e54-f63b-4b0b-9901-76c96741055d","Type":"ContainerStarted","Data":"e13a4d24639b1327df976dd4544f78a0622286ab60fa530ed02fd6c045527e51"}
Jul 09 18:05:32.360861 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:32.360563    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh"
Jul 09 18:05:32.364263 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:32.364234    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a243ec7-6930-4bff-a320-0ddb7037d639","Type":"ContainerStarted","Data":"8963b3fcf51a51d208e0b4eb244d32131e9ae2cd501a3f657c7eb734f73dd212"}
Jul 09 18:05:32.391605 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:32.391537    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh" podStartSLOduration=4.500030468 podStartE2EDuration="9.391517382s" totalImagesPullingTime="4.891486914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:32.391015675 +0000 UTC m=+78.060096630" watchObservedRunningTime="2026-07-09 18:05:32.391517382 +0000 UTC m=+78.060598338"
Jul 09 18:05:32.428720 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:32.428634    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.241278147 podStartE2EDuration="10.428612963s" totalImagesPullingTime="8.168911861s" totalInitContainerRuntime="18.422955ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:32.427346996 +0000 UTC m=+78.096427953" watchObservedRunningTime="2026-07-09 18:05:32.428612963 +0000 UTC m=+78.097693918"
Jul 09 18:05:33.372438 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:33.372373    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a823a0e1-a248-468c-8bbf-027a46435ceb","Type":"ContainerStarted","Data":"125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f"}
Jul 09 18:05:33.372740 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:33.372450    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a823a0e1-a248-468c-8bbf-027a46435ceb","Type":"ContainerStarted","Data":"3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e"}
Jul 09 18:05:34.380132 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:34.380095    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a823a0e1-a248-468c-8bbf-027a46435ceb","Type":"ContainerStarted","Data":"2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e"}
Jul 09 18:05:34.380502 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:34.380189    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a823a0e1-a248-468c-8bbf-027a46435ceb","Type":"ContainerStarted","Data":"04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134"}
Jul 09 18:05:34.380502 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:34.380248    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a823a0e1-a248-468c-8bbf-027a46435ceb","Type":"ContainerStarted","Data":"19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83"}
Jul 09 18:05:34.380502 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:34.380274    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a823a0e1-a248-468c-8bbf-027a46435ceb","Type":"ContainerStarted","Data":"57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d"}
Jul 09 18:05:34.409365 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:34.409307    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.535871492 podStartE2EDuration="7.409292992s" totalImagesPullingTime="3.847119676s" totalInitContainerRuntime="26.301824ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:34.408403585 +0000 UTC m=+80.077484535" watchObservedRunningTime="2026-07-09 18:05:34.409292992 +0000 UTC m=+80.078373945"
Jul 09 18:05:37.975908 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:37.975759    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:05:38.378877 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:38.378781    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6f8c49d449-vp9xh"
Jul 09 18:05:45.749815 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:45.749777    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-578687bf9d-r7prf"
Jul 09 18:05:45.750270 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:45.749845    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-578687bf9d-r7prf"
Jul 09 18:05:58.332069 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:05:58.332037    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-sf462"
Jul 09 18:06:05.754436 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:05.754406    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-578687bf9d-r7prf"
Jul 09 18:06:05.758398 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:05.758374    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-578687bf9d-r7prf"
Jul 09 18:06:27.975894 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:27.975856    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:27.995402 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:27.995373    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:28.623196 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:28.623168    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:41.536026 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:41.535989    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 18:06:41.536582 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:41.536338    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4a243ec7-6930-4bff-a320-0ddb7037d639" containerName="alertmanager" containerID="cri-o://54f34503cf00c5679dd7d5e3214c503ca125c25299f90223856231e85b243b88" gracePeriod=120
Jul 09 18:06:41.536582 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:41.536386    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4a243ec7-6930-4bff-a320-0ddb7037d639" containerName="kube-rbac-proxy-metric" containerID="cri-o://7f19b2d0090355bce6dad6f393e102b54e48e02af168f1dd82238fd6430e29df" gracePeriod=120
Jul 09 18:06:41.536582 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:41.536459    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4a243ec7-6930-4bff-a320-0ddb7037d639" containerName="prom-label-proxy" containerID="cri-o://8963b3fcf51a51d208e0b4eb244d32131e9ae2cd501a3f657c7eb734f73dd212" gracePeriod=120
Jul 09 18:06:41.536582 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:41.536433    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4a243ec7-6930-4bff-a320-0ddb7037d639" containerName="kube-rbac-proxy-web" containerID="cri-o://a0e440970b4dacaa6d7c7bf28bb41c4a89f94a313f637e1a14dfd361e5de6aa4" gracePeriod=120
Jul 09 18:06:41.536582 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:41.536460    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4a243ec7-6930-4bff-a320-0ddb7037d639" containerName="config-reloader" containerID="cri-o://db73e791e536291d320e60782162f055bcaa6c3e56f228c65667e8a3c5ae866b" gracePeriod=120
Jul 09 18:06:41.536582 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:41.536459    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4a243ec7-6930-4bff-a320-0ddb7037d639" containerName="kube-rbac-proxy" containerID="cri-o://0b9cd0fdc2448d85def1ed3cb18d3cfdff5f57152bc4edd0db489c170a733c6c" gracePeriod=120
Jul 09 18:06:41.661340 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:41.661312    2563 generic.go:356] "Generic (PLEG): container finished" podID="4a243ec7-6930-4bff-a320-0ddb7037d639" containerID="8963b3fcf51a51d208e0b4eb244d32131e9ae2cd501a3f657c7eb734f73dd212" exitCode=0
Jul 09 18:06:41.661340 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:41.661339    2563 generic.go:356] "Generic (PLEG): container finished" podID="4a243ec7-6930-4bff-a320-0ddb7037d639" containerID="7f19b2d0090355bce6dad6f393e102b54e48e02af168f1dd82238fd6430e29df" exitCode=0
Jul 09 18:06:41.661487 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:41.661347    2563 generic.go:356] "Generic (PLEG): container finished" podID="4a243ec7-6930-4bff-a320-0ddb7037d639" containerID="0b9cd0fdc2448d85def1ed3cb18d3cfdff5f57152bc4edd0db489c170a733c6c" exitCode=0
Jul 09 18:06:41.661487 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:41.661352    2563 generic.go:356] "Generic (PLEG): container finished" podID="4a243ec7-6930-4bff-a320-0ddb7037d639" containerID="db73e791e536291d320e60782162f055bcaa6c3e56f228c65667e8a3c5ae866b" exitCode=0
Jul 09 18:06:41.661487 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:41.661358    2563 generic.go:356] "Generic (PLEG): container finished" podID="4a243ec7-6930-4bff-a320-0ddb7037d639" containerID="54f34503cf00c5679dd7d5e3214c503ca125c25299f90223856231e85b243b88" exitCode=0
Jul 09 18:06:41.661487 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:41.661387    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a243ec7-6930-4bff-a320-0ddb7037d639","Type":"ContainerDied","Data":"8963b3fcf51a51d208e0b4eb244d32131e9ae2cd501a3f657c7eb734f73dd212"}
Jul 09 18:06:41.661487 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:41.661452    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a243ec7-6930-4bff-a320-0ddb7037d639","Type":"ContainerDied","Data":"7f19b2d0090355bce6dad6f393e102b54e48e02af168f1dd82238fd6430e29df"}
Jul 09 18:06:41.661487 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:41.661478    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a243ec7-6930-4bff-a320-0ddb7037d639","Type":"ContainerDied","Data":"0b9cd0fdc2448d85def1ed3cb18d3cfdff5f57152bc4edd0db489c170a733c6c"}
Jul 09 18:06:41.661654 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:41.661502    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a243ec7-6930-4bff-a320-0ddb7037d639","Type":"ContainerDied","Data":"db73e791e536291d320e60782162f055bcaa6c3e56f228c65667e8a3c5ae866b"}
Jul 09 18:06:41.661654 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:41.661527    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a243ec7-6930-4bff-a320-0ddb7037d639","Type":"ContainerDied","Data":"54f34503cf00c5679dd7d5e3214c503ca125c25299f90223856231e85b243b88"}
Jul 09 18:06:42.669672 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:42.669641    2563 generic.go:356] "Generic (PLEG): container finished" podID="4a243ec7-6930-4bff-a320-0ddb7037d639" containerID="a0e440970b4dacaa6d7c7bf28bb41c4a89f94a313f637e1a14dfd361e5de6aa4" exitCode=0
Jul 09 18:06:42.669672 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:42.669688    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a243ec7-6930-4bff-a320-0ddb7037d639","Type":"ContainerDied","Data":"a0e440970b4dacaa6d7c7bf28bb41c4a89f94a313f637e1a14dfd361e5de6aa4"}
Jul 09 18:06:42.780519 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:42.780493    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:42.912172 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:42.912071    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-web-config\") pod \"4a243ec7-6930-4bff-a320-0ddb7037d639\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") "
Jul 09 18:06:42.912172 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:42.912133    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-secret-alertmanager-kube-rbac-proxy-web\") pod \"4a243ec7-6930-4bff-a320-0ddb7037d639\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") "
Jul 09 18:06:42.912444 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:42.912181    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a243ec7-6930-4bff-a320-0ddb7037d639-alertmanager-trusted-ca-bundle\") pod \"4a243ec7-6930-4bff-a320-0ddb7037d639\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") "
Jul 09 18:06:42.912444 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:42.912206    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a243ec7-6930-4bff-a320-0ddb7037d639-metrics-client-ca\") pod \"4a243ec7-6930-4bff-a320-0ddb7037d639\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") "
Jul 09 18:06:42.912444 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:42.912282    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-config-volume\") pod \"4a243ec7-6930-4bff-a320-0ddb7037d639\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") "
Jul 09 18:06:42.912444 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:42.912306    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-secret-alertmanager-kube-rbac-proxy\") pod \"4a243ec7-6930-4bff-a320-0ddb7037d639\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") "
Jul 09 18:06:42.912444 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:42.912339    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-cluster-tls-config\") pod \"4a243ec7-6930-4bff-a320-0ddb7037d639\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") "
Jul 09 18:06:42.912444 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:42.912374    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a243ec7-6930-4bff-a320-0ddb7037d639-config-out\") pod \"4a243ec7-6930-4bff-a320-0ddb7037d639\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") "
Jul 09 18:06:42.912444 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:42.912402    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4a243ec7-6930-4bff-a320-0ddb7037d639-alertmanager-main-db\") pod \"4a243ec7-6930-4bff-a320-0ddb7037d639\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") "
Jul 09 18:06:42.912444 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:42.912431    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-secret-alertmanager-main-tls\") pod \"4a243ec7-6930-4bff-a320-0ddb7037d639\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") "
Jul 09 18:06:42.912819 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:42.912464    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-secret-alertmanager-kube-rbac-proxy-metric\") pod \"4a243ec7-6930-4bff-a320-0ddb7037d639\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") "
Jul 09 18:06:42.912819 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:42.912503    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a243ec7-6930-4bff-a320-0ddb7037d639-tls-assets\") pod \"4a243ec7-6930-4bff-a320-0ddb7037d639\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") "
Jul 09 18:06:42.912819 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:42.912532    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7p48b\" (UniqueName: \"kubernetes.io/projected/4a243ec7-6930-4bff-a320-0ddb7037d639-kube-api-access-7p48b\") pod \"4a243ec7-6930-4bff-a320-0ddb7037d639\" (UID: \"4a243ec7-6930-4bff-a320-0ddb7037d639\") "
Jul 09 18:06:42.912819 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:42.912589    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a243ec7-6930-4bff-a320-0ddb7037d639-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "4a243ec7-6930-4bff-a320-0ddb7037d639" (UID: "4a243ec7-6930-4bff-a320-0ddb7037d639"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:42.912819 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:42.912633    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a243ec7-6930-4bff-a320-0ddb7037d639-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "4a243ec7-6930-4bff-a320-0ddb7037d639" (UID: "4a243ec7-6930-4bff-a320-0ddb7037d639"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:42.913090 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:42.912821    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a243ec7-6930-4bff-a320-0ddb7037d639-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:06:42.913090 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:42.912842    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a243ec7-6930-4bff-a320-0ddb7037d639-metrics-client-ca\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:06:42.913359 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:42.913330    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a243ec7-6930-4bff-a320-0ddb7037d639-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "4a243ec7-6930-4bff-a320-0ddb7037d639" (UID: "4a243ec7-6930-4bff-a320-0ddb7037d639"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:06:42.915745 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:42.915695    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a243ec7-6930-4bff-a320-0ddb7037d639-kube-api-access-7p48b" (OuterVolumeSpecName: "kube-api-access-7p48b") pod "4a243ec7-6930-4bff-a320-0ddb7037d639" (UID: "4a243ec7-6930-4bff-a320-0ddb7037d639"). InnerVolumeSpecName "kube-api-access-7p48b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:06:42.916063 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:42.916010    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "4a243ec7-6930-4bff-a320-0ddb7037d639" (UID: "4a243ec7-6930-4bff-a320-0ddb7037d639"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:42.916170 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:42.916075    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a243ec7-6930-4bff-a320-0ddb7037d639-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "4a243ec7-6930-4bff-a320-0ddb7037d639" (UID: "4a243ec7-6930-4bff-a320-0ddb7037d639"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:06:42.916265 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:42.916182    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "4a243ec7-6930-4bff-a320-0ddb7037d639" (UID: "4a243ec7-6930-4bff-a320-0ddb7037d639"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:42.916367 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:42.916344    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "4a243ec7-6930-4bff-a320-0ddb7037d639" (UID: "4a243ec7-6930-4bff-a320-0ddb7037d639"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:42.916499 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:42.916474    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "4a243ec7-6930-4bff-a320-0ddb7037d639" (UID: "4a243ec7-6930-4bff-a320-0ddb7037d639"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:42.916671 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:42.916652    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a243ec7-6930-4bff-a320-0ddb7037d639-config-out" (OuterVolumeSpecName: "config-out") pod "4a243ec7-6930-4bff-a320-0ddb7037d639" (UID: "4a243ec7-6930-4bff-a320-0ddb7037d639"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:06:42.916758 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:42.916685    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-config-volume" (OuterVolumeSpecName: "config-volume") pod "4a243ec7-6930-4bff-a320-0ddb7037d639" (UID: "4a243ec7-6930-4bff-a320-0ddb7037d639"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:42.919779 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:42.919755    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "4a243ec7-6930-4bff-a320-0ddb7037d639" (UID: "4a243ec7-6930-4bff-a320-0ddb7037d639"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:42.926929 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:42.926898    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-web-config" (OuterVolumeSpecName: "web-config") pod "4a243ec7-6930-4bff-a320-0ddb7037d639" (UID: "4a243ec7-6930-4bff-a320-0ddb7037d639"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:43.014046 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.014005    2563 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-config-volume\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:06:43.014046 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.014035    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:06:43.014046 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.014046    2563 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-cluster-tls-config\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:06:43.014046 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.014055    2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a243ec7-6930-4bff-a320-0ddb7037d639-config-out\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:06:43.014363 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.014063    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4a243ec7-6930-4bff-a320-0ddb7037d639-alertmanager-main-db\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:06:43.014363 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.014077    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-secret-alertmanager-main-tls\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:06:43.014363 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.014087    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:06:43.014363 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.014095    2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a243ec7-6930-4bff-a320-0ddb7037d639-tls-assets\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:06:43.014363 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.014103    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p48b\" (UniqueName: \"kubernetes.io/projected/4a243ec7-6930-4bff-a320-0ddb7037d639-kube-api-access-7p48b\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:06:43.014363 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.014111    2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-web-config\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:06:43.014363 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.014120    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a243ec7-6930-4bff-a320-0ddb7037d639-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:06:43.680950 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.680912    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a243ec7-6930-4bff-a320-0ddb7037d639","Type":"ContainerDied","Data":"29e7d43696af8ee5df39eae37209a826af734531b01b3d47c2749ed0772fa54e"}
Jul 09 18:06:43.680950 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.680944    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.681478 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.681018    2563 scope.go:117] "RemoveContainer" containerID="8963b3fcf51a51d208e0b4eb244d32131e9ae2cd501a3f657c7eb734f73dd212"
Jul 09 18:06:43.689443 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.689422    2563 scope.go:117] "RemoveContainer" containerID="7f19b2d0090355bce6dad6f393e102b54e48e02af168f1dd82238fd6430e29df"
Jul 09 18:06:43.696330 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.696311    2563 scope.go:117] "RemoveContainer" containerID="0b9cd0fdc2448d85def1ed3cb18d3cfdff5f57152bc4edd0db489c170a733c6c"
Jul 09 18:06:43.702885 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.702862    2563 scope.go:117] "RemoveContainer" containerID="a0e440970b4dacaa6d7c7bf28bb41c4a89f94a313f637e1a14dfd361e5de6aa4"
Jul 09 18:06:43.704656 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.704612    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 18:06:43.711633 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.711614    2563 scope.go:117] "RemoveContainer" containerID="db73e791e536291d320e60782162f055bcaa6c3e56f228c65667e8a3c5ae866b"
Jul 09 18:06:43.711701 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.711684    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 18:06:43.718879 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.718862    2563 scope.go:117] "RemoveContainer" containerID="54f34503cf00c5679dd7d5e3214c503ca125c25299f90223856231e85b243b88"
Jul 09 18:06:43.725879 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.725862    2563 scope.go:117] "RemoveContainer" containerID="789d0472b35001148c9bf4da083d3897990a241db9534e1f5cf765feefda9a65"
Jul 09 18:06:43.744155 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.744132    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 18:06:43.744449 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.744436    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a243ec7-6930-4bff-a320-0ddb7037d639" containerName="kube-rbac-proxy"
Jul 09 18:06:43.744496 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.744451    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a243ec7-6930-4bff-a320-0ddb7037d639" containerName="kube-rbac-proxy"
Jul 09 18:06:43.744496 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.744462    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a243ec7-6930-4bff-a320-0ddb7037d639" containerName="alertmanager"
Jul 09 18:06:43.744496 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.744468    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a243ec7-6930-4bff-a320-0ddb7037d639" containerName="alertmanager"
Jul 09 18:06:43.744496 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.744478    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a243ec7-6930-4bff-a320-0ddb7037d639" containerName="prom-label-proxy"
Jul 09 18:06:43.744496 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.744484    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a243ec7-6930-4bff-a320-0ddb7037d639" containerName="prom-label-proxy"
Jul 09 18:06:43.744496 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.744489    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a243ec7-6930-4bff-a320-0ddb7037d639" containerName="init-config-reloader"
Jul 09 18:06:43.744496 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.744495    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a243ec7-6930-4bff-a320-0ddb7037d639" containerName="init-config-reloader"
Jul 09 18:06:43.744682 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.744507    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a243ec7-6930-4bff-a320-0ddb7037d639" containerName="config-reloader"
Jul 09 18:06:43.744682 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.744512    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a243ec7-6930-4bff-a320-0ddb7037d639" containerName="config-reloader"
Jul 09 18:06:43.744682 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.744521    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a243ec7-6930-4bff-a320-0ddb7037d639" containerName="kube-rbac-proxy-web"
Jul 09 18:06:43.744682 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.744526    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a243ec7-6930-4bff-a320-0ddb7037d639" containerName="kube-rbac-proxy-web"
Jul 09 18:06:43.744682 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.744533    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a243ec7-6930-4bff-a320-0ddb7037d639" containerName="kube-rbac-proxy-metric"
Jul 09 18:06:43.744682 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.744538    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a243ec7-6930-4bff-a320-0ddb7037d639" containerName="kube-rbac-proxy-metric"
Jul 09 18:06:43.744682 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.744584    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a243ec7-6930-4bff-a320-0ddb7037d639" containerName="kube-rbac-proxy"
Jul 09 18:06:43.744682 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.744593    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a243ec7-6930-4bff-a320-0ddb7037d639" containerName="kube-rbac-proxy-metric"
Jul 09 18:06:43.744682 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.744599    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a243ec7-6930-4bff-a320-0ddb7037d639" containerName="alertmanager"
Jul 09 18:06:43.744682 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.744604    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a243ec7-6930-4bff-a320-0ddb7037d639" containerName="prom-label-proxy"
Jul 09 18:06:43.744682 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.744610    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a243ec7-6930-4bff-a320-0ddb7037d639" containerName="config-reloader"
Jul 09 18:06:43.744682 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.744615    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a243ec7-6930-4bff-a320-0ddb7037d639" containerName="kube-rbac-proxy-web"
Jul 09 18:06:43.748128 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.748110    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.750411 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.750392    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 09 18:06:43.750536 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.750503    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-c8nvb\""
Jul 09 18:06:43.750713 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.750673    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 09 18:06:43.750794 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.750744    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 09 18:06:43.751022 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.750871    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 09 18:06:43.751022 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.750946    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 09 18:06:43.751239 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.751174    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 09 18:06:43.751239 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.751174    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 09 18:06:43.751401 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.751263    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 09 18:06:43.756041 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.756019    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 09 18:06:43.764056 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.764033    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 18:06:43.819009 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.818966    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0ecd97df-2046-4189-a979-df898eb3faaa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.819009 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.819008    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ecd97df-2046-4189-a979-df898eb3faaa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.819247 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.819036    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0ecd97df-2046-4189-a979-df898eb3faaa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.819247 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.819084    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0ecd97df-2046-4189-a979-df898eb3faaa-config-out\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.819247 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.819107    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0ecd97df-2046-4189-a979-df898eb3faaa-web-config\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.819247 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.819142    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0ecd97df-2046-4189-a979-df898eb3faaa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.819247 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.819163    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ecd97df-2046-4189-a979-df898eb3faaa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.819247 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.819182    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0ecd97df-2046-4189-a979-df898eb3faaa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.819247 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.819245    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0ecd97df-2046-4189-a979-df898eb3faaa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.819450 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.819271    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0ecd97df-2046-4189-a979-df898eb3faaa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.819450 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.819292    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0ecd97df-2046-4189-a979-df898eb3faaa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.819450 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.819306    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srtwf\" (UniqueName: \"kubernetes.io/projected/0ecd97df-2046-4189-a979-df898eb3faaa-kube-api-access-srtwf\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.819450 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.819328    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0ecd97df-2046-4189-a979-df898eb3faaa-config-volume\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.920120 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.920084    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0ecd97df-2046-4189-a979-df898eb3faaa-config-volume\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.920360 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.920141    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0ecd97df-2046-4189-a979-df898eb3faaa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.920360 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.920167    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ecd97df-2046-4189-a979-df898eb3faaa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.920360 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.920201    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0ecd97df-2046-4189-a979-df898eb3faaa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.920360 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.920253    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0ecd97df-2046-4189-a979-df898eb3faaa-config-out\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.920360 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.920277    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0ecd97df-2046-4189-a979-df898eb3faaa-web-config\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.920360 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.920312    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0ecd97df-2046-4189-a979-df898eb3faaa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.920360 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.920344    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ecd97df-2046-4189-a979-df898eb3faaa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.920700 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.920377    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0ecd97df-2046-4189-a979-df898eb3faaa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.920700 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.920408    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0ecd97df-2046-4189-a979-df898eb3faaa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.920700 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.920434    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0ecd97df-2046-4189-a979-df898eb3faaa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.920700 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.920463    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0ecd97df-2046-4189-a979-df898eb3faaa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.920700 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.920488    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srtwf\" (UniqueName: \"kubernetes.io/projected/0ecd97df-2046-4189-a979-df898eb3faaa-kube-api-access-srtwf\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.920700 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.920650    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0ecd97df-2046-4189-a979-df898eb3faaa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.921444 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.921084    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ecd97df-2046-4189-a979-df898eb3faaa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.923263 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.923238    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0ecd97df-2046-4189-a979-df898eb3faaa-config-out\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.923564 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.923407    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0ecd97df-2046-4189-a979-df898eb3faaa-web-config\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.923564 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.923478    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0ecd97df-2046-4189-a979-df898eb3faaa-config-volume\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.923802 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.923596    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0ecd97df-2046-4189-a979-df898eb3faaa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.923802 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.923669    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0ecd97df-2046-4189-a979-df898eb3faaa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.923802 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.923720    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0ecd97df-2046-4189-a979-df898eb3faaa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.923967 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.923926    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0ecd97df-2046-4189-a979-df898eb3faaa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.924190 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.924169    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0ecd97df-2046-4189-a979-df898eb3faaa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.924430 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.924405    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ecd97df-2046-4189-a979-df898eb3faaa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.925424 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.925406    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0ecd97df-2046-4189-a979-df898eb3faaa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:43.928769 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:43.928748    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srtwf\" (UniqueName: \"kubernetes.io/projected/0ecd97df-2046-4189-a979-df898eb3faaa-kube-api-access-srtwf\") pod \"alertmanager-main-0\" (UID: \"0ecd97df-2046-4189-a979-df898eb3faaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.059280 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:44.059229    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 09 18:06:44.207007 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:44.206976    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 09 18:06:44.208766 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:06:44.208736    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ecd97df_2046_4189_a979_df898eb3faaa.slice/crio-d8007b9b879fb5d5fb0f1b9cd69ed21355f147578543b345302a00899c26110a WatchSource:0}: Error finding container d8007b9b879fb5d5fb0f1b9cd69ed21355f147578543b345302a00899c26110a: Status 404 returned error can't find the container with id d8007b9b879fb5d5fb0f1b9cd69ed21355f147578543b345302a00899c26110a
Jul 09 18:06:44.686955 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:44.686922    2563 generic.go:356] "Generic (PLEG): container finished" podID="0ecd97df-2046-4189-a979-df898eb3faaa" containerID="b04bc4ef66327ea80d9e86e09ed255c719c1c1cab7f8d8e93593eb9085e650e3" exitCode=0
Jul 09 18:06:44.687346 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:44.686962    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0ecd97df-2046-4189-a979-df898eb3faaa","Type":"ContainerDied","Data":"b04bc4ef66327ea80d9e86e09ed255c719c1c1cab7f8d8e93593eb9085e650e3"}
Jul 09 18:06:44.687346 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:44.687022    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0ecd97df-2046-4189-a979-df898eb3faaa","Type":"ContainerStarted","Data":"d8007b9b879fb5d5fb0f1b9cd69ed21355f147578543b345302a00899c26110a"}
Jul 09 18:06:44.960589 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:44.960564    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a243ec7-6930-4bff-a320-0ddb7037d639" path="/var/lib/kubelet/pods/4a243ec7-6930-4bff-a320-0ddb7037d639/volumes"
Jul 09 18:06:45.539011 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.538930    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6f77f67f4b-gkd94"]
Jul 09 18:06:45.542291 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.542270    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f77f67f4b-gkd94"
Jul 09 18:06:45.544764 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.544741    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 09 18:06:45.544876 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.544794    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 09 18:06:45.545125 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.545107    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 09 18:06:45.545189 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.545146    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 09 18:06:45.545189 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.545179    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 09 18:06:45.545644 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.545618    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-72z9z\""
Jul 09 18:06:45.551272 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.551187    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 09 18:06:45.559612 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.559587    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f77f67f4b-gkd94"]
Jul 09 18:06:45.632645 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.632609    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e1f45c65-dd77-474d-8768-4b93049bffd9-secret-telemeter-client\") pod \"telemeter-client-6f77f67f4b-gkd94\" (UID: \"e1f45c65-dd77-474d-8768-4b93049bffd9\") " pod="openshift-monitoring/telemeter-client-6f77f67f4b-gkd94"
Jul 09 18:06:45.632816 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.632654    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1f45c65-dd77-474d-8768-4b93049bffd9-metrics-client-ca\") pod \"telemeter-client-6f77f67f4b-gkd94\" (UID: \"e1f45c65-dd77-474d-8768-4b93049bffd9\") " pod="openshift-monitoring/telemeter-client-6f77f67f4b-gkd94"
Jul 09 18:06:45.632816 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.632680    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1f45c65-dd77-474d-8768-4b93049bffd9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f77f67f4b-gkd94\" (UID: \"e1f45c65-dd77-474d-8768-4b93049bffd9\") " pod="openshift-monitoring/telemeter-client-6f77f67f4b-gkd94"
Jul 09 18:06:45.632816 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.632776    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1f45c65-dd77-474d-8768-4b93049bffd9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f77f67f4b-gkd94\" (UID: \"e1f45c65-dd77-474d-8768-4b93049bffd9\") " pod="openshift-monitoring/telemeter-client-6f77f67f4b-gkd94"
Jul 09 18:06:45.632945 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.632867    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1f45c65-dd77-474d-8768-4b93049bffd9-serving-certs-ca-bundle\") pod \"telemeter-client-6f77f67f4b-gkd94\" (UID: \"e1f45c65-dd77-474d-8768-4b93049bffd9\") " pod="openshift-monitoring/telemeter-client-6f77f67f4b-gkd94"
Jul 09 18:06:45.632945 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.632895    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e1f45c65-dd77-474d-8768-4b93049bffd9-telemeter-client-tls\") pod \"telemeter-client-6f77f67f4b-gkd94\" (UID: \"e1f45c65-dd77-474d-8768-4b93049bffd9\") " pod="openshift-monitoring/telemeter-client-6f77f67f4b-gkd94"
Jul 09 18:06:45.632945 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.632914    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e1f45c65-dd77-474d-8768-4b93049bffd9-federate-client-tls\") pod \"telemeter-client-6f77f67f4b-gkd94\" (UID: \"e1f45c65-dd77-474d-8768-4b93049bffd9\") " pod="openshift-monitoring/telemeter-client-6f77f67f4b-gkd94"
Jul 09 18:06:45.632945 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.632939    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7bsr\" (UniqueName: \"kubernetes.io/projected/e1f45c65-dd77-474d-8768-4b93049bffd9-kube-api-access-k7bsr\") pod \"telemeter-client-6f77f67f4b-gkd94\" (UID: \"e1f45c65-dd77-474d-8768-4b93049bffd9\") " pod="openshift-monitoring/telemeter-client-6f77f67f4b-gkd94"
Jul 09 18:06:45.695401 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.695362    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0ecd97df-2046-4189-a979-df898eb3faaa","Type":"ContainerStarted","Data":"3e22f2fcba88d70b6c851721560bf33b62ce91959f8bb3c6e6df62df18d4b048"}
Jul 09 18:06:45.695816 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.695444    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0ecd97df-2046-4189-a979-df898eb3faaa","Type":"ContainerStarted","Data":"956ec7571eb12b6ef94a477cf12be176ae3202fb4adc1141103011709cbae32e"}
Jul 09 18:06:45.695816 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.695487    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0ecd97df-2046-4189-a979-df898eb3faaa","Type":"ContainerStarted","Data":"b5e440a426d2043a48c67ebe636b274db6fe24f60924bd3bd607f4de1c7bc766"}
Jul 09 18:06:45.695816 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.695522    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0ecd97df-2046-4189-a979-df898eb3faaa","Type":"ContainerStarted","Data":"f5a55c6bb97484d2ef013ebb37d16f57d808c43fcb1a167834bbbe6c79847f81"}
Jul 09 18:06:45.695816 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.695555    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0ecd97df-2046-4189-a979-df898eb3faaa","Type":"ContainerStarted","Data":"f06fc7508d52e37c62b6198ea0aea56855bfab3327d3ad46d512f951623041d4"}
Jul 09 18:06:45.695816 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.695592    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0ecd97df-2046-4189-a979-df898eb3faaa","Type":"ContainerStarted","Data":"aab45a591b219b28a4a04abbc4346951193d81f8c05c2b1c56e218b9cdb62c53"}
Jul 09 18:06:45.732466 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.732403    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.717544955 podStartE2EDuration="2.732379985s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.83503ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:06:45.730686867 +0000 UTC m=+151.399767847" watchObservedRunningTime="2026-07-09 18:06:45.732379985 +0000 UTC m=+151.401460941"
Jul 09 18:06:45.734313 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.734276    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1f45c65-dd77-474d-8768-4b93049bffd9-serving-certs-ca-bundle\") pod \"telemeter-client-6f77f67f4b-gkd94\" (UID: \"e1f45c65-dd77-474d-8768-4b93049bffd9\") " pod="openshift-monitoring/telemeter-client-6f77f67f4b-gkd94"
Jul 09 18:06:45.734457 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.734326    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e1f45c65-dd77-474d-8768-4b93049bffd9-telemeter-client-tls\") pod \"telemeter-client-6f77f67f4b-gkd94\" (UID: \"e1f45c65-dd77-474d-8768-4b93049bffd9\") " pod="openshift-monitoring/telemeter-client-6f77f67f4b-gkd94"
Jul 09 18:06:45.734457 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.734353    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e1f45c65-dd77-474d-8768-4b93049bffd9-federate-client-tls\") pod \"telemeter-client-6f77f67f4b-gkd94\" (UID: \"e1f45c65-dd77-474d-8768-4b93049bffd9\") " pod="openshift-monitoring/telemeter-client-6f77f67f4b-gkd94"
Jul 09 18:06:45.734457 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.734402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7bsr\" (UniqueName: \"kubernetes.io/projected/e1f45c65-dd77-474d-8768-4b93049bffd9-kube-api-access-k7bsr\") pod \"telemeter-client-6f77f67f4b-gkd94\" (UID: \"e1f45c65-dd77-474d-8768-4b93049bffd9\") " pod="openshift-monitoring/telemeter-client-6f77f67f4b-gkd94"
Jul 09 18:06:45.734640 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.734498    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e1f45c65-dd77-474d-8768-4b93049bffd9-secret-telemeter-client\") pod \"telemeter-client-6f77f67f4b-gkd94\" (UID: \"e1f45c65-dd77-474d-8768-4b93049bffd9\") " pod="openshift-monitoring/telemeter-client-6f77f67f4b-gkd94"
Jul 09 18:06:45.734640 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.734543    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1f45c65-dd77-474d-8768-4b93049bffd9-metrics-client-ca\") pod \"telemeter-client-6f77f67f4b-gkd94\" (UID: \"e1f45c65-dd77-474d-8768-4b93049bffd9\") " pod="openshift-monitoring/telemeter-client-6f77f67f4b-gkd94"
Jul 09 18:06:45.734640 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.734565    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1f45c65-dd77-474d-8768-4b93049bffd9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f77f67f4b-gkd94\" (UID: \"e1f45c65-dd77-474d-8768-4b93049bffd9\") " pod="openshift-monitoring/telemeter-client-6f77f67f4b-gkd94"
Jul 09 18:06:45.734792 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.734642    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1f45c65-dd77-474d-8768-4b93049bffd9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f77f67f4b-gkd94\" (UID: \"e1f45c65-dd77-474d-8768-4b93049bffd9\") " pod="openshift-monitoring/telemeter-client-6f77f67f4b-gkd94"
Jul 09 18:06:45.736299 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.736273    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1f45c65-dd77-474d-8768-4b93049bffd9-serving-certs-ca-bundle\") pod \"telemeter-client-6f77f67f4b-gkd94\" (UID: \"e1f45c65-dd77-474d-8768-4b93049bffd9\") " pod="openshift-monitoring/telemeter-client-6f77f67f4b-gkd94"
Jul 09 18:06:45.736642 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.736614    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1f45c65-dd77-474d-8768-4b93049bffd9-metrics-client-ca\") pod \"telemeter-client-6f77f67f4b-gkd94\" (UID: \"e1f45c65-dd77-474d-8768-4b93049bffd9\") " pod="openshift-monitoring/telemeter-client-6f77f67f4b-gkd94"
Jul 09 18:06:45.737133 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.737104    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1f45c65-dd77-474d-8768-4b93049bffd9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f77f67f4b-gkd94\" (UID: \"e1f45c65-dd77-474d-8768-4b93049bffd9\") " pod="openshift-monitoring/telemeter-client-6f77f67f4b-gkd94"
Jul 09 18:06:45.738392 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.738372    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e1f45c65-dd77-474d-8768-4b93049bffd9-secret-telemeter-client\") pod \"telemeter-client-6f77f67f4b-gkd94\" (UID: \"e1f45c65-dd77-474d-8768-4b93049bffd9\") " pod="openshift-monitoring/telemeter-client-6f77f67f4b-gkd94"
Jul 09 18:06:45.738531 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.738514    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e1f45c65-dd77-474d-8768-4b93049bffd9-telemeter-client-tls\") pod \"telemeter-client-6f77f67f4b-gkd94\" (UID: \"e1f45c65-dd77-474d-8768-4b93049bffd9\") " pod="openshift-monitoring/telemeter-client-6f77f67f4b-gkd94"
Jul 09 18:06:45.738737 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.738719    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e1f45c65-dd77-474d-8768-4b93049bffd9-federate-client-tls\") pod \"telemeter-client-6f77f67f4b-gkd94\" (UID: \"e1f45c65-dd77-474d-8768-4b93049bffd9\") " pod="openshift-monitoring/telemeter-client-6f77f67f4b-gkd94"
Jul 09 18:06:45.738788 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.738770    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1f45c65-dd77-474d-8768-4b93049bffd9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f77f67f4b-gkd94\" (UID: \"e1f45c65-dd77-474d-8768-4b93049bffd9\") " pod="openshift-monitoring/telemeter-client-6f77f67f4b-gkd94"
Jul 09 18:06:45.748830 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.748804    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7bsr\" (UniqueName: \"kubernetes.io/projected/e1f45c65-dd77-474d-8768-4b93049bffd9-kube-api-access-k7bsr\") pod \"telemeter-client-6f77f67f4b-gkd94\" (UID: \"e1f45c65-dd77-474d-8768-4b93049bffd9\") " pod="openshift-monitoring/telemeter-client-6f77f67f4b-gkd94"
Jul 09 18:06:45.853590 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.853497    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f77f67f4b-gkd94"
Jul 09 18:06:45.864335 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.864036    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 18:06:45.864484 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.864410    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a823a0e1-a248-468c-8bbf-027a46435ceb" containerName="prometheus" containerID="cri-o://3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e" gracePeriod=600
Jul 09 18:06:45.864659 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.864628    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a823a0e1-a248-468c-8bbf-027a46435ceb" containerName="kube-rbac-proxy" containerID="cri-o://04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134" gracePeriod=600
Jul 09 18:06:45.865080 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.864626    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a823a0e1-a248-468c-8bbf-027a46435ceb" containerName="thanos-sidecar" containerID="cri-o://57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d" gracePeriod=600
Jul 09 18:06:45.865200 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.864671    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a823a0e1-a248-468c-8bbf-027a46435ceb" containerName="config-reloader" containerID="cri-o://125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f" gracePeriod=600
Jul 09 18:06:45.865296 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.864676    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a823a0e1-a248-468c-8bbf-027a46435ceb" containerName="kube-rbac-proxy-web" containerID="cri-o://19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83" gracePeriod=600
Jul 09 18:06:45.865359 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:45.864884    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a823a0e1-a248-468c-8bbf-027a46435ceb" containerName="kube-rbac-proxy-thanos" containerID="cri-o://2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e" gracePeriod=600
Jul 09 18:06:46.028182 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.028145    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f77f67f4b-gkd94"]
Jul 09 18:06:46.031869 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:06:46.031836    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1f45c65_dd77_474d_8768_4b93049bffd9.slice/crio-d4833bdd79a706990c19d6fb1973e942717b72534daf5db9caabf768e0ebb2e7 WatchSource:0}: Error finding container d4833bdd79a706990c19d6fb1973e942717b72534daf5db9caabf768e0ebb2e7: Status 404 returned error can't find the container with id d4833bdd79a706990c19d6fb1973e942717b72534daf5db9caabf768e0ebb2e7
Jul 09 18:06:46.128269 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.128241    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.137733 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.137696    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a823a0e1-a248-468c-8bbf-027a46435ceb-config-out\") pod \"a823a0e1-a248-468c-8bbf-027a46435ceb\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") "
Jul 09 18:06:46.137733 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.137731    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"a823a0e1-a248-468c-8bbf-027a46435ceb\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") "
Jul 09 18:06:46.137955 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.137754    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-metrics-client-certs\") pod \"a823a0e1-a248-468c-8bbf-027a46435ceb\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") "
Jul 09 18:06:46.137955 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.137770    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-prometheus-trusted-ca-bundle\") pod \"a823a0e1-a248-468c-8bbf-027a46435ceb\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") "
Jul 09 18:06:46.137955 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.137893    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-prometheus-k8s-rulefiles-2\") pod \"a823a0e1-a248-468c-8bbf-027a46435ceb\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") "
Jul 09 18:06:46.137955 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.137932    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a823a0e1-a248-468c-8bbf-027a46435ceb-tls-assets\") pod \"a823a0e1-a248-468c-8bbf-027a46435ceb\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") "
Jul 09 18:06:46.138142 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.137968    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-configmap-kubelet-serving-ca-bundle\") pod \"a823a0e1-a248-468c-8bbf-027a46435ceb\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") "
Jul 09 18:06:46.138142 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.137998    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-grpc-tls\") pod \"a823a0e1-a248-468c-8bbf-027a46435ceb\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") "
Jul 09 18:06:46.138142 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.138041    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-thanos-prometheus-http-client-file\") pod \"a823a0e1-a248-468c-8bbf-027a46435ceb\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") "
Jul 09 18:06:46.138142 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.138084    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-web-config\") pod \"a823a0e1-a248-468c-8bbf-027a46435ceb\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") "
Jul 09 18:06:46.138142 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.138112    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-prometheus-k8s-rulefiles-0\") pod \"a823a0e1-a248-468c-8bbf-027a46435ceb\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") "
Jul 09 18:06:46.138411 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.138145    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-kube-rbac-proxy\") pod \"a823a0e1-a248-468c-8bbf-027a46435ceb\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") "
Jul 09 18:06:46.138411 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.138163    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "a823a0e1-a248-468c-8bbf-027a46435ceb" (UID: "a823a0e1-a248-468c-8bbf-027a46435ceb"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:46.138411 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.138169    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-config\") pod \"a823a0e1-a248-468c-8bbf-027a46435ceb\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") "
Jul 09 18:06:46.138411 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.138317    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-prometheus-k8s-rulefiles-1\") pod \"a823a0e1-a248-468c-8bbf-027a46435ceb\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") "
Jul 09 18:06:46.138411 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.138362    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7cb9\" (UniqueName: \"kubernetes.io/projected/a823a0e1-a248-468c-8bbf-027a46435ceb-kube-api-access-c7cb9\") pod \"a823a0e1-a248-468c-8bbf-027a46435ceb\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") "
Jul 09 18:06:46.138411 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.138390    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-configmap-serving-certs-ca-bundle\") pod \"a823a0e1-a248-468c-8bbf-027a46435ceb\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") "
Jul 09 18:06:46.138411 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.138394    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "a823a0e1-a248-468c-8bbf-027a46435ceb" (UID: "a823a0e1-a248-468c-8bbf-027a46435ceb"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:46.138791 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.138423    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"a823a0e1-a248-468c-8bbf-027a46435ceb\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") "
Jul 09 18:06:46.138791 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.138490    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "a823a0e1-a248-468c-8bbf-027a46435ceb" (UID: "a823a0e1-a248-468c-8bbf-027a46435ceb"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:46.138900 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.138849    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "a823a0e1-a248-468c-8bbf-027a46435ceb" (UID: "a823a0e1-a248-468c-8bbf-027a46435ceb"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:46.139918 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.139167    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "a823a0e1-a248-468c-8bbf-027a46435ceb" (UID: "a823a0e1-a248-468c-8bbf-027a46435ceb"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:46.139918 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.139230    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-prometheus-k8s-tls\") pod \"a823a0e1-a248-468c-8bbf-027a46435ceb\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") "
Jul 09 18:06:46.139918 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.139274    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a823a0e1-a248-468c-8bbf-027a46435ceb-prometheus-k8s-db\") pod \"a823a0e1-a248-468c-8bbf-027a46435ceb\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") "
Jul 09 18:06:46.139918 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.139300    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-configmap-metrics-client-ca\") pod \"a823a0e1-a248-468c-8bbf-027a46435ceb\" (UID: \"a823a0e1-a248-468c-8bbf-027a46435ceb\") "
Jul 09 18:06:46.139918 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.139596    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:06:46.139918 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.139618    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:06:46.139918 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.139633    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:06:46.139918 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.139648    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:06:46.139918 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.139661    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:06:46.140420 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.140010    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "a823a0e1-a248-468c-8bbf-027a46435ceb" (UID: "a823a0e1-a248-468c-8bbf-027a46435ceb"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:46.142475 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.141280    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "a823a0e1-a248-468c-8bbf-027a46435ceb" (UID: "a823a0e1-a248-468c-8bbf-027a46435ceb"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:46.142475 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.141621    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a823a0e1-a248-468c-8bbf-027a46435ceb-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "a823a0e1-a248-468c-8bbf-027a46435ceb" (UID: "a823a0e1-a248-468c-8bbf-027a46435ceb"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:06:46.142951 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.142904    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a823a0e1-a248-468c-8bbf-027a46435ceb-kube-api-access-c7cb9" (OuterVolumeSpecName: "kube-api-access-c7cb9") pod "a823a0e1-a248-468c-8bbf-027a46435ceb" (UID: "a823a0e1-a248-468c-8bbf-027a46435ceb"). InnerVolumeSpecName "kube-api-access-c7cb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:06:46.144086 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.143937    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "a823a0e1-a248-468c-8bbf-027a46435ceb" (UID: "a823a0e1-a248-468c-8bbf-027a46435ceb"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:46.144086 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.144006    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-config" (OuterVolumeSpecName: "config") pod "a823a0e1-a248-468c-8bbf-027a46435ceb" (UID: "a823a0e1-a248-468c-8bbf-027a46435ceb"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:46.144086 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.144078    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "a823a0e1-a248-468c-8bbf-027a46435ceb" (UID: "a823a0e1-a248-468c-8bbf-027a46435ceb"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:46.144833 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.144731    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "a823a0e1-a248-468c-8bbf-027a46435ceb" (UID: "a823a0e1-a248-468c-8bbf-027a46435ceb"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:46.144833 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.144777    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a823a0e1-a248-468c-8bbf-027a46435ceb-config-out" (OuterVolumeSpecName: "config-out") pod "a823a0e1-a248-468c-8bbf-027a46435ceb" (UID: "a823a0e1-a248-468c-8bbf-027a46435ceb"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:06:46.144833 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.144774    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "a823a0e1-a248-468c-8bbf-027a46435ceb" (UID: "a823a0e1-a248-468c-8bbf-027a46435ceb"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:46.145087 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.145057    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "a823a0e1-a248-468c-8bbf-027a46435ceb" (UID: "a823a0e1-a248-468c-8bbf-027a46435ceb"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:46.145197 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.145113    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "a823a0e1-a248-468c-8bbf-027a46435ceb" (UID: "a823a0e1-a248-468c-8bbf-027a46435ceb"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:46.145393 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.145373    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "a823a0e1-a248-468c-8bbf-027a46435ceb" (UID: "a823a0e1-a248-468c-8bbf-027a46435ceb"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:46.145618 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.145596    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a823a0e1-a248-468c-8bbf-027a46435ceb-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a823a0e1-a248-468c-8bbf-027a46435ceb" (UID: "a823a0e1-a248-468c-8bbf-027a46435ceb"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:06:46.154967 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.154937    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-web-config" (OuterVolumeSpecName: "web-config") pod "a823a0e1-a248-468c-8bbf-027a46435ceb" (UID: "a823a0e1-a248-468c-8bbf-027a46435ceb"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:46.240451 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.240418    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:06:46.240451 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.240446    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:06:46.240451 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.240459    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a823a0e1-a248-468c-8bbf-027a46435ceb-prometheus-k8s-db\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:06:46.240681 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.240468    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-configmap-metrics-client-ca\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:06:46.240681 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.240478    2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a823a0e1-a248-468c-8bbf-027a46435ceb-config-out\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:06:46.240681 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.240488    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:06:46.240681 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.240496    2563 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-metrics-client-certs\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:06:46.240681 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.240505    2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a823a0e1-a248-468c-8bbf-027a46435ceb-tls-assets\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:06:46.240681 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.240512    2563 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-grpc-tls\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:06:46.240681 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.240520    2563 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:06:46.240681 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.240529    2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-web-config\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:06:46.240681 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.240537    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a823a0e1-a248-468c-8bbf-027a46435ceb-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:06:46.240681 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.240545    2563 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-secret-kube-rbac-proxy\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:06:46.240681 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.240554    2563 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a823a0e1-a248-468c-8bbf-027a46435ceb-config\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:06:46.240681 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.240562    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7cb9\" (UniqueName: \"kubernetes.io/projected/a823a0e1-a248-468c-8bbf-027a46435ceb-kube-api-access-c7cb9\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:06:46.701231 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.701185    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f77f67f4b-gkd94" event={"ID":"e1f45c65-dd77-474d-8768-4b93049bffd9","Type":"ContainerStarted","Data":"d4833bdd79a706990c19d6fb1973e942717b72534daf5db9caabf768e0ebb2e7"}
Jul 09 18:06:46.704316 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.704287    2563 generic.go:356] "Generic (PLEG): container finished" podID="a823a0e1-a248-468c-8bbf-027a46435ceb" containerID="2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e" exitCode=0
Jul 09 18:06:46.704316 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.704311    2563 generic.go:356] "Generic (PLEG): container finished" podID="a823a0e1-a248-468c-8bbf-027a46435ceb" containerID="04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134" exitCode=0
Jul 09 18:06:46.704316 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.704319    2563 generic.go:356] "Generic (PLEG): container finished" podID="a823a0e1-a248-468c-8bbf-027a46435ceb" containerID="19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83" exitCode=0
Jul 09 18:06:46.704548 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.704325    2563 generic.go:356] "Generic (PLEG): container finished" podID="a823a0e1-a248-468c-8bbf-027a46435ceb" containerID="57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d" exitCode=0
Jul 09 18:06:46.704548 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.704330    2563 generic.go:356] "Generic (PLEG): container finished" podID="a823a0e1-a248-468c-8bbf-027a46435ceb" containerID="125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f" exitCode=0
Jul 09 18:06:46.704548 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.704335    2563 generic.go:356] "Generic (PLEG): container finished" podID="a823a0e1-a248-468c-8bbf-027a46435ceb" containerID="3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e" exitCode=0
Jul 09 18:06:46.704548 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.704377    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a823a0e1-a248-468c-8bbf-027a46435ceb","Type":"ContainerDied","Data":"2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e"}
Jul 09 18:06:46.704548 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.704409    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.704548 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.704454    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a823a0e1-a248-468c-8bbf-027a46435ceb","Type":"ContainerDied","Data":"04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134"}
Jul 09 18:06:46.704548 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.704470    2563 scope.go:117] "RemoveContainer" containerID="2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e"
Jul 09 18:06:46.704548 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.704498    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a823a0e1-a248-468c-8bbf-027a46435ceb","Type":"ContainerDied","Data":"19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83"}
Jul 09 18:06:46.704548 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.704537    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a823a0e1-a248-468c-8bbf-027a46435ceb","Type":"ContainerDied","Data":"57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d"}
Jul 09 18:06:46.704939 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.704576    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a823a0e1-a248-468c-8bbf-027a46435ceb","Type":"ContainerDied","Data":"125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f"}
Jul 09 18:06:46.704939 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.704611    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a823a0e1-a248-468c-8bbf-027a46435ceb","Type":"ContainerDied","Data":"3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e"}
Jul 09 18:06:46.704939 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.704644    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a823a0e1-a248-468c-8bbf-027a46435ceb","Type":"ContainerDied","Data":"1f684464b0d961bb1b187b572ac473f65f5f9808877abbf5a454fabd5484dd67"}
Jul 09 18:06:46.713321 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.713304    2563 scope.go:117] "RemoveContainer" containerID="04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134"
Jul 09 18:06:46.721274 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.721251    2563 scope.go:117] "RemoveContainer" containerID="19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83"
Jul 09 18:06:46.728607 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.728590    2563 scope.go:117] "RemoveContainer" containerID="57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d"
Jul 09 18:06:46.735402 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.735375    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 18:06:46.739981 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.739097    2563 scope.go:117] "RemoveContainer" containerID="125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f"
Jul 09 18:06:46.742959 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.742929    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 18:06:46.753399 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.753371    2563 scope.go:117] "RemoveContainer" containerID="3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e"
Jul 09 18:06:46.762555 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.762529    2563 scope.go:117] "RemoveContainer" containerID="9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233"
Jul 09 18:06:46.767294 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.767266    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 18:06:46.767660 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.767643    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a823a0e1-a248-468c-8bbf-027a46435ceb" containerName="prometheus"
Jul 09 18:06:46.767750 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.767662    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a823a0e1-a248-468c-8bbf-027a46435ceb" containerName="prometheus"
Jul 09 18:06:46.767750 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.767675    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a823a0e1-a248-468c-8bbf-027a46435ceb" containerName="init-config-reloader"
Jul 09 18:06:46.767750 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.767684    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a823a0e1-a248-468c-8bbf-027a46435ceb" containerName="init-config-reloader"
Jul 09 18:06:46.767750 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.767698    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a823a0e1-a248-468c-8bbf-027a46435ceb" containerName="kube-rbac-proxy-web"
Jul 09 18:06:46.767750 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.767706    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a823a0e1-a248-468c-8bbf-027a46435ceb" containerName="kube-rbac-proxy-web"
Jul 09 18:06:46.767750 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.767718    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a823a0e1-a248-468c-8bbf-027a46435ceb" containerName="thanos-sidecar"
Jul 09 18:06:46.767750 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.767726    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a823a0e1-a248-468c-8bbf-027a46435ceb" containerName="thanos-sidecar"
Jul 09 18:06:46.767750 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.767739    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a823a0e1-a248-468c-8bbf-027a46435ceb" containerName="kube-rbac-proxy-thanos"
Jul 09 18:06:46.767750 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.767747    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a823a0e1-a248-468c-8bbf-027a46435ceb" containerName="kube-rbac-proxy-thanos"
Jul 09 18:06:46.768153 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.767761    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a823a0e1-a248-468c-8bbf-027a46435ceb" containerName="kube-rbac-proxy"
Jul 09 18:06:46.768153 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.767769    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a823a0e1-a248-468c-8bbf-027a46435ceb" containerName="kube-rbac-proxy"
Jul 09 18:06:46.768153 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.767781    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a823a0e1-a248-468c-8bbf-027a46435ceb" containerName="config-reloader"
Jul 09 18:06:46.768153 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.767789    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a823a0e1-a248-468c-8bbf-027a46435ceb" containerName="config-reloader"
Jul 09 18:06:46.768153 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.767861    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a823a0e1-a248-468c-8bbf-027a46435ceb" containerName="prometheus"
Jul 09 18:06:46.768153 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.767876    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a823a0e1-a248-468c-8bbf-027a46435ceb" containerName="kube-rbac-proxy"
Jul 09 18:06:46.768153 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.767888    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a823a0e1-a248-468c-8bbf-027a46435ceb" containerName="config-reloader"
Jul 09 18:06:46.768153 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.767899    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a823a0e1-a248-468c-8bbf-027a46435ceb" containerName="kube-rbac-proxy-web"
Jul 09 18:06:46.768153 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.767910    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a823a0e1-a248-468c-8bbf-027a46435ceb" containerName="kube-rbac-proxy-thanos"
Jul 09 18:06:46.768153 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.767918    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a823a0e1-a248-468c-8bbf-027a46435ceb" containerName="thanos-sidecar"
Jul 09 18:06:46.770944 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.770926    2563 scope.go:117] "RemoveContainer" containerID="2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e"
Jul 09 18:06:46.771303 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:06:46.771274    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e\": container with ID starting with 2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e not found: ID does not exist" containerID="2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e"
Jul 09 18:06:46.771408 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.771314    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e"} err="failed to get container status \"2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e\": rpc error: code = NotFound desc = could not find container \"2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e\": container with ID starting with 2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e not found: ID does not exist"
Jul 09 18:06:46.771408 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.771358    2563 scope.go:117] "RemoveContainer" containerID="04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134"
Jul 09 18:06:46.771641 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:06:46.771613    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134\": container with ID starting with 04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134 not found: ID does not exist" containerID="04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134"
Jul 09 18:06:46.771688 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.771647    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134"} err="failed to get container status \"04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134\": rpc error: code = NotFound desc = could not find container \"04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134\": container with ID starting with 04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134 not found: ID does not exist"
Jul 09 18:06:46.771688 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.771664    2563 scope.go:117] "RemoveContainer" containerID="19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83"
Jul 09 18:06:46.771925 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:06:46.771904    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83\": container with ID starting with 19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83 not found: ID does not exist" containerID="19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83"
Jul 09 18:06:46.771995 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.771928    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83"} err="failed to get container status \"19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83\": rpc error: code = NotFound desc = could not find container \"19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83\": container with ID starting with 19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83 not found: ID does not exist"
Jul 09 18:06:46.771995 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.771944    2563 scope.go:117] "RemoveContainer" containerID="57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d"
Jul 09 18:06:46.772104 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.772085    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.772175 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:06:46.772156    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d\": container with ID starting with 57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d not found: ID does not exist" containerID="57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d"
Jul 09 18:06:46.772243 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.772182    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d"} err="failed to get container status \"57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d\": rpc error: code = NotFound desc = could not find container \"57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d\": container with ID starting with 57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d not found: ID does not exist"
Jul 09 18:06:46.772243 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.772200    2563 scope.go:117] "RemoveContainer" containerID="125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f"
Jul 09 18:06:46.772714 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:06:46.772693    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f\": container with ID starting with 125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f not found: ID does not exist" containerID="125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f"
Jul 09 18:06:46.772780 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.772718    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f"} err="failed to get container status \"125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f\": rpc error: code = NotFound desc = could not find container \"125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f\": container with ID starting with 125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f not found: ID does not exist"
Jul 09 18:06:46.772780 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.772738    2563 scope.go:117] "RemoveContainer" containerID="3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e"
Jul 09 18:06:46.773021 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:06:46.772993    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e\": container with ID starting with 3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e not found: ID does not exist" containerID="3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e"
Jul 09 18:06:46.773108 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.773027    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e"} err="failed to get container status \"3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e\": rpc error: code = NotFound desc = could not find container \"3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e\": container with ID starting with 3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e not found: ID does not exist"
Jul 09 18:06:46.773108 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.773048    2563 scope.go:117] "RemoveContainer" containerID="9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233"
Jul 09 18:06:46.773357 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:06:46.773336    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233\": container with ID starting with 9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233 not found: ID does not exist" containerID="9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233"
Jul 09 18:06:46.773449 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.773359    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233"} err="failed to get container status \"9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233\": rpc error: code = NotFound desc = could not find container \"9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233\": container with ID starting with 9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233 not found: ID does not exist"
Jul 09 18:06:46.773449 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.773377    2563 scope.go:117] "RemoveContainer" containerID="2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e"
Jul 09 18:06:46.773639 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.773622    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e"} err="failed to get container status \"2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e\": rpc error: code = NotFound desc = could not find container \"2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e\": container with ID starting with 2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e not found: ID does not exist"
Jul 09 18:06:46.773681 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.773640    2563 scope.go:117] "RemoveContainer" containerID="04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134"
Jul 09 18:06:46.773899 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.773883    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134"} err="failed to get container status \"04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134\": rpc error: code = NotFound desc = could not find container \"04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134\": container with ID starting with 04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134 not found: ID does not exist"
Jul 09 18:06:46.773941 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.773900    2563 scope.go:117] "RemoveContainer" containerID="19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83"
Jul 09 18:06:46.774113 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.774096    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83"} err="failed to get container status \"19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83\": rpc error: code = NotFound desc = could not find container \"19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83\": container with ID starting with 19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83 not found: ID does not exist"
Jul 09 18:06:46.774154 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.774113    2563 scope.go:117] "RemoveContainer" containerID="57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d"
Jul 09 18:06:46.774347 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.774326    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d"} err="failed to get container status \"57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d\": rpc error: code = NotFound desc = could not find container \"57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d\": container with ID starting with 57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d not found: ID does not exist"
Jul 09 18:06:46.774402 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.774350    2563 scope.go:117] "RemoveContainer" containerID="125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f"
Jul 09 18:06:46.774470 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.774452    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 09 18:06:46.774519 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.774459    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 09 18:06:46.774599 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.774576    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f"} err="failed to get container status \"125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f\": rpc error: code = NotFound desc = could not find container \"125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f\": container with ID starting with 125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f not found: ID does not exist"
Jul 09 18:06:46.774662 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.774602    2563 scope.go:117] "RemoveContainer" containerID="3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e"
Jul 09 18:06:46.774829 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.774798    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e"} err="failed to get container status \"3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e\": rpc error: code = NotFound desc = could not find container \"3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e\": container with ID starting with 3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e not found: ID does not exist"
Jul 09 18:06:46.774829 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.774825    2563 scope.go:117] "RemoveContainer" containerID="9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233"
Jul 09 18:06:46.775061 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.775043    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233"} err="failed to get container status \"9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233\": rpc error: code = NotFound desc = could not find container \"9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233\": container with ID starting with 9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233 not found: ID does not exist"
Jul 09 18:06:46.775106 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.775070    2563 scope.go:117] "RemoveContainer" containerID="2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e"
Jul 09 18:06:46.775365 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.775342    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e"} err="failed to get container status \"2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e\": rpc error: code = NotFound desc = could not find container \"2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e\": container with ID starting with 2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e not found: ID does not exist"
Jul 09 18:06:46.775365 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.775365    2563 scope.go:117] "RemoveContainer" containerID="04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134"
Jul 09 18:06:46.775644 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.775620    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134"} err="failed to get container status \"04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134\": rpc error: code = NotFound desc = could not find container \"04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134\": container with ID starting with 04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134 not found: ID does not exist"
Jul 09 18:06:46.775728 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.775645    2563 scope.go:117] "RemoveContainer" containerID="19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83"
Jul 09 18:06:46.775728 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.775673    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 09 18:06:46.775830 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.775797    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 09 18:06:46.776059 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.776019    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83"} err="failed to get container status \"19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83\": rpc error: code = NotFound desc = could not find container \"19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83\": container with ID starting with 19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83 not found: ID does not exist"
Jul 09 18:06:46.776059 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.776050    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 09 18:06:46.776205 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.776060    2563 scope.go:117] "RemoveContainer" containerID="57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d"
Jul 09 18:06:46.776205 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.776074    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 09 18:06:46.776549 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.776497    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 09 18:06:46.776549 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.776534    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 09 18:06:46.776709 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.776541    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d"} err="failed to get container status \"57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d\": rpc error: code = NotFound desc = could not find container \"57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d\": container with ID starting with 57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d not found: ID does not exist"
Jul 09 18:06:46.776709 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.776565    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9hj8rkhi5mk78\""
Jul 09 18:06:46.776709 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.776565    2563 scope.go:117] "RemoveContainer" containerID="125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f"
Jul 09 18:06:46.776709 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.776704    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 09 18:06:46.777075 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.776743    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 09 18:06:46.777075 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.776856    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 09 18:06:46.777075 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.776885    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f"} err="failed to get container status \"125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f\": rpc error: code = NotFound desc = could not find container \"125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f\": container with ID starting with 125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f not found: ID does not exist"
Jul 09 18:06:46.777075 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.776907    2563 scope.go:117] "RemoveContainer" containerID="3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e"
Jul 09 18:06:46.777633 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.777156    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e"} err="failed to get container status \"3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e\": rpc error: code = NotFound desc = could not find container \"3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e\": container with ID starting with 3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e not found: ID does not exist"
Jul 09 18:06:46.777633 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.777179    2563 scope.go:117] "RemoveContainer" containerID="9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233"
Jul 09 18:06:46.777633 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.777420    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233"} err="failed to get container status \"9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233\": rpc error: code = NotFound desc = could not find container \"9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233\": container with ID starting with 9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233 not found: ID does not exist"
Jul 09 18:06:46.777633 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.777441    2563 scope.go:117] "RemoveContainer" containerID="2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e"
Jul 09 18:06:46.777633 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.777497    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-g5ggg\""
Jul 09 18:06:46.777633 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.777509    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 09 18:06:46.777975 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.777859    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e"} err="failed to get container status \"2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e\": rpc error: code = NotFound desc = could not find container \"2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e\": container with ID starting with 2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e not found: ID does not exist"
Jul 09 18:06:46.777975 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.777886    2563 scope.go:117] "RemoveContainer" containerID="04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134"
Jul 09 18:06:46.778239 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.778198    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134"} err="failed to get container status \"04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134\": rpc error: code = NotFound desc = could not find container \"04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134\": container with ID starting with 04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134 not found: ID does not exist"
Jul 09 18:06:46.778334 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.778313    2563 scope.go:117] "RemoveContainer" containerID="19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83"
Jul 09 18:06:46.778608 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.778590    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83"} err="failed to get container status \"19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83\": rpc error: code = NotFound desc = could not find container \"19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83\": container with ID starting with 19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83 not found: ID does not exist"
Jul 09 18:06:46.778677 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.778610    2563 scope.go:117] "RemoveContainer" containerID="57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d"
Jul 09 18:06:46.778894 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.778859    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d"} err="failed to get container status \"57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d\": rpc error: code = NotFound desc = could not find container \"57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d\": container with ID starting with 57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d not found: ID does not exist"
Jul 09 18:06:46.778894 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.778888    2563 scope.go:117] "RemoveContainer" containerID="125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f"
Jul 09 18:06:46.779191 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.779164    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f"} err="failed to get container status \"125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f\": rpc error: code = NotFound desc = could not find container \"125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f\": container with ID starting with 125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f not found: ID does not exist"
Jul 09 18:06:46.779191 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.779191    2563 scope.go:117] "RemoveContainer" containerID="3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e"
Jul 09 18:06:46.779352 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.779321    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 09 18:06:46.779561 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.779536    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e"} err="failed to get container status \"3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e\": rpc error: code = NotFound desc = could not find container \"3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e\": container with ID starting with 3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e not found: ID does not exist"
Jul 09 18:06:46.779634 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.779562    2563 scope.go:117] "RemoveContainer" containerID="9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233"
Jul 09 18:06:46.779827 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.779802    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233"} err="failed to get container status \"9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233\": rpc error: code = NotFound desc = could not find container \"9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233\": container with ID starting with 9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233 not found: ID does not exist"
Jul 09 18:06:46.779887 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.779828    2563 scope.go:117] "RemoveContainer" containerID="2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e"
Jul 09 18:06:46.780546 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.780521    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e"} err="failed to get container status \"2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e\": rpc error: code = NotFound desc = could not find container \"2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e\": container with ID starting with 2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e not found: ID does not exist"
Jul 09 18:06:46.780618 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.780549    2563 scope.go:117] "RemoveContainer" containerID="04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134"
Jul 09 18:06:46.780805 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.780787    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134"} err="failed to get container status \"04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134\": rpc error: code = NotFound desc = could not find container \"04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134\": container with ID starting with 04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134 not found: ID does not exist"
Jul 09 18:06:46.780805 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.780805    2563 scope.go:117] "RemoveContainer" containerID="19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83"
Jul 09 18:06:46.781045 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.781025    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83"} err="failed to get container status \"19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83\": rpc error: code = NotFound desc = could not find container \"19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83\": container with ID starting with 19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83 not found: ID does not exist"
Jul 09 18:06:46.781045 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.781045    2563 scope.go:117] "RemoveContainer" containerID="57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d"
Jul 09 18:06:46.781271 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.781254    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d"} err="failed to get container status \"57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d\": rpc error: code = NotFound desc = could not find container \"57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d\": container with ID starting with 57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d not found: ID does not exist"
Jul 09 18:06:46.781315 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.781273    2563 scope.go:117] "RemoveContainer" containerID="125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f"
Jul 09 18:06:46.781465 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.781450    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f"} err="failed to get container status \"125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f\": rpc error: code = NotFound desc = could not find container \"125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f\": container with ID starting with 125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f not found: ID does not exist"
Jul 09 18:06:46.781465 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.781465    2563 scope.go:117] "RemoveContainer" containerID="3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e"
Jul 09 18:06:46.781641 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.781624    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e"} err="failed to get container status \"3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e\": rpc error: code = NotFound desc = could not find container \"3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e\": container with ID starting with 3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e not found: ID does not exist"
Jul 09 18:06:46.781682 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.781641    2563 scope.go:117] "RemoveContainer" containerID="9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233"
Jul 09 18:06:46.781848 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.781825    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233"} err="failed to get container status \"9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233\": rpc error: code = NotFound desc = could not find container \"9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233\": container with ID starting with 9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233 not found: ID does not exist"
Jul 09 18:06:46.781892 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.781852    2563 scope.go:117] "RemoveContainer" containerID="2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e"
Jul 09 18:06:46.782119 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.782099    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e"} err="failed to get container status \"2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e\": rpc error: code = NotFound desc = could not find container \"2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e\": container with ID starting with 2315d7d7dd11cfd8a7ab43e390e90cc2eb6e4c400e6e4b22d752f82b08b9953e not found: ID does not exist"
Jul 09 18:06:46.782175 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.782122    2563 scope.go:117] "RemoveContainer" containerID="04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134"
Jul 09 18:06:46.782372 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.782350    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134"} err="failed to get container status \"04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134\": rpc error: code = NotFound desc = could not find container \"04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134\": container with ID starting with 04b28dd7acf3c5b92209ee6de32daad3ff3d8bfd5ad3e78fe5a2b99db0e9c134 not found: ID does not exist"
Jul 09 18:06:46.782436 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.782374    2563 scope.go:117] "RemoveContainer" containerID="19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83"
Jul 09 18:06:46.782610 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.782588    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83"} err="failed to get container status \"19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83\": rpc error: code = NotFound desc = could not find container \"19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83\": container with ID starting with 19ab7e74db412e683626fef3234f7494439eae50affa06f2b63831a4f8795c83 not found: ID does not exist"
Jul 09 18:06:46.782663 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.782614    2563 scope.go:117] "RemoveContainer" containerID="57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d"
Jul 09 18:06:46.782891 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.782872    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d"} err="failed to get container status \"57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d\": rpc error: code = NotFound desc = could not find container \"57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d\": container with ID starting with 57ad20552a19a069a83c2658367ebb76cebfaf69b58a8f239f284ca1d626d18d not found: ID does not exist"
Jul 09 18:06:46.782938 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.782894    2563 scope.go:117] "RemoveContainer" containerID="125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f"
Jul 09 18:06:46.783138 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.783118    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f"} err="failed to get container status \"125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f\": rpc error: code = NotFound desc = could not find container \"125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f\": container with ID starting with 125b56de3abb3eff8f5a1885a8e520255a82b8cc58208890392cbd3551933a9f not found: ID does not exist"
Jul 09 18:06:46.783239 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.783139    2563 scope.go:117] "RemoveContainer" containerID="3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e"
Jul 09 18:06:46.783430 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.783379    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e"} err="failed to get container status \"3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e\": rpc error: code = NotFound desc = could not find container \"3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e\": container with ID starting with 3990e81213fba0eb126cbd813f96cad69fabe0418e7a28a8a0f8f5f0bf607a4e not found: ID does not exist"
Jul 09 18:06:46.783480 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.783431    2563 scope.go:117] "RemoveContainer" containerID="9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233"
Jul 09 18:06:46.783673 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.783657    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 09 18:06:46.783713 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.783672    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233"} err="failed to get container status \"9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233\": rpc error: code = NotFound desc = could not find container \"9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233\": container with ID starting with 9e8f86ad7fe7f9535c588c974e8d0434e4f99685b04a93c6fa8424c822ec0233 not found: ID does not exist"
Jul 09 18:06:46.788986 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.788965    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 18:06:46.844731 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.844701    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.844731 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.844743    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.844957 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.844772    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-config\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.844957 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.844856    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.844957 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.844883    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.844957 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.844908    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.844957 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.844935    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-config-out\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.844957 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.844959    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.845284 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.844986    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.845284 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.845010    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.845284 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.845026    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.845284 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.845054    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.845284 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.845122    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vn5r\" (UniqueName: \"kubernetes.io/projected/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-kube-api-access-9vn5r\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.845284 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.845156    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.845284 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.845192    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.845284 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.845235    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-web-config\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.845652 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.845293    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.845652 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.845329    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.845652 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.845355    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.845652 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.845370    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.946677 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.946643    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.946841 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.946683    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.946841 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.946701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.946841 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.946723    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.946841 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.946749    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.946841 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.946777    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-config\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.946841 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.946834    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.947209 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.946859    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.947209 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.946887    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.947209 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.946911    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-config-out\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.947209 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.946930    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.947209 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.946963    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.947209 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.946992    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.947209 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.947015    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.947209 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.947048    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.947209 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.947091    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vn5r\" (UniqueName: \"kubernetes.io/projected/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-kube-api-access-9vn5r\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.947209 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.947130    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.947209 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.947160    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.947209 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.947192    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-web-config\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.947798 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.947266    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.947798 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.947497    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.947798 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.947775    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.948324 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.948298    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.949756 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.949716    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.949852 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.949787    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.950434 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.950046    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.950434 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.950230    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.950434 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.950350    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.950727 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.950699    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.951376 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.950882    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.951376 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.951110    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-config\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.951376 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.951302    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.951376 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.951326    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.951620 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.951402    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.951897 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.951871    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.951989 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.951972    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-config-out\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.952451 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.952426    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.953198 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.953165    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.953420 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.953398    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-web-config\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.959580 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.959562    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vn5r\" (UniqueName: \"kubernetes.io/projected/86f6b712-a9d3-45ef-bd10-a2f783e76c8f-kube-api-access-9vn5r\") pod \"prometheus-k8s-0\" (UID: \"86f6b712-a9d3-45ef-bd10-a2f783e76c8f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:46.962191 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:46.962168    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a823a0e1-a248-468c-8bbf-027a46435ceb" path="/var/lib/kubelet/pods/a823a0e1-a248-468c-8bbf-027a46435ceb/volumes"
Jul 09 18:06:47.084679 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:47.084644    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:06:47.226166 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:47.226094    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 18:06:47.229335 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:06:47.229308    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86f6b712_a9d3_45ef_bd10_a2f783e76c8f.slice/crio-391e9a4d13937e78f539be8af6a156723831ca0a527372ec01d5c14875293e88 WatchSource:0}: Error finding container 391e9a4d13937e78f539be8af6a156723831ca0a527372ec01d5c14875293e88: Status 404 returned error can't find the container with id 391e9a4d13937e78f539be8af6a156723831ca0a527372ec01d5c14875293e88
Jul 09 18:06:47.710641 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:47.710606    2563 generic.go:356] "Generic (PLEG): container finished" podID="86f6b712-a9d3-45ef-bd10-a2f783e76c8f" containerID="7a404199636385566b1c4ac9874f4d6c910f655ef54bbd44e1b37d8fab9958f3" exitCode=0
Jul 09 18:06:47.711092 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:47.710687    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86f6b712-a9d3-45ef-bd10-a2f783e76c8f","Type":"ContainerDied","Data":"7a404199636385566b1c4ac9874f4d6c910f655ef54bbd44e1b37d8fab9958f3"}
Jul 09 18:06:47.711092 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:47.710778    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86f6b712-a9d3-45ef-bd10-a2f783e76c8f","Type":"ContainerStarted","Data":"391e9a4d13937e78f539be8af6a156723831ca0a527372ec01d5c14875293e88"}
Jul 09 18:06:48.717942 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:48.717900    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f77f67f4b-gkd94" event={"ID":"e1f45c65-dd77-474d-8768-4b93049bffd9","Type":"ContainerStarted","Data":"5bdaa12cc28ddc0b44ff364d617a710bf739abd2c69f4b71b559fb75ecd82651"}
Jul 09 18:06:48.721354 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:48.721328    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86f6b712-a9d3-45ef-bd10-a2f783e76c8f","Type":"ContainerStarted","Data":"ec044fbdd647056e485de17b61380d60258efe9639a60e23e3dfdac82bdc73ad"}
Jul 09 18:06:48.721502 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:48.721391    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86f6b712-a9d3-45ef-bd10-a2f783e76c8f","Type":"ContainerStarted","Data":"f34a99ca7d225ab5dd4f01bdc8d07b90aec4c569ac637fa0ec5aa1e4f99fbc0b"}
Jul 09 18:06:48.721502 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:48.721418    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86f6b712-a9d3-45ef-bd10-a2f783e76c8f","Type":"ContainerStarted","Data":"f9859e14d1db4661bd4dfd447c435be458509a790067b93553273bfe8d9721fe"}
Jul 09 18:06:48.721502 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:48.721442    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86f6b712-a9d3-45ef-bd10-a2f783e76c8f","Type":"ContainerStarted","Data":"f4a9236f773798b86deb97b9a1a03d802ca7850f82343579898a205f6098a798"}
Jul 09 18:06:48.721502 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:48.721471    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86f6b712-a9d3-45ef-bd10-a2f783e76c8f","Type":"ContainerStarted","Data":"975db9487e8d92d6dcd19bb34b04e974faf841d6d73f25911cb973003ebe4001"}
Jul 09 18:06:48.721502 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:48.721497    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86f6b712-a9d3-45ef-bd10-a2f783e76c8f","Type":"ContainerStarted","Data":"33e281d41ce3614b559b6dd6ad51c6762293a177f7131668da22d680626d1a51"}
Jul 09 18:06:48.750480 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:48.750425    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.728452515 podStartE2EDuration="2.750408354s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.955839ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:06:48.749276791 +0000 UTC m=+154.418357758" watchObservedRunningTime="2026-07-09 18:06:48.750408354 +0000 UTC m=+154.419489312"
Jul 09 18:06:49.727999 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:49.727962    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f77f67f4b-gkd94" event={"ID":"e1f45c65-dd77-474d-8768-4b93049bffd9","Type":"ContainerStarted","Data":"e4faae8ad344c50696ad1ba3abc56bb8f1d7c9636e275fb5662317234cc62717"}
Jul 09 18:06:49.728448 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:49.728020    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f77f67f4b-gkd94" event={"ID":"e1f45c65-dd77-474d-8768-4b93049bffd9","Type":"ContainerStarted","Data":"1fbb52f134d78cd7383a534be7c58218b72021946e35d1f3f14403df0b994783"}
Jul 09 18:06:49.753160 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:49.753103    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6f77f67f4b-gkd94" podStartSLOduration=2.182960749 podStartE2EDuration="4.753078983s" totalImagesPullingTime="2.570118234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:06:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:06:49.751319548 +0000 UTC m=+155.420400502" watchObservedRunningTime="2026-07-09 18:06:49.753078983 +0000 UTC m=+155.422159939"
Jul 09 18:06:52.085550 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:06:52.085510    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:07:47.085391 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:07:47.085358    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:07:47.100808 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:07:47.100784    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:07:47.973302 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:07:47.973270    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:09:14.781203 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:09:14.781171    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbftk_5d463fb2-45f9-4554-9525-637e1b84351a/ovn-acl-logging/0.log"
Jul 09 18:09:14.787687 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:09:14.787665    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbftk_5d463fb2-45f9-4554-9525-637e1b84351a/ovn-acl-logging/0.log"
Jul 09 18:09:14.788236 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:09:14.788198    2563 kubelet.go:1648] "Image garbage collection succeeded"
Jul 09 18:12:19.886610 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:12:19.886576    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-gbdw7"]
Jul 09 18:12:19.888833 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:12:19.888814    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-gbdw7"
Jul 09 18:12:19.891347 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:12:19.891323    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 09 18:12:19.891496 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:12:19.891398    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 09 18:12:19.891496 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:12:19.891398    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 09 18:12:19.892048 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:12:19.892017    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-mdfvk\""
Jul 09 18:12:19.899131 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:12:19.899103    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-gbdw7"]
Jul 09 18:12:19.912352 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:12:19.912321    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxt8h\" (UniqueName: \"kubernetes.io/projected/0abbe120-9dc5-4092-87c0-b0d9bec20c5c-kube-api-access-hxt8h\") pod \"seaweedfs-64568bcd49-gbdw7\" (UID: \"0abbe120-9dc5-4092-87c0-b0d9bec20c5c\") " pod="kserve/seaweedfs-64568bcd49-gbdw7"
Jul 09 18:12:19.912507 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:12:19.912387    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0abbe120-9dc5-4092-87c0-b0d9bec20c5c-data\") pod \"seaweedfs-64568bcd49-gbdw7\" (UID: \"0abbe120-9dc5-4092-87c0-b0d9bec20c5c\") " pod="kserve/seaweedfs-64568bcd49-gbdw7"
Jul 09 18:12:20.013097 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:12:20.013066    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxt8h\" (UniqueName: \"kubernetes.io/projected/0abbe120-9dc5-4092-87c0-b0d9bec20c5c-kube-api-access-hxt8h\") pod \"seaweedfs-64568bcd49-gbdw7\" (UID: \"0abbe120-9dc5-4092-87c0-b0d9bec20c5c\") " pod="kserve/seaweedfs-64568bcd49-gbdw7"
Jul 09 18:12:20.013314 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:12:20.013111    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0abbe120-9dc5-4092-87c0-b0d9bec20c5c-data\") pod \"seaweedfs-64568bcd49-gbdw7\" (UID: \"0abbe120-9dc5-4092-87c0-b0d9bec20c5c\") " pod="kserve/seaweedfs-64568bcd49-gbdw7"
Jul 09 18:12:20.013477 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:12:20.013461    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0abbe120-9dc5-4092-87c0-b0d9bec20c5c-data\") pod \"seaweedfs-64568bcd49-gbdw7\" (UID: \"0abbe120-9dc5-4092-87c0-b0d9bec20c5c\") " pod="kserve/seaweedfs-64568bcd49-gbdw7"
Jul 09 18:12:20.021489 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:12:20.021467    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxt8h\" (UniqueName: \"kubernetes.io/projected/0abbe120-9dc5-4092-87c0-b0d9bec20c5c-kube-api-access-hxt8h\") pod \"seaweedfs-64568bcd49-gbdw7\" (UID: \"0abbe120-9dc5-4092-87c0-b0d9bec20c5c\") " pod="kserve/seaweedfs-64568bcd49-gbdw7"
Jul 09 18:12:20.198678 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:12:20.198653    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-gbdw7"
Jul 09 18:12:20.321982 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:12:20.321953    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-gbdw7"]
Jul 09 18:12:20.325103 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:12:20.325071    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0abbe120_9dc5_4092_87c0_b0d9bec20c5c.slice/crio-93c058cfc434999a645e594e5baef241e57dbebb0ba1a0536d80cb9913f763b6 WatchSource:0}: Error finding container 93c058cfc434999a645e594e5baef241e57dbebb0ba1a0536d80cb9913f763b6: Status 404 returned error can't find the container with id 93c058cfc434999a645e594e5baef241e57dbebb0ba1a0536d80cb9913f763b6
Jul 09 18:12:20.326580 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:12:20.326558    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:12:21.065281 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:12:21.065249    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-gbdw7" event={"ID":"0abbe120-9dc5-4092-87c0-b0d9bec20c5c","Type":"ContainerStarted","Data":"93c058cfc434999a645e594e5baef241e57dbebb0ba1a0536d80cb9913f763b6"}
Jul 09 18:12:24.079852 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:12:24.079822    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-gbdw7" event={"ID":"0abbe120-9dc5-4092-87c0-b0d9bec20c5c","Type":"ContainerStarted","Data":"b0ebd58cd33e0e34c5dd0f1747e7a9378c1dcd5a85698058232291690297ccda"}
Jul 09 18:12:24.080275 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:12:24.080046    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-gbdw7"
Jul 09 18:12:24.098024 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:12:24.097985    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-gbdw7" podStartSLOduration=1.773018475 podStartE2EDuration="5.097971376s" totalImagesPullingTime="3.324952901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:12:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:12:24.097035223 +0000 UTC m=+489.766116191" watchObservedRunningTime="2026-07-09 18:12:24.097971376 +0000 UTC m=+489.767052330"
Jul 09 18:12:30.086664 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:12:30.086634    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-gbdw7"
Jul 09 18:13:32.389258 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:13:32.389225    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-ftnd9"]
Jul 09 18:13:32.392766 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:13:32.392745    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-ftnd9"
Jul 09 18:13:32.395460 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:13:32.395440    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 09 18:13:32.395596 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:13:32.395578    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-wb78x\""
Jul 09 18:13:32.402324 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:13:32.402300    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-ftnd9"]
Jul 09 18:13:32.547829 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:13:32.547790    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8c3dc45-9521-401c-a070-bb3c2114fbbc-cert\") pod \"odh-model-controller-84b54b7f97-ftnd9\" (UID: \"d8c3dc45-9521-401c-a070-bb3c2114fbbc\") " pod="kserve/odh-model-controller-84b54b7f97-ftnd9"
Jul 09 18:13:32.548002 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:13:32.547853    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwhdk\" (UniqueName: \"kubernetes.io/projected/d8c3dc45-9521-401c-a070-bb3c2114fbbc-kube-api-access-kwhdk\") pod \"odh-model-controller-84b54b7f97-ftnd9\" (UID: \"d8c3dc45-9521-401c-a070-bb3c2114fbbc\") " pod="kserve/odh-model-controller-84b54b7f97-ftnd9"
Jul 09 18:13:32.648430 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:13:32.648339    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8c3dc45-9521-401c-a070-bb3c2114fbbc-cert\") pod \"odh-model-controller-84b54b7f97-ftnd9\" (UID: \"d8c3dc45-9521-401c-a070-bb3c2114fbbc\") " pod="kserve/odh-model-controller-84b54b7f97-ftnd9"
Jul 09 18:13:32.648430 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:13:32.648415    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwhdk\" (UniqueName: \"kubernetes.io/projected/d8c3dc45-9521-401c-a070-bb3c2114fbbc-kube-api-access-kwhdk\") pod \"odh-model-controller-84b54b7f97-ftnd9\" (UID: \"d8c3dc45-9521-401c-a070-bb3c2114fbbc\") " pod="kserve/odh-model-controller-84b54b7f97-ftnd9"
Jul 09 18:13:32.650716 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:13:32.650693    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8c3dc45-9521-401c-a070-bb3c2114fbbc-cert\") pod \"odh-model-controller-84b54b7f97-ftnd9\" (UID: \"d8c3dc45-9521-401c-a070-bb3c2114fbbc\") " pod="kserve/odh-model-controller-84b54b7f97-ftnd9"
Jul 09 18:13:32.657832 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:13:32.657807    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwhdk\" (UniqueName: \"kubernetes.io/projected/d8c3dc45-9521-401c-a070-bb3c2114fbbc-kube-api-access-kwhdk\") pod \"odh-model-controller-84b54b7f97-ftnd9\" (UID: \"d8c3dc45-9521-401c-a070-bb3c2114fbbc\") " pod="kserve/odh-model-controller-84b54b7f97-ftnd9"
Jul 09 18:13:32.705819 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:13:32.705787    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-ftnd9"
Jul 09 18:13:32.834605 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:13:32.834573    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-ftnd9"]
Jul 09 18:13:32.837632 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:13:32.837607    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c3dc45_9521_401c_a070_bb3c2114fbbc.slice/crio-8bb3d90445c3a3cd699f9642dcf848f4c7d791aba600aa2ea44a9ba205150416 WatchSource:0}: Error finding container 8bb3d90445c3a3cd699f9642dcf848f4c7d791aba600aa2ea44a9ba205150416: Status 404 returned error can't find the container with id 8bb3d90445c3a3cd699f9642dcf848f4c7d791aba600aa2ea44a9ba205150416
Jul 09 18:13:33.355308 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:13:33.355269    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-ftnd9" event={"ID":"d8c3dc45-9521-401c-a070-bb3c2114fbbc","Type":"ContainerStarted","Data":"8bb3d90445c3a3cd699f9642dcf848f4c7d791aba600aa2ea44a9ba205150416"}
Jul 09 18:13:36.369490 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:13:36.369401    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-ftnd9" event={"ID":"d8c3dc45-9521-401c-a070-bb3c2114fbbc","Type":"ContainerStarted","Data":"a55721b443a03a67759a75e584e6a7250c0469b7b179d1b723c70a4fc8eab036"}
Jul 09 18:13:36.369490 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:13:36.369465    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-ftnd9"
Jul 09 18:13:36.391253 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:13:36.391180    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-ftnd9" podStartSLOduration=1.276720381 podStartE2EDuration="4.391158711s" totalImagesPullingTime="3.11443833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:13:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:13:36.389723027 +0000 UTC m=+562.058804000" watchObservedRunningTime="2026-07-09 18:13:36.391158711 +0000 UTC m=+562.060239667"
Jul 09 18:13:47.376182 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:13:47.376154    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-ftnd9"
Jul 09 18:13:48.281684 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:13:48.281650    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-czmts"]
Jul 09 18:13:48.284801 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:13:48.284785    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-czmts"
Jul 09 18:13:48.302752 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:13:48.302722    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-czmts"]
Jul 09 18:13:48.379097 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:13:48.379068    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwxqm\" (UniqueName: \"kubernetes.io/projected/8ff484ff-f431-4f3d-9df8-a318387eb715-kube-api-access-hwxqm\") pod \"s3-init-czmts\" (UID: \"8ff484ff-f431-4f3d-9df8-a318387eb715\") " pod="kserve/s3-init-czmts"
Jul 09 18:13:48.379463 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:13:48.379124    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ff484ff-f431-4f3d-9df8-a318387eb715-model-cache\") pod \"s3-init-czmts\" (UID: \"8ff484ff-f431-4f3d-9df8-a318387eb715\") " pod="kserve/s3-init-czmts"
Jul 09 18:13:48.480515 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:13:48.480481    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwxqm\" (UniqueName: \"kubernetes.io/projected/8ff484ff-f431-4f3d-9df8-a318387eb715-kube-api-access-hwxqm\") pod \"s3-init-czmts\" (UID: \"8ff484ff-f431-4f3d-9df8-a318387eb715\") " pod="kserve/s3-init-czmts"
Jul 09 18:13:48.480683 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:13:48.480536    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ff484ff-f431-4f3d-9df8-a318387eb715-model-cache\") pod \"s3-init-czmts\" (UID: \"8ff484ff-f431-4f3d-9df8-a318387eb715\") " pod="kserve/s3-init-czmts"
Jul 09 18:13:48.480871 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:13:48.480857    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ff484ff-f431-4f3d-9df8-a318387eb715-model-cache\") pod \"s3-init-czmts\" (UID: \"8ff484ff-f431-4f3d-9df8-a318387eb715\") " pod="kserve/s3-init-czmts"
Jul 09 18:13:48.489238 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:13:48.489191    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwxqm\" (UniqueName: \"kubernetes.io/projected/8ff484ff-f431-4f3d-9df8-a318387eb715-kube-api-access-hwxqm\") pod \"s3-init-czmts\" (UID: \"8ff484ff-f431-4f3d-9df8-a318387eb715\") " pod="kserve/s3-init-czmts"
Jul 09 18:13:48.619600 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:13:48.619517    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-czmts"
Jul 09 18:13:48.763087 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:13:48.763058    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-czmts"]
Jul 09 18:13:48.766378 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:13:48.766351    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ff484ff_f431_4f3d_9df8_a318387eb715.slice/crio-0d910da4ebd903d5f68748d2453b54409bd559a32e109e50fde2957d46676b3a WatchSource:0}: Error finding container 0d910da4ebd903d5f68748d2453b54409bd559a32e109e50fde2957d46676b3a: Status 404 returned error can't find the container with id 0d910da4ebd903d5f68748d2453b54409bd559a32e109e50fde2957d46676b3a
Jul 09 18:13:49.426105 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:13:49.426061    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-czmts" event={"ID":"8ff484ff-f431-4f3d-9df8-a318387eb715","Type":"ContainerStarted","Data":"0d910da4ebd903d5f68748d2453b54409bd559a32e109e50fde2957d46676b3a"}
Jul 09 18:13:53.446816 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:13:53.446768    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-czmts" event={"ID":"8ff484ff-f431-4f3d-9df8-a318387eb715","Type":"ContainerStarted","Data":"b016e4e1cceeeb40ea61b84d9c6937a1b7db9786e1067ab00d96b80dfb6b05f4"}
Jul 09 18:13:54.451999 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:13:54.451961    2563 generic.go:356] "Generic (PLEG): container finished" podID="8ff484ff-f431-4f3d-9df8-a318387eb715" containerID="b016e4e1cceeeb40ea61b84d9c6937a1b7db9786e1067ab00d96b80dfb6b05f4" exitCode=0
Jul 09 18:13:54.452378 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:13:54.452013    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-czmts" event={"ID":"8ff484ff-f431-4f3d-9df8-a318387eb715","Type":"ContainerDied","Data":"b016e4e1cceeeb40ea61b84d9c6937a1b7db9786e1067ab00d96b80dfb6b05f4"}
Jul 09 18:13:59.475115 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:13:59.475083    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-czmts" event={"ID":"8ff484ff-f431-4f3d-9df8-a318387eb715","Type":"ContainerStarted","Data":"e25d11ed976cb0427a50d956500d86fc65142a33a04de9dfd68a00c6c21bee54"}
Jul 09 18:14:14.812512 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:14:14.812482    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbftk_5d463fb2-45f9-4554-9525-637e1b84351a/ovn-acl-logging/0.log"
Jul 09 18:14:14.814136 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:14:14.814115    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbftk_5d463fb2-45f9-4554-9525-637e1b84351a/ovn-acl-logging/0.log"
Jul 09 18:15:04.744984 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:04.744951    2563 generic.go:356] "Generic (PLEG): container finished" podID="8ff484ff-f431-4f3d-9df8-a318387eb715" containerID="e25d11ed976cb0427a50d956500d86fc65142a33a04de9dfd68a00c6c21bee54" exitCode=0
Jul 09 18:15:04.745405 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:04.744994    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-czmts" event={"ID":"8ff484ff-f431-4f3d-9df8-a318387eb715","Type":"ContainerDied","Data":"e25d11ed976cb0427a50d956500d86fc65142a33a04de9dfd68a00c6c21bee54"}
Jul 09 18:15:05.878416 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:05.878389    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-czmts"
Jul 09 18:15:05.965885 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:05.965851    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ff484ff-f431-4f3d-9df8-a318387eb715-model-cache\") pod \"8ff484ff-f431-4f3d-9df8-a318387eb715\" (UID: \"8ff484ff-f431-4f3d-9df8-a318387eb715\") "
Jul 09 18:15:05.966080 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:05.965910    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwxqm\" (UniqueName: \"kubernetes.io/projected/8ff484ff-f431-4f3d-9df8-a318387eb715-kube-api-access-hwxqm\") pod \"8ff484ff-f431-4f3d-9df8-a318387eb715\" (UID: \"8ff484ff-f431-4f3d-9df8-a318387eb715\") "
Jul 09 18:15:05.968039 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:05.968011    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ff484ff-f431-4f3d-9df8-a318387eb715-kube-api-access-hwxqm" (OuterVolumeSpecName: "kube-api-access-hwxqm") pod "8ff484ff-f431-4f3d-9df8-a318387eb715" (UID: "8ff484ff-f431-4f3d-9df8-a318387eb715"). InnerVolumeSpecName "kube-api-access-hwxqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:15:05.984250 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:05.984200    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ff484ff-f431-4f3d-9df8-a318387eb715-model-cache" (OuterVolumeSpecName: "model-cache") pod "8ff484ff-f431-4f3d-9df8-a318387eb715" (UID: "8ff484ff-f431-4f3d-9df8-a318387eb715"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:15:06.067534 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:06.067445    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ff484ff-f431-4f3d-9df8-a318387eb715-model-cache\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:15:06.067534 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:06.067475    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwxqm\" (UniqueName: \"kubernetes.io/projected/8ff484ff-f431-4f3d-9df8-a318387eb715-kube-api-access-hwxqm\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:15:06.754204 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:06.754174    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-czmts"
Jul 09 18:15:06.754204 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:06.754188    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-czmts" event={"ID":"8ff484ff-f431-4f3d-9df8-a318387eb715","Type":"ContainerDied","Data":"0d910da4ebd903d5f68748d2453b54409bd559a32e109e50fde2957d46676b3a"}
Jul 09 18:15:06.754204 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:06.754236    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d910da4ebd903d5f68748d2453b54409bd559a32e109e50fde2957d46676b3a"
Jul 09 18:15:19.265511 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:19.265481    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-45wf6"]
Jul 09 18:15:19.265889 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:19.265824    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ff484ff-f431-4f3d-9df8-a318387eb715" containerName="s3-init"
Jul 09 18:15:19.265889 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:19.265835    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ff484ff-f431-4f3d-9df8-a318387eb715" containerName="s3-init"
Jul 09 18:15:19.265889 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:19.265847    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ff484ff-f431-4f3d-9df8-a318387eb715" containerName="download-hf-model"
Jul 09 18:15:19.265889 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:19.265853    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ff484ff-f431-4f3d-9df8-a318387eb715" containerName="download-hf-model"
Jul 09 18:15:19.266017 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:19.265903    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ff484ff-f431-4f3d-9df8-a318387eb715" containerName="s3-init"
Jul 09 18:15:19.268296 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:19.268281    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-45wf6"
Jul 09 18:15:19.270559 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:19.270541    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 09 18:15:19.277691 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:19.277668    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-45wf6"]
Jul 09 18:15:19.281182 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:19.281159    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2795f7e9-d85b-49de-aa43-a20d2caf463c-model-cache\") pod \"s3-tls-init-custom-45wf6\" (UID: \"2795f7e9-d85b-49de-aa43-a20d2caf463c\") " pod="kserve/s3-tls-init-custom-45wf6"
Jul 09 18:15:19.281285 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:19.281193    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xmgj\" (UniqueName: \"kubernetes.io/projected/2795f7e9-d85b-49de-aa43-a20d2caf463c-kube-api-access-5xmgj\") pod \"s3-tls-init-custom-45wf6\" (UID: \"2795f7e9-d85b-49de-aa43-a20d2caf463c\") " pod="kserve/s3-tls-init-custom-45wf6"
Jul 09 18:15:19.382622 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:19.382589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2795f7e9-d85b-49de-aa43-a20d2caf463c-model-cache\") pod \"s3-tls-init-custom-45wf6\" (UID: \"2795f7e9-d85b-49de-aa43-a20d2caf463c\") " pod="kserve/s3-tls-init-custom-45wf6"
Jul 09 18:15:19.382792 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:19.382628    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xmgj\" (UniqueName: \"kubernetes.io/projected/2795f7e9-d85b-49de-aa43-a20d2caf463c-kube-api-access-5xmgj\") pod \"s3-tls-init-custom-45wf6\" (UID: \"2795f7e9-d85b-49de-aa43-a20d2caf463c\") " pod="kserve/s3-tls-init-custom-45wf6"
Jul 09 18:15:19.382964 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:19.382948    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2795f7e9-d85b-49de-aa43-a20d2caf463c-model-cache\") pod \"s3-tls-init-custom-45wf6\" (UID: \"2795f7e9-d85b-49de-aa43-a20d2caf463c\") " pod="kserve/s3-tls-init-custom-45wf6"
Jul 09 18:15:19.391235 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:19.391198    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xmgj\" (UniqueName: \"kubernetes.io/projected/2795f7e9-d85b-49de-aa43-a20d2caf463c-kube-api-access-5xmgj\") pod \"s3-tls-init-custom-45wf6\" (UID: \"2795f7e9-d85b-49de-aa43-a20d2caf463c\") " pod="kserve/s3-tls-init-custom-45wf6"
Jul 09 18:15:19.589052 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:19.588968    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-45wf6"
Jul 09 18:15:19.713688 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:19.713659    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-45wf6"]
Jul 09 18:15:19.717269 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:15:19.717245    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2795f7e9_d85b_49de_aa43_a20d2caf463c.slice/crio-091f9cca46fd0267b4de51cc36fc07a7983c46976e2bbfab89434967d840af23 WatchSource:0}: Error finding container 091f9cca46fd0267b4de51cc36fc07a7983c46976e2bbfab89434967d840af23: Status 404 returned error can't find the container with id 091f9cca46fd0267b4de51cc36fc07a7983c46976e2bbfab89434967d840af23
Jul 09 18:15:19.808207 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:19.808167    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-45wf6" event={"ID":"2795f7e9-d85b-49de-aa43-a20d2caf463c","Type":"ContainerStarted","Data":"040753785ecdcc41a4b65c5e9c8ab28aa5e3f3c98f4ab4ecdfa5e1f24b114b96"}
Jul 09 18:15:19.808405 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:19.808239    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-45wf6" event={"ID":"2795f7e9-d85b-49de-aa43-a20d2caf463c","Type":"ContainerStarted","Data":"091f9cca46fd0267b4de51cc36fc07a7983c46976e2bbfab89434967d840af23"}
Jul 09 18:15:21.817465 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:21.817434    2563 generic.go:356] "Generic (PLEG): container finished" podID="2795f7e9-d85b-49de-aa43-a20d2caf463c" containerID="040753785ecdcc41a4b65c5e9c8ab28aa5e3f3c98f4ab4ecdfa5e1f24b114b96" exitCode=0
Jul 09 18:15:21.817835 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:21.817506    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-45wf6" event={"ID":"2795f7e9-d85b-49de-aa43-a20d2caf463c","Type":"ContainerDied","Data":"040753785ecdcc41a4b65c5e9c8ab28aa5e3f3c98f4ab4ecdfa5e1f24b114b96"}
Jul 09 18:15:22.829520 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:22.829485    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-45wf6" event={"ID":"2795f7e9-d85b-49de-aa43-a20d2caf463c","Type":"ContainerStarted","Data":"f5298afb474dba5c02928a5f6488ab44d1c6936b1a76d4eb83c96a6d5ccded69"}
Jul 09 18:15:22.850450 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:22.850395    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-45wf6" podStartSLOduration=2.301082153 podStartE2EDuration="3.850378851s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.549296698s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:15:22.849782503 +0000 UTC m=+668.518863460" watchObservedRunningTime="2026-07-09 18:15:22.850378851 +0000 UTC m=+668.519459805"
Jul 09 18:15:27.855359 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:27.855273    2563 generic.go:356] "Generic (PLEG): container finished" podID="2795f7e9-d85b-49de-aa43-a20d2caf463c" containerID="f5298afb474dba5c02928a5f6488ab44d1c6936b1a76d4eb83c96a6d5ccded69" exitCode=0
Jul 09 18:15:27.855730 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:27.855350    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-45wf6" event={"ID":"2795f7e9-d85b-49de-aa43-a20d2caf463c","Type":"ContainerDied","Data":"f5298afb474dba5c02928a5f6488ab44d1c6936b1a76d4eb83c96a6d5ccded69"}
Jul 09 18:15:28.997468 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:28.997446    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-45wf6"
Jul 09 18:15:29.068838 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:29.068801    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2795f7e9-d85b-49de-aa43-a20d2caf463c-model-cache\") pod \"2795f7e9-d85b-49de-aa43-a20d2caf463c\" (UID: \"2795f7e9-d85b-49de-aa43-a20d2caf463c\") "
Jul 09 18:15:29.069007 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:29.068854    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xmgj\" (UniqueName: \"kubernetes.io/projected/2795f7e9-d85b-49de-aa43-a20d2caf463c-kube-api-access-5xmgj\") pod \"2795f7e9-d85b-49de-aa43-a20d2caf463c\" (UID: \"2795f7e9-d85b-49de-aa43-a20d2caf463c\") "
Jul 09 18:15:29.071447 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:29.071420    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2795f7e9-d85b-49de-aa43-a20d2caf463c-kube-api-access-5xmgj" (OuterVolumeSpecName: "kube-api-access-5xmgj") pod "2795f7e9-d85b-49de-aa43-a20d2caf463c" (UID: "2795f7e9-d85b-49de-aa43-a20d2caf463c"). InnerVolumeSpecName "kube-api-access-5xmgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:15:29.091278 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:29.091241    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2795f7e9-d85b-49de-aa43-a20d2caf463c-model-cache" (OuterVolumeSpecName: "model-cache") pod "2795f7e9-d85b-49de-aa43-a20d2caf463c" (UID: "2795f7e9-d85b-49de-aa43-a20d2caf463c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:15:29.169601 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:29.169572    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2795f7e9-d85b-49de-aa43-a20d2caf463c-model-cache\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:15:29.169601 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:29.169601    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xmgj\" (UniqueName: \"kubernetes.io/projected/2795f7e9-d85b-49de-aa43-a20d2caf463c-kube-api-access-5xmgj\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:15:29.865448 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:29.865422    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-45wf6"
Jul 09 18:15:29.865606 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:29.865418    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-45wf6" event={"ID":"2795f7e9-d85b-49de-aa43-a20d2caf463c","Type":"ContainerDied","Data":"091f9cca46fd0267b4de51cc36fc07a7983c46976e2bbfab89434967d840af23"}
Jul 09 18:15:29.865606 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:29.865532    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="091f9cca46fd0267b4de51cc36fc07a7983c46976e2bbfab89434967d840af23"
Jul 09 18:15:32.567854 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:32.567825    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-njsm4"]
Jul 09 18:15:32.568263 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:32.568228    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2795f7e9-d85b-49de-aa43-a20d2caf463c" containerName="download-hf-model"
Jul 09 18:15:32.568263 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:32.568243    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2795f7e9-d85b-49de-aa43-a20d2caf463c" containerName="download-hf-model"
Jul 09 18:15:32.568263 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:32.568259    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2795f7e9-d85b-49de-aa43-a20d2caf463c" containerName="s3-tls-init-custom"
Jul 09 18:15:32.568263 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:32.568264    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2795f7e9-d85b-49de-aa43-a20d2caf463c" containerName="s3-tls-init-custom"
Jul 09 18:15:32.568401 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:32.568336    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2795f7e9-d85b-49de-aa43-a20d2caf463c" containerName="s3-tls-init-custom"
Jul 09 18:15:32.570921 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:32.570904    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-njsm4"
Jul 09 18:15:32.573390 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:32.573371    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 09 18:15:32.580497 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:32.580472    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-njsm4"]
Jul 09 18:15:32.597648 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:32.597619    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c6bed38-10d2-4e48-9db3-42fd880f9543-model-cache\") pod \"s3-tls-init-serving-njsm4\" (UID: \"8c6bed38-10d2-4e48-9db3-42fd880f9543\") " pod="kserve/s3-tls-init-serving-njsm4"
Jul 09 18:15:32.597804 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:32.597657    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv6br\" (UniqueName: \"kubernetes.io/projected/8c6bed38-10d2-4e48-9db3-42fd880f9543-kube-api-access-pv6br\") pod \"s3-tls-init-serving-njsm4\" (UID: \"8c6bed38-10d2-4e48-9db3-42fd880f9543\") " pod="kserve/s3-tls-init-serving-njsm4"
Jul 09 18:15:32.698443 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:32.698406    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c6bed38-10d2-4e48-9db3-42fd880f9543-model-cache\") pod \"s3-tls-init-serving-njsm4\" (UID: \"8c6bed38-10d2-4e48-9db3-42fd880f9543\") " pod="kserve/s3-tls-init-serving-njsm4"
Jul 09 18:15:32.698443 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:32.698443    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv6br\" (UniqueName: \"kubernetes.io/projected/8c6bed38-10d2-4e48-9db3-42fd880f9543-kube-api-access-pv6br\") pod \"s3-tls-init-serving-njsm4\" (UID: \"8c6bed38-10d2-4e48-9db3-42fd880f9543\") " pod="kserve/s3-tls-init-serving-njsm4"
Jul 09 18:15:32.698828 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:32.698796    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c6bed38-10d2-4e48-9db3-42fd880f9543-model-cache\") pod \"s3-tls-init-serving-njsm4\" (UID: \"8c6bed38-10d2-4e48-9db3-42fd880f9543\") " pod="kserve/s3-tls-init-serving-njsm4"
Jul 09 18:15:32.707146 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:32.707119    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv6br\" (UniqueName: \"kubernetes.io/projected/8c6bed38-10d2-4e48-9db3-42fd880f9543-kube-api-access-pv6br\") pod \"s3-tls-init-serving-njsm4\" (UID: \"8c6bed38-10d2-4e48-9db3-42fd880f9543\") " pod="kserve/s3-tls-init-serving-njsm4"
Jul 09 18:15:32.887095 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:32.887019    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-njsm4"
Jul 09 18:15:33.017743 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:33.017717    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-njsm4"]
Jul 09 18:15:33.020293 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:15:33.020266    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c6bed38_10d2_4e48_9db3_42fd880f9543.slice/crio-b80d8c352ae6491b9ca23a1b818188b1c2a1e0f7b26070211ecf4f08b8391dc4 WatchSource:0}: Error finding container b80d8c352ae6491b9ca23a1b818188b1c2a1e0f7b26070211ecf4f08b8391dc4: Status 404 returned error can't find the container with id b80d8c352ae6491b9ca23a1b818188b1c2a1e0f7b26070211ecf4f08b8391dc4
Jul 09 18:15:33.884935 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:33.884899    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-njsm4" event={"ID":"8c6bed38-10d2-4e48-9db3-42fd880f9543","Type":"ContainerStarted","Data":"ed85b6b9d6423b4bdc878af487978e55b3b25cce70be038a634d9f1034a476d3"}
Jul 09 18:15:33.884935 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:33.884940    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-njsm4" event={"ID":"8c6bed38-10d2-4e48-9db3-42fd880f9543","Type":"ContainerStarted","Data":"b80d8c352ae6491b9ca23a1b818188b1c2a1e0f7b26070211ecf4f08b8391dc4"}
Jul 09 18:15:34.890465 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:34.890430    2563 generic.go:356] "Generic (PLEG): container finished" podID="8c6bed38-10d2-4e48-9db3-42fd880f9543" containerID="ed85b6b9d6423b4bdc878af487978e55b3b25cce70be038a634d9f1034a476d3" exitCode=0
Jul 09 18:15:34.890854 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:34.890489    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-njsm4" event={"ID":"8c6bed38-10d2-4e48-9db3-42fd880f9543","Type":"ContainerDied","Data":"ed85b6b9d6423b4bdc878af487978e55b3b25cce70be038a634d9f1034a476d3"}
Jul 09 18:15:35.896944 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:35.896906    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-njsm4" event={"ID":"8c6bed38-10d2-4e48-9db3-42fd880f9543","Type":"ContainerStarted","Data":"43bcb821f113e1b4dfe6860ecbb24349b0c94308ea441716b440072510ce0924"}
Jul 09 18:15:35.914828 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:35.914769    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-njsm4" podStartSLOduration=2.211851704 podStartE2EDuration="3.914752744s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.70290104s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:15:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:15:35.913762483 +0000 UTC m=+681.582843459" watchObservedRunningTime="2026-07-09 18:15:35.914752744 +0000 UTC m=+681.583833698"
Jul 09 18:15:42.929883 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:42.929843    2563 generic.go:356] "Generic (PLEG): container finished" podID="8c6bed38-10d2-4e48-9db3-42fd880f9543" containerID="43bcb821f113e1b4dfe6860ecbb24349b0c94308ea441716b440072510ce0924" exitCode=0
Jul 09 18:15:42.930286 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:42.929908    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-njsm4" event={"ID":"8c6bed38-10d2-4e48-9db3-42fd880f9543","Type":"ContainerDied","Data":"43bcb821f113e1b4dfe6860ecbb24349b0c94308ea441716b440072510ce0924"}
Jul 09 18:15:44.089380 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:44.089357    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-njsm4"
Jul 09 18:15:44.201651 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:44.201568    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pv6br\" (UniqueName: \"kubernetes.io/projected/8c6bed38-10d2-4e48-9db3-42fd880f9543-kube-api-access-pv6br\") pod \"8c6bed38-10d2-4e48-9db3-42fd880f9543\" (UID: \"8c6bed38-10d2-4e48-9db3-42fd880f9543\") "
Jul 09 18:15:44.201651 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:44.201624    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c6bed38-10d2-4e48-9db3-42fd880f9543-model-cache\") pod \"8c6bed38-10d2-4e48-9db3-42fd880f9543\" (UID: \"8c6bed38-10d2-4e48-9db3-42fd880f9543\") "
Jul 09 18:15:44.203595 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:44.203563    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c6bed38-10d2-4e48-9db3-42fd880f9543-kube-api-access-pv6br" (OuterVolumeSpecName: "kube-api-access-pv6br") pod "8c6bed38-10d2-4e48-9db3-42fd880f9543" (UID: "8c6bed38-10d2-4e48-9db3-42fd880f9543"). InnerVolumeSpecName "kube-api-access-pv6br". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:15:44.224678 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:44.224648    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c6bed38-10d2-4e48-9db3-42fd880f9543-model-cache" (OuterVolumeSpecName: "model-cache") pod "8c6bed38-10d2-4e48-9db3-42fd880f9543" (UID: "8c6bed38-10d2-4e48-9db3-42fd880f9543"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:15:44.303138 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:44.303106    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c6bed38-10d2-4e48-9db3-42fd880f9543-model-cache\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:15:44.303138 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:44.303133    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv6br\" (UniqueName: \"kubernetes.io/projected/8c6bed38-10d2-4e48-9db3-42fd880f9543-kube-api-access-pv6br\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:15:44.939700 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:44.939672    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-njsm4"
Jul 09 18:15:44.939884 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:44.939665    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-njsm4" event={"ID":"8c6bed38-10d2-4e48-9db3-42fd880f9543","Type":"ContainerDied","Data":"b80d8c352ae6491b9ca23a1b818188b1c2a1e0f7b26070211ecf4f08b8391dc4"}
Jul 09 18:15:44.939884 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:44.939788    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b80d8c352ae6491b9ca23a1b818188b1c2a1e0f7b26070211ecf4f08b8391dc4"
Jul 09 18:15:56.661925 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:56.661893    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts"]
Jul 09 18:15:56.662309 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:56.662266    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c6bed38-10d2-4e48-9db3-42fd880f9543" containerName="s3-tls-init-serving"
Jul 09 18:15:56.662309 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:56.662284    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c6bed38-10d2-4e48-9db3-42fd880f9543" containerName="s3-tls-init-serving"
Jul 09 18:15:56.662309 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:56.662297    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c6bed38-10d2-4e48-9db3-42fd880f9543" containerName="download-hf-model"
Jul 09 18:15:56.662309 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:56.662303    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c6bed38-10d2-4e48-9db3-42fd880f9543" containerName="download-hf-model"
Jul 09 18:15:56.662439 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:56.662357    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c6bed38-10d2-4e48-9db3-42fd880f9543" containerName="s3-tls-init-serving"
Jul 09 18:15:56.667124 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:56.667108    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts"
Jul 09 18:15:56.669685 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:56.669660    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:15:56.669835 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:56.669770    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 09 18:15:56.669921 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:56.669904    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tx5df\""
Jul 09 18:15:56.669985 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:56.669910    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:15:56.670047 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:56.670007    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:15:56.670139 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:56.670123    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 09 18:15:56.670189 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:56.670146    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 18:15:56.677395 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:56.677374    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts"]
Jul 09 18:15:56.817714 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:56.817670    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kggd\" (UniqueName: \"kubernetes.io/projected/3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c-kube-api-access-5kggd\") pod \"isvc-sklearn-batcher-predictor-8cc45868-t8wts\" (UID: \"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts"
Jul 09 18:15:56.817903 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:56.817736    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-8cc45868-t8wts\" (UID: \"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts"
Jul 09 18:15:56.817903 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:56.817803    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-8cc45868-t8wts\" (UID: \"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts"
Jul 09 18:15:56.817903 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:56.817834    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-8cc45868-t8wts\" (UID: \"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts"
Jul 09 18:15:56.817903 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:56.817874    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-8cc45868-t8wts\" (UID: \"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts"
Jul 09 18:15:56.918561 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:56.918530    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-8cc45868-t8wts\" (UID: \"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts"
Jul 09 18:15:56.918706 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:56.918576    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-8cc45868-t8wts\" (UID: \"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts"
Jul 09 18:15:56.918706 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:56.918597    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-8cc45868-t8wts\" (UID: \"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts"
Jul 09 18:15:56.918706 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:56.918672    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-8cc45868-t8wts\" (UID: \"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts"
Jul 09 18:15:56.918852 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:56.918758    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kggd\" (UniqueName: \"kubernetes.io/projected/3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c-kube-api-access-5kggd\") pod \"isvc-sklearn-batcher-predictor-8cc45868-t8wts\" (UID: \"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts"
Jul 09 18:15:56.918979 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:56.918962    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-8cc45868-t8wts\" (UID: \"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts"
Jul 09 18:15:56.919043 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:15:56.918964    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-serving-cert: secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Jul 09 18:15:56.919098 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:15:56.919045    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c-proxy-tls podName:3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c nodeName:}" failed. No retries permitted until 2026-07-09 18:15:57.419029759 +0000 UTC m=+703.088110692 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c-proxy-tls") pod "isvc-sklearn-batcher-predictor-8cc45868-t8wts" (UID: "3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c") : secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Jul 09 18:15:56.919280 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:56.919260    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-8cc45868-t8wts\" (UID: \"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts"
Jul 09 18:15:56.919402 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:56.919382    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-8cc45868-t8wts\" (UID: \"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts"
Jul 09 18:15:56.927512 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:56.927492    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kggd\" (UniqueName: \"kubernetes.io/projected/3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c-kube-api-access-5kggd\") pod \"isvc-sklearn-batcher-predictor-8cc45868-t8wts\" (UID: \"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts"
Jul 09 18:15:57.424404 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:57.424369    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-8cc45868-t8wts\" (UID: \"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts"
Jul 09 18:15:57.426745 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:57.426723    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-8cc45868-t8wts\" (UID: \"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts"
Jul 09 18:15:57.577866 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:57.577831    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts"
Jul 09 18:15:57.711591 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:57.711564    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts"]
Jul 09 18:15:57.713809 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:15:57.713783    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ecb7dc6_8b7b_46aa_9589_0a187ad8fa6c.slice/crio-d5df0896eefbd838e9222a0ad94b32f0c295b7b868e891beed4035a351a9099d WatchSource:0}: Error finding container d5df0896eefbd838e9222a0ad94b32f0c295b7b868e891beed4035a351a9099d: Status 404 returned error can't find the container with id d5df0896eefbd838e9222a0ad94b32f0c295b7b868e891beed4035a351a9099d
Jul 09 18:15:57.998661 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:57.998585    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" event={"ID":"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c","Type":"ContainerStarted","Data":"778955f31b3cf599599870ada1d270972edae3c37ac2673dec9f065a578dec2c"}
Jul 09 18:15:57.998794 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:15:57.998705    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" event={"ID":"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c","Type":"ContainerStarted","Data":"d5df0896eefbd838e9222a0ad94b32f0c295b7b868e891beed4035a351a9099d"}
Jul 09 18:16:02.017048 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:16:02.017018    2563 generic.go:356] "Generic (PLEG): container finished" podID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerID="778955f31b3cf599599870ada1d270972edae3c37ac2673dec9f065a578dec2c" exitCode=0
Jul 09 18:16:02.017472 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:16:02.017068    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" event={"ID":"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c","Type":"ContainerDied","Data":"778955f31b3cf599599870ada1d270972edae3c37ac2673dec9f065a578dec2c"}
Jul 09 18:16:16.111060 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:16:16.111016    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" event={"ID":"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c","Type":"ContainerStarted","Data":"167e20877836ae67d6871767928bc416f5cd1eb90c2f2014840c90979ed61604"}
Jul 09 18:16:18.122422 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:16:18.122383    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" event={"ID":"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c","Type":"ContainerStarted","Data":"1eb24a15b5f815f51db332c6ce066a0b7d75e732d7f034479383a5b45daabfcc"}
Jul 09 18:16:21.140013 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:16:21.139973    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" event={"ID":"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c","Type":"ContainerStarted","Data":"d4334763a2c7dfee749aa57d5e2eca7796659205ab19a7ca6106a15e86660b0d"}
Jul 09 18:16:21.140499 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:16:21.140039    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts"
Jul 09 18:16:21.140499 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:16:21.140091    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts"
Jul 09 18:16:21.141566 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:16:21.141534    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 09 18:16:21.163204 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:16:21.163158    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" podStartSLOduration=3.0031578 podStartE2EDuration="25.163141799s" totalImagesPullingTime="18.332761364s" totalInitContainerRuntime="3.827222635s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:15:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:16:21.161914624 +0000 UTC m=+726.830995611" watchObservedRunningTime="2026-07-09 18:16:21.163141799 +0000 UTC m=+726.832222753"
Jul 09 18:16:22.144715 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:16:22.144677    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 09 18:16:22.145180 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:16:22.144733    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts"
Jul 09 18:16:22.145816 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:16:22.145784    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:16:22.148542 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:16:22.148523    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts"
Jul 09 18:16:23.148717 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:16:23.148675    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 09 18:16:23.149148 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:16:23.148981    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:16:24.153447 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:16:24.153402    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 09 18:16:24.153855 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:16:24.153742    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:16:34.154151 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:16:34.154105    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 09 18:16:34.154659 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:16:34.154638    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:16:44.154194 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:16:44.154143    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 09 18:16:44.154667 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:16:44.154554    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:16:54.153589 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:16:54.153535    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 09 18:16:54.154022 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:16:54.153998    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:17:04.154256 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:04.154186    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 09 18:17:04.154724 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:04.154700    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:17:14.153961 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:14.153919    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 09 18:17:14.154503 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:14.154473    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:17:24.154061 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:24.154012    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 09 18:17:24.154567 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:24.154538    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:17:34.154421 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:34.154391    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts"
Jul 09 18:17:34.155048 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:34.155025    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts"
Jul 09 18:17:41.833248 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:41.833157    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts"]
Jul 09 18:17:41.833688 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:41.833495    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="kserve-container" containerID="cri-o://167e20877836ae67d6871767928bc416f5cd1eb90c2f2014840c90979ed61604" gracePeriod=30
Jul 09 18:17:41.833688 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:41.833529    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="agent" containerID="cri-o://d4334763a2c7dfee749aa57d5e2eca7796659205ab19a7ca6106a15e86660b0d" gracePeriod=30
Jul 09 18:17:41.833688 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:41.833558    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="kube-rbac-proxy" containerID="cri-o://1eb24a15b5f815f51db332c6ce066a0b7d75e732d7f034479383a5b45daabfcc" gracePeriod=30
Jul 09 18:17:41.974846 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:41.974799    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg"]
Jul 09 18:17:41.978109 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:41.978091    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg"
Jul 09 18:17:41.982131 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:41.982107    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 09 18:17:41.982392 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:41.982378    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 09 18:17:41.994312 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:41.994291    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg"]
Jul 09 18:17:42.145135 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:42.145041    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 09 18:17:42.151745 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:42.151719    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eff876f6-cb34-46ef-9e76-967790f5bf86-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg\" (UID: \"eff876f6-cb34-46ef-9e76-967790f5bf86\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg"
Jul 09 18:17:42.151878 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:42.151759    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eff876f6-cb34-46ef-9e76-967790f5bf86-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg\" (UID: \"eff876f6-cb34-46ef-9e76-967790f5bf86\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg"
Jul 09 18:17:42.151916 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:42.151897    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eff876f6-cb34-46ef-9e76-967790f5bf86-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg\" (UID: \"eff876f6-cb34-46ef-9e76-967790f5bf86\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg"
Jul 09 18:17:42.151956 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:42.151932    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eff876f6-cb34-46ef-9e76-967790f5bf86-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg\" (UID: \"eff876f6-cb34-46ef-9e76-967790f5bf86\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg"
Jul 09 18:17:42.152006 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:42.151954    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt6r8\" (UniqueName: \"kubernetes.io/projected/eff876f6-cb34-46ef-9e76-967790f5bf86-kube-api-access-nt6r8\") pod \"isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg\" (UID: \"eff876f6-cb34-46ef-9e76-967790f5bf86\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg"
Jul 09 18:17:42.252513 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:42.252478    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eff876f6-cb34-46ef-9e76-967790f5bf86-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg\" (UID: \"eff876f6-cb34-46ef-9e76-967790f5bf86\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg"
Jul 09 18:17:42.252513 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:42.252515    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eff876f6-cb34-46ef-9e76-967790f5bf86-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg\" (UID: \"eff876f6-cb34-46ef-9e76-967790f5bf86\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg"
Jul 09 18:17:42.252758 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:42.252536    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt6r8\" (UniqueName: \"kubernetes.io/projected/eff876f6-cb34-46ef-9e76-967790f5bf86-kube-api-access-nt6r8\") pod \"isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg\" (UID: \"eff876f6-cb34-46ef-9e76-967790f5bf86\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg"
Jul 09 18:17:42.252758 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:42.252655    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eff876f6-cb34-46ef-9e76-967790f5bf86-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg\" (UID: \"eff876f6-cb34-46ef-9e76-967790f5bf86\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg"
Jul 09 18:17:42.252758 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:42.252705    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eff876f6-cb34-46ef-9e76-967790f5bf86-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg\" (UID: \"eff876f6-cb34-46ef-9e76-967790f5bf86\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg"
Jul 09 18:17:42.253112 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:42.253087    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eff876f6-cb34-46ef-9e76-967790f5bf86-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg\" (UID: \"eff876f6-cb34-46ef-9e76-967790f5bf86\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg"
Jul 09 18:17:42.253306 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:42.253285    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eff876f6-cb34-46ef-9e76-967790f5bf86-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg\" (UID: \"eff876f6-cb34-46ef-9e76-967790f5bf86\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg"
Jul 09 18:17:42.253382 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:42.253323    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eff876f6-cb34-46ef-9e76-967790f5bf86-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg\" (UID: \"eff876f6-cb34-46ef-9e76-967790f5bf86\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg"
Jul 09 18:17:42.255081 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:42.255062    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eff876f6-cb34-46ef-9e76-967790f5bf86-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg\" (UID: \"eff876f6-cb34-46ef-9e76-967790f5bf86\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg"
Jul 09 18:17:42.261457 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:42.261435    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt6r8\" (UniqueName: \"kubernetes.io/projected/eff876f6-cb34-46ef-9e76-967790f5bf86-kube-api-access-nt6r8\") pod \"isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg\" (UID: \"eff876f6-cb34-46ef-9e76-967790f5bf86\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg"
Jul 09 18:17:42.288516 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:42.288488    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg"
Jul 09 18:17:42.427390 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:42.427360    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg"]
Jul 09 18:17:42.430009 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:17:42.429986    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff876f6_cb34_46ef_9e76_967790f5bf86.slice/crio-39c67c57d49230c01f09c69cecefd1a42ab21f03381f19f6c97bbd36a22b8350 WatchSource:0}: Error finding container 39c67c57d49230c01f09c69cecefd1a42ab21f03381f19f6c97bbd36a22b8350: Status 404 returned error can't find the container with id 39c67c57d49230c01f09c69cecefd1a42ab21f03381f19f6c97bbd36a22b8350
Jul 09 18:17:42.496607 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:42.496576    2563 generic.go:356] "Generic (PLEG): container finished" podID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerID="1eb24a15b5f815f51db332c6ce066a0b7d75e732d7f034479383a5b45daabfcc" exitCode=2
Jul 09 18:17:42.496736 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:42.496651    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" event={"ID":"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c","Type":"ContainerDied","Data":"1eb24a15b5f815f51db332c6ce066a0b7d75e732d7f034479383a5b45daabfcc"}
Jul 09 18:17:42.498302 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:42.498280    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" event={"ID":"eff876f6-cb34-46ef-9e76-967790f5bf86","Type":"ContainerStarted","Data":"b7ed3ff56c934ce955e844920145a7981fed741ceacc14c10c1c561b86386925"}
Jul 09 18:17:42.498429 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:42.498341    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" event={"ID":"eff876f6-cb34-46ef-9e76-967790f5bf86","Type":"ContainerStarted","Data":"39c67c57d49230c01f09c69cecefd1a42ab21f03381f19f6c97bbd36a22b8350"}
Jul 09 18:17:44.153598 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:44.153558    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 09 18:17:44.156256 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:44.156227    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:17:46.518095 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:46.518066    2563 generic.go:356] "Generic (PLEG): container finished" podID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerID="b7ed3ff56c934ce955e844920145a7981fed741ceacc14c10c1c561b86386925" exitCode=0
Jul 09 18:17:46.518464 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:46.518138    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" event={"ID":"eff876f6-cb34-46ef-9e76-967790f5bf86","Type":"ContainerDied","Data":"b7ed3ff56c934ce955e844920145a7981fed741ceacc14c10c1c561b86386925"}
Jul 09 18:17:47.145896 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:47.145856    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 09 18:17:47.525284 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:47.525246    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" event={"ID":"eff876f6-cb34-46ef-9e76-967790f5bf86","Type":"ContainerStarted","Data":"8a43ab6cffc1e45f116d544f5834fd581f46d92021882afbef366598ea11de29"}
Jul 09 18:17:47.525732 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:47.525333    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" event={"ID":"eff876f6-cb34-46ef-9e76-967790f5bf86","Type":"ContainerStarted","Data":"3ee13eca3092af8c99a074034e423b84dfc244a436abbda55a396c1da0b448a4"}
Jul 09 18:17:47.525732 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:47.525373    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" event={"ID":"eff876f6-cb34-46ef-9e76-967790f5bf86","Type":"ContainerStarted","Data":"82475c63fa09877d38b69b5f6c2e14c51b90527d5bb5fbfb8e11858217cb89e3"}
Jul 09 18:17:47.525732 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:47.525537    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg"
Jul 09 18:17:47.525732 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:47.525600    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg"
Jul 09 18:17:47.527296 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:47.527263    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
Jul 09 18:17:47.527829 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:47.527809    2563 generic.go:356] "Generic (PLEG): container finished" podID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerID="167e20877836ae67d6871767928bc416f5cd1eb90c2f2014840c90979ed61604" exitCode=0
Jul 09 18:17:47.527892 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:47.527878    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" event={"ID":"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c","Type":"ContainerDied","Data":"167e20877836ae67d6871767928bc416f5cd1eb90c2f2014840c90979ed61604"}
Jul 09 18:17:47.548849 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:47.548811    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" podStartSLOduration=2.994610637 podStartE2EDuration="6.548798901s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.554188264s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:17:47.546807632 +0000 UTC m=+813.215888599" watchObservedRunningTime="2026-07-09 18:17:47.548798901 +0000 UTC m=+813.217879916"
Jul 09 18:17:48.534712 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:48.534681    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg"
Jul 09 18:17:48.535134 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:48.534793    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
Jul 09 18:17:48.535957 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:48.535931    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:17:49.539932 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:49.539892    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
Jul 09 18:17:49.540428 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:49.540377    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:17:52.145370 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:52.145330    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 09 18:17:52.145755 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:52.145455    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts"
Jul 09 18:17:54.153402 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:54.153358    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 09 18:17:54.156113 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:54.156082    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:17:54.544584 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:54.544555    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg"
Jul 09 18:17:54.545320 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:54.545290    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
Jul 09 18:17:54.545587 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:54.545561    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:17:57.145266 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:17:57.145203    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 09 18:18:02.145182 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:02.145136    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 09 18:18:04.154391 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:04.154353    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 09 18:18:04.154769 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:04.154485    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts"
Jul 09 18:18:04.155962 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:04.155930    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:18:04.156134 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:04.156035    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts"
Jul 09 18:18:04.545849 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:04.545811    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
Jul 09 18:18:04.546283 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:04.546256    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:18:07.145191 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:07.145141    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 09 18:18:11.990617 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:11.990583    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts"
Jul 09 18:18:12.114309 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:12.114232    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kggd\" (UniqueName: \"kubernetes.io/projected/3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c-kube-api-access-5kggd\") pod \"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c\" (UID: \"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c\") "
Jul 09 18:18:12.114309 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:12.114276    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c-proxy-tls\") pod \"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c\" (UID: \"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c\") "
Jul 09 18:18:12.114309 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:12.114309    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c-kserve-provision-location\") pod \"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c\" (UID: \"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c\") "
Jul 09 18:18:12.114565 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:12.114354    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c-cabundle-cert\") pod \"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c\" (UID: \"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c\") "
Jul 09 18:18:12.114565 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:12.114381    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c-kube-rbac-proxy-sar-config\") pod \"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c\" (UID: \"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c\") "
Jul 09 18:18:12.114766 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:12.114708    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" (UID: "3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:18:12.114852 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:12.114796    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" (UID: "3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:18:12.114912 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:12.114895    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" (UID: "3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:18:12.116577 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:12.116551    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" (UID: "3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:18:12.116577 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:12.116563    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c-kube-api-access-5kggd" (OuterVolumeSpecName: "kube-api-access-5kggd") pod "3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" (UID: "3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c"). InnerVolumeSpecName "kube-api-access-5kggd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:18:12.215301 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:12.215267    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kggd\" (UniqueName: \"kubernetes.io/projected/3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c-kube-api-access-5kggd\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:18:12.215301 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:12.215299    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c-proxy-tls\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:18:12.215301 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:12.215312    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c-kserve-provision-location\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:18:12.215535 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:12.215322    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c-cabundle-cert\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:18:12.215535 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:12.215334    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:18:12.655288 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:12.655246    2563 generic.go:356] "Generic (PLEG): container finished" podID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerID="d4334763a2c7dfee749aa57d5e2eca7796659205ab19a7ca6106a15e86660b0d" exitCode=0
Jul 09 18:18:12.655455 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:12.655301    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" event={"ID":"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c","Type":"ContainerDied","Data":"d4334763a2c7dfee749aa57d5e2eca7796659205ab19a7ca6106a15e86660b0d"}
Jul 09 18:18:12.655455 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:12.655324    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts"
Jul 09 18:18:12.655455 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:12.655373    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts" event={"ID":"3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c","Type":"ContainerDied","Data":"d5df0896eefbd838e9222a0ad94b32f0c295b7b868e891beed4035a351a9099d"}
Jul 09 18:18:12.655455 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:12.655404    2563 scope.go:117] "RemoveContainer" containerID="d4334763a2c7dfee749aa57d5e2eca7796659205ab19a7ca6106a15e86660b0d"
Jul 09 18:18:12.664290 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:12.664268    2563 scope.go:117] "RemoveContainer" containerID="1eb24a15b5f815f51db332c6ce066a0b7d75e732d7f034479383a5b45daabfcc"
Jul 09 18:18:12.672272 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:12.672254    2563 scope.go:117] "RemoveContainer" containerID="167e20877836ae67d6871767928bc416f5cd1eb90c2f2014840c90979ed61604"
Jul 09 18:18:12.681457 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:12.681425    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts"]
Jul 09 18:18:12.686338 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:12.686314    2563 scope.go:117] "RemoveContainer" containerID="778955f31b3cf599599870ada1d270972edae3c37ac2673dec9f065a578dec2c"
Jul 09 18:18:12.686338 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:12.686327    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8cc45868-t8wts"]
Jul 09 18:18:12.694525 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:12.694495    2563 scope.go:117] "RemoveContainer" containerID="d4334763a2c7dfee749aa57d5e2eca7796659205ab19a7ca6106a15e86660b0d"
Jul 09 18:18:12.694871 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:18:12.694848    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4334763a2c7dfee749aa57d5e2eca7796659205ab19a7ca6106a15e86660b0d\": container with ID starting with d4334763a2c7dfee749aa57d5e2eca7796659205ab19a7ca6106a15e86660b0d not found: ID does not exist" containerID="d4334763a2c7dfee749aa57d5e2eca7796659205ab19a7ca6106a15e86660b0d"
Jul 09 18:18:12.694943 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:12.694881    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4334763a2c7dfee749aa57d5e2eca7796659205ab19a7ca6106a15e86660b0d"} err="failed to get container status \"d4334763a2c7dfee749aa57d5e2eca7796659205ab19a7ca6106a15e86660b0d\": rpc error: code = NotFound desc = could not find container \"d4334763a2c7dfee749aa57d5e2eca7796659205ab19a7ca6106a15e86660b0d\": container with ID starting with d4334763a2c7dfee749aa57d5e2eca7796659205ab19a7ca6106a15e86660b0d not found: ID does not exist"
Jul 09 18:18:12.694943 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:12.694902    2563 scope.go:117] "RemoveContainer" containerID="1eb24a15b5f815f51db332c6ce066a0b7d75e732d7f034479383a5b45daabfcc"
Jul 09 18:18:12.695165 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:18:12.695147    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eb24a15b5f815f51db332c6ce066a0b7d75e732d7f034479383a5b45daabfcc\": container with ID starting with 1eb24a15b5f815f51db332c6ce066a0b7d75e732d7f034479383a5b45daabfcc not found: ID does not exist" containerID="1eb24a15b5f815f51db332c6ce066a0b7d75e732d7f034479383a5b45daabfcc"
Jul 09 18:18:12.695259 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:12.695171    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eb24a15b5f815f51db332c6ce066a0b7d75e732d7f034479383a5b45daabfcc"} err="failed to get container status \"1eb24a15b5f815f51db332c6ce066a0b7d75e732d7f034479383a5b45daabfcc\": rpc error: code = NotFound desc = could not find container \"1eb24a15b5f815f51db332c6ce066a0b7d75e732d7f034479383a5b45daabfcc\": container with ID starting with 1eb24a15b5f815f51db332c6ce066a0b7d75e732d7f034479383a5b45daabfcc not found: ID does not exist"
Jul 09 18:18:12.695259 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:12.695186    2563 scope.go:117] "RemoveContainer" containerID="167e20877836ae67d6871767928bc416f5cd1eb90c2f2014840c90979ed61604"
Jul 09 18:18:12.695430 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:18:12.695416    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"167e20877836ae67d6871767928bc416f5cd1eb90c2f2014840c90979ed61604\": container with ID starting with 167e20877836ae67d6871767928bc416f5cd1eb90c2f2014840c90979ed61604 not found: ID does not exist" containerID="167e20877836ae67d6871767928bc416f5cd1eb90c2f2014840c90979ed61604"
Jul 09 18:18:12.695469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:12.695435    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"167e20877836ae67d6871767928bc416f5cd1eb90c2f2014840c90979ed61604"} err="failed to get container status \"167e20877836ae67d6871767928bc416f5cd1eb90c2f2014840c90979ed61604\": rpc error: code = NotFound desc = could not find container \"167e20877836ae67d6871767928bc416f5cd1eb90c2f2014840c90979ed61604\": container with ID starting with 167e20877836ae67d6871767928bc416f5cd1eb90c2f2014840c90979ed61604 not found: ID does not exist"
Jul 09 18:18:12.695469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:12.695446    2563 scope.go:117] "RemoveContainer" containerID="778955f31b3cf599599870ada1d270972edae3c37ac2673dec9f065a578dec2c"
Jul 09 18:18:12.695681 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:18:12.695666    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"778955f31b3cf599599870ada1d270972edae3c37ac2673dec9f065a578dec2c\": container with ID starting with 778955f31b3cf599599870ada1d270972edae3c37ac2673dec9f065a578dec2c not found: ID does not exist" containerID="778955f31b3cf599599870ada1d270972edae3c37ac2673dec9f065a578dec2c"
Jul 09 18:18:12.695731 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:12.695685    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"778955f31b3cf599599870ada1d270972edae3c37ac2673dec9f065a578dec2c"} err="failed to get container status \"778955f31b3cf599599870ada1d270972edae3c37ac2673dec9f065a578dec2c\": rpc error: code = NotFound desc = could not find container \"778955f31b3cf599599870ada1d270972edae3c37ac2673dec9f065a578dec2c\": container with ID starting with 778955f31b3cf599599870ada1d270972edae3c37ac2673dec9f065a578dec2c not found: ID does not exist"
Jul 09 18:18:12.960705 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:12.960673    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" path="/var/lib/kubelet/pods/3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c/volumes"
Jul 09 18:18:14.546004 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:14.545964    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
Jul 09 18:18:14.546592 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:14.546572    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:18:24.545698 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:24.545647    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
Jul 09 18:18:24.546149 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:24.546073    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:18:34.546086 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:34.546038    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
Jul 09 18:18:34.546555 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:34.546531    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:18:44.546052 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:44.546005    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
Jul 09 18:18:44.546559 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:44.546463    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:18:54.545846 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:54.545804    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
Jul 09 18:18:54.546350 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:18:54.546327    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:19:04.546045 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:04.546007    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg"
Jul 09 18:19:04.546526 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:04.546274    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg"
Jul 09 18:19:14.852296 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:14.852209    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbftk_5d463fb2-45f9-4554-9525-637e1b84351a/ovn-acl-logging/0.log"
Jul 09 18:19:14.852750 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:14.852652    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbftk_5d463fb2-45f9-4554-9525-637e1b84351a/ovn-acl-logging/0.log"
Jul 09 18:19:17.067532 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:17.067490    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg"]
Jul 09 18:19:17.068163 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:17.067845    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="kserve-container" containerID="cri-o://82475c63fa09877d38b69b5f6c2e14c51b90527d5bb5fbfb8e11858217cb89e3" gracePeriod=30
Jul 09 18:19:17.068163 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:17.067911    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="agent" containerID="cri-o://8a43ab6cffc1e45f116d544f5834fd581f46d92021882afbef366598ea11de29" gracePeriod=30
Jul 09 18:19:17.068163 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:17.068033    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="kube-rbac-proxy" containerID="cri-o://3ee13eca3092af8c99a074034e423b84dfc244a436abbda55a396c1da0b448a4" gracePeriod=30
Jul 09 18:19:17.234270 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:17.234238    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-m8bbr"]
Jul 09 18:19:17.234695 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:17.234675    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="storage-initializer"
Jul 09 18:19:17.234695 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:17.234693    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="storage-initializer"
Jul 09 18:19:17.234866 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:17.234705    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="kserve-container"
Jul 09 18:19:17.234866 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:17.234711    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="kserve-container"
Jul 09 18:19:17.234866 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:17.234719    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="agent"
Jul 09 18:19:17.234866 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:17.234725    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="agent"
Jul 09 18:19:17.234866 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:17.234732    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="kube-rbac-proxy"
Jul 09 18:19:17.234866 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:17.234738    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="kube-rbac-proxy"
Jul 09 18:19:17.234866 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:17.234818    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="kube-rbac-proxy"
Jul 09 18:19:17.234866 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:17.234831    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="agent"
Jul 09 18:19:17.234866 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:17.234840    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ecb7dc6-8b7b-46aa-9589-0a187ad8fa6c" containerName="kserve-container"
Jul 09 18:19:17.237011 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:17.236993    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-m8bbr"
Jul 09 18:19:17.239380 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:17.239357    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jul 09 18:19:17.239489 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:17.239446    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jul 09 18:19:17.251246 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:17.251202    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-m8bbr"]
Jul 09 18:19:17.403284 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:17.403174    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49c252a0-9096-4936-bf5a-b07e515cd43a-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-m8bbr\" (UID: \"49c252a0-9096-4936-bf5a-b07e515cd43a\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-m8bbr"
Jul 09 18:19:17.403284 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:17.403270    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n494b\" (UniqueName: \"kubernetes.io/projected/49c252a0-9096-4936-bf5a-b07e515cd43a-kube-api-access-n494b\") pod \"message-dumper-predictor-6575bcfdfd-m8bbr\" (UID: \"49c252a0-9096-4936-bf5a-b07e515cd43a\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-m8bbr"
Jul 09 18:19:17.403495 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:17.403321    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49c252a0-9096-4936-bf5a-b07e515cd43a-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-m8bbr\" (UID: \"49c252a0-9096-4936-bf5a-b07e515cd43a\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-m8bbr"
Jul 09 18:19:17.504560 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:17.504506    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49c252a0-9096-4936-bf5a-b07e515cd43a-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-m8bbr\" (UID: \"49c252a0-9096-4936-bf5a-b07e515cd43a\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-m8bbr"
Jul 09 18:19:17.504758 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:17.504574    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49c252a0-9096-4936-bf5a-b07e515cd43a-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-m8bbr\" (UID: \"49c252a0-9096-4936-bf5a-b07e515cd43a\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-m8bbr"
Jul 09 18:19:17.504758 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:17.504634    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n494b\" (UniqueName: \"kubernetes.io/projected/49c252a0-9096-4936-bf5a-b07e515cd43a-kube-api-access-n494b\") pod \"message-dumper-predictor-6575bcfdfd-m8bbr\" (UID: \"49c252a0-9096-4936-bf5a-b07e515cd43a\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-m8bbr"
Jul 09 18:19:17.505176 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:17.505154    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49c252a0-9096-4936-bf5a-b07e515cd43a-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-m8bbr\" (UID: \"49c252a0-9096-4936-bf5a-b07e515cd43a\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-m8bbr"
Jul 09 18:19:17.507081 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:17.507056    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49c252a0-9096-4936-bf5a-b07e515cd43a-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-m8bbr\" (UID: \"49c252a0-9096-4936-bf5a-b07e515cd43a\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-m8bbr"
Jul 09 18:19:17.512691 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:17.512665    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n494b\" (UniqueName: \"kubernetes.io/projected/49c252a0-9096-4936-bf5a-b07e515cd43a-kube-api-access-n494b\") pod \"message-dumper-predictor-6575bcfdfd-m8bbr\" (UID: \"49c252a0-9096-4936-bf5a-b07e515cd43a\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-m8bbr"
Jul 09 18:19:17.547289 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:17.547246    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-m8bbr"
Jul 09 18:19:17.682100 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:17.681974    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-m8bbr"]
Jul 09 18:19:17.684707 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:19:17.684670    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c252a0_9096_4936_bf5a_b07e515cd43a.slice/crio-c6fcc6cd3b42d506d73a585b2e22c103ff73e0103fe21db0214e6690c0db27f8 WatchSource:0}: Error finding container c6fcc6cd3b42d506d73a585b2e22c103ff73e0103fe21db0214e6690c0db27f8: Status 404 returned error can't find the container with id c6fcc6cd3b42d506d73a585b2e22c103ff73e0103fe21db0214e6690c0db27f8
Jul 09 18:19:17.686760 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:17.686744    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:19:17.950895 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:17.950803    2563 generic.go:356] "Generic (PLEG): container finished" podID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerID="3ee13eca3092af8c99a074034e423b84dfc244a436abbda55a396c1da0b448a4" exitCode=2
Jul 09 18:19:17.950895 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:17.950879    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" event={"ID":"eff876f6-cb34-46ef-9e76-967790f5bf86","Type":"ContainerDied","Data":"3ee13eca3092af8c99a074034e423b84dfc244a436abbda55a396c1da0b448a4"}
Jul 09 18:19:17.957775 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:17.957738    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-m8bbr" event={"ID":"49c252a0-9096-4936-bf5a-b07e515cd43a","Type":"ContainerStarted","Data":"c6fcc6cd3b42d506d73a585b2e22c103ff73e0103fe21db0214e6690c0db27f8"}
Jul 09 18:19:19.540613 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:19.540567    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 09 18:19:19.970058 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:19.970023    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-m8bbr" event={"ID":"49c252a0-9096-4936-bf5a-b07e515cd43a","Type":"ContainerStarted","Data":"39d3848da14876604bb3b024327126ceae1ac7e8b21c28821a816d5593f4ede3"}
Jul 09 18:19:19.970280 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:19.970080    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-m8bbr" event={"ID":"49c252a0-9096-4936-bf5a-b07e515cd43a","Type":"ContainerStarted","Data":"59ae15fa429cc93564ce75aacb4051fa8ad63c217d0b447c199cfcd088275d4f"}
Jul 09 18:19:19.970280 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:19.970104    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-m8bbr"
Jul 09 18:19:19.970280 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:19.970152    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-m8bbr"
Jul 09 18:19:19.972005 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:19.971990    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-m8bbr"
Jul 09 18:19:19.988438 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:19.988392    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-m8bbr" podStartSLOduration=1.681975703 podStartE2EDuration="2.988378209s" totalImagesPullingTime="1.306402506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:19:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:19:19.986680485 +0000 UTC m=+905.655761439" watchObservedRunningTime="2026-07-09 18:19:19.988378209 +0000 UTC m=+905.657459164"
Jul 09 18:19:22.986122 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:22.986090    2563 generic.go:356] "Generic (PLEG): container finished" podID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerID="82475c63fa09877d38b69b5f6c2e14c51b90527d5bb5fbfb8e11858217cb89e3" exitCode=0
Jul 09 18:19:22.986532 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:22.986161    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" event={"ID":"eff876f6-cb34-46ef-9e76-967790f5bf86","Type":"ContainerDied","Data":"82475c63fa09877d38b69b5f6c2e14c51b90527d5bb5fbfb8e11858217cb89e3"}
Jul 09 18:19:24.540975 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:24.540937    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 09 18:19:24.545968 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:24.545935    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
Jul 09 18:19:24.546277 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:24.546248    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:19:26.984126 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:26.984096    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-m8bbr"
Jul 09 18:19:29.540568 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:29.540527    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 09 18:19:29.540963 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:29.540677    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg"
Jul 09 18:19:34.540998 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:34.540955    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 09 18:19:34.545423 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:34.545394    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
Jul 09 18:19:34.545779 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:34.545750    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:19:37.178260 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:37.178210    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf"]
Jul 09 18:19:37.182310 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:37.182292    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf"
Jul 09 18:19:37.184646 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:37.184609    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 09 18:19:37.184783 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:37.184714    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 09 18:19:37.193022 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:37.192998    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf"]
Jul 09 18:19:37.268963 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:37.268926    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0726cbab-f9ce-4bf2-b81c-418065d4df95-proxy-tls\") pod \"isvc-logger-predictor-69476bc7f-vs7sf\" (UID: \"0726cbab-f9ce-4bf2-b81c-418065d4df95\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf"
Jul 09 18:19:37.269137 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:37.268973    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0726cbab-f9ce-4bf2-b81c-418065d4df95-kserve-provision-location\") pod \"isvc-logger-predictor-69476bc7f-vs7sf\" (UID: \"0726cbab-f9ce-4bf2-b81c-418065d4df95\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf"
Jul 09 18:19:37.269137 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:37.269055    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0726cbab-f9ce-4bf2-b81c-418065d4df95-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-69476bc7f-vs7sf\" (UID: \"0726cbab-f9ce-4bf2-b81c-418065d4df95\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf"
Jul 09 18:19:37.269137 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:37.269110    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s42l\" (UniqueName: \"kubernetes.io/projected/0726cbab-f9ce-4bf2-b81c-418065d4df95-kube-api-access-4s42l\") pod \"isvc-logger-predictor-69476bc7f-vs7sf\" (UID: \"0726cbab-f9ce-4bf2-b81c-418065d4df95\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf"
Jul 09 18:19:37.269279 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:37.269164    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0726cbab-f9ce-4bf2-b81c-418065d4df95-cabundle-cert\") pod \"isvc-logger-predictor-69476bc7f-vs7sf\" (UID: \"0726cbab-f9ce-4bf2-b81c-418065d4df95\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf"
Jul 09 18:19:37.370666 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:37.370628    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0726cbab-f9ce-4bf2-b81c-418065d4df95-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-69476bc7f-vs7sf\" (UID: \"0726cbab-f9ce-4bf2-b81c-418065d4df95\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf"
Jul 09 18:19:37.370856 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:37.370781    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s42l\" (UniqueName: \"kubernetes.io/projected/0726cbab-f9ce-4bf2-b81c-418065d4df95-kube-api-access-4s42l\") pod \"isvc-logger-predictor-69476bc7f-vs7sf\" (UID: \"0726cbab-f9ce-4bf2-b81c-418065d4df95\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf"
Jul 09 18:19:37.370856 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:37.370826    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0726cbab-f9ce-4bf2-b81c-418065d4df95-cabundle-cert\") pod \"isvc-logger-predictor-69476bc7f-vs7sf\" (UID: \"0726cbab-f9ce-4bf2-b81c-418065d4df95\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf"
Jul 09 18:19:37.370980 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:37.370961    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0726cbab-f9ce-4bf2-b81c-418065d4df95-proxy-tls\") pod \"isvc-logger-predictor-69476bc7f-vs7sf\" (UID: \"0726cbab-f9ce-4bf2-b81c-418065d4df95\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf"
Jul 09 18:19:37.371042 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:37.371026    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0726cbab-f9ce-4bf2-b81c-418065d4df95-kserve-provision-location\") pod \"isvc-logger-predictor-69476bc7f-vs7sf\" (UID: \"0726cbab-f9ce-4bf2-b81c-418065d4df95\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf"
Jul 09 18:19:37.371473 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:37.371453    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0726cbab-f9ce-4bf2-b81c-418065d4df95-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-69476bc7f-vs7sf\" (UID: \"0726cbab-f9ce-4bf2-b81c-418065d4df95\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf"
Jul 09 18:19:37.371561 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:37.371478    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0726cbab-f9ce-4bf2-b81c-418065d4df95-kserve-provision-location\") pod \"isvc-logger-predictor-69476bc7f-vs7sf\" (UID: \"0726cbab-f9ce-4bf2-b81c-418065d4df95\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf"
Jul 09 18:19:37.371561 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:37.371525    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0726cbab-f9ce-4bf2-b81c-418065d4df95-cabundle-cert\") pod \"isvc-logger-predictor-69476bc7f-vs7sf\" (UID: \"0726cbab-f9ce-4bf2-b81c-418065d4df95\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf"
Jul 09 18:19:37.373607 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:37.373584    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0726cbab-f9ce-4bf2-b81c-418065d4df95-proxy-tls\") pod \"isvc-logger-predictor-69476bc7f-vs7sf\" (UID: \"0726cbab-f9ce-4bf2-b81c-418065d4df95\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf"
Jul 09 18:19:37.383173 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:37.383144    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s42l\" (UniqueName: \"kubernetes.io/projected/0726cbab-f9ce-4bf2-b81c-418065d4df95-kube-api-access-4s42l\") pod \"isvc-logger-predictor-69476bc7f-vs7sf\" (UID: \"0726cbab-f9ce-4bf2-b81c-418065d4df95\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf"
Jul 09 18:19:37.493954 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:37.493813    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf"
Jul 09 18:19:37.632308 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:37.632283    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf"]
Jul 09 18:19:37.635038 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:19:37.635010    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0726cbab_f9ce_4bf2_b81c_418065d4df95.slice/crio-1d1c935ff9658135ca6355840af03c3ec33da8e36630cd1b0b614a49dab514ff WatchSource:0}: Error finding container 1d1c935ff9658135ca6355840af03c3ec33da8e36630cd1b0b614a49dab514ff: Status 404 returned error can't find the container with id 1d1c935ff9658135ca6355840af03c3ec33da8e36630cd1b0b614a49dab514ff
Jul 09 18:19:38.056844 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:38.056806    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" event={"ID":"0726cbab-f9ce-4bf2-b81c-418065d4df95","Type":"ContainerStarted","Data":"dac9ea8516ebbe20f5572e26733066bbe3167a96eeac6810f165bdbb3d1866c7"}
Jul 09 18:19:38.057044 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:38.056872    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" event={"ID":"0726cbab-f9ce-4bf2-b81c-418065d4df95","Type":"ContainerStarted","Data":"1d1c935ff9658135ca6355840af03c3ec33da8e36630cd1b0b614a49dab514ff"}
Jul 09 18:19:39.540619 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:39.540575    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 09 18:19:42.077532 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:42.077495    2563 generic.go:356] "Generic (PLEG): container finished" podID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerID="dac9ea8516ebbe20f5572e26733066bbe3167a96eeac6810f165bdbb3d1866c7" exitCode=0
Jul 09 18:19:42.077924 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:42.077560    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" event={"ID":"0726cbab-f9ce-4bf2-b81c-418065d4df95","Type":"ContainerDied","Data":"dac9ea8516ebbe20f5572e26733066bbe3167a96eeac6810f165bdbb3d1866c7"}
Jul 09 18:19:43.085339 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:43.085300    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" event={"ID":"0726cbab-f9ce-4bf2-b81c-418065d4df95","Type":"ContainerStarted","Data":"1dbe7b6f20c10c11b593e18f6bcf4b013043b95a0412ecb226ede2043887c7e5"}
Jul 09 18:19:43.085770 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:43.085367    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" event={"ID":"0726cbab-f9ce-4bf2-b81c-418065d4df95","Type":"ContainerStarted","Data":"01ac2fce5aa88bb9d7554fdd32ab65e5fc7cb01f4a0810f5a8d967825c2067ab"}
Jul 09 18:19:43.085770 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:43.085391    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" event={"ID":"0726cbab-f9ce-4bf2-b81c-418065d4df95","Type":"ContainerStarted","Data":"c486fd7a3fc37c66d80e8296690ed146ee88784739df2b68b58027fccb2f7714"}
Jul 09 18:19:43.085770 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:43.085582    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf"
Jul 09 18:19:43.107660 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:43.107612    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" podStartSLOduration=2.302066763 podStartE2EDuration="6.107594375s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.805527612s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:19:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:19:43.106207739 +0000 UTC m=+928.775288696" watchObservedRunningTime="2026-07-09 18:19:43.107594375 +0000 UTC m=+928.776675329"
Jul 09 18:19:44.090338 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:44.090298    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf"
Jul 09 18:19:44.090751 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:44.090411    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf"
Jul 09 18:19:44.091841 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:44.091799    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 09 18:19:44.092507 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:44.092482    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:19:44.540850 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:44.540806    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 09 18:19:44.546072 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:44.546039    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
Jul 09 18:19:44.546241 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:44.546209    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg"
Jul 09 18:19:44.546439 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:44.546416    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:19:44.546544 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:44.546529    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg"
Jul 09 18:19:45.094785 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:45.094739    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 09 18:19:45.095281 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:45.095259    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:19:47.109867 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:47.109837    2563 generic.go:356] "Generic (PLEG): container finished" podID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerID="8a43ab6cffc1e45f116d544f5834fd581f46d92021882afbef366598ea11de29" exitCode=0
Jul 09 18:19:47.110193 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:47.109880    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" event={"ID":"eff876f6-cb34-46ef-9e76-967790f5bf86","Type":"ContainerDied","Data":"8a43ab6cffc1e45f116d544f5834fd581f46d92021882afbef366598ea11de29"}
Jul 09 18:19:47.220578 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:47.220551    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg"
Jul 09 18:19:47.360493 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:47.360397    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eff876f6-cb34-46ef-9e76-967790f5bf86-kube-rbac-proxy-sar-config\") pod \"eff876f6-cb34-46ef-9e76-967790f5bf86\" (UID: \"eff876f6-cb34-46ef-9e76-967790f5bf86\") "
Jul 09 18:19:47.360493 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:47.360448    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eff876f6-cb34-46ef-9e76-967790f5bf86-cabundle-cert\") pod \"eff876f6-cb34-46ef-9e76-967790f5bf86\" (UID: \"eff876f6-cb34-46ef-9e76-967790f5bf86\") "
Jul 09 18:19:47.360703 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:47.360506    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nt6r8\" (UniqueName: \"kubernetes.io/projected/eff876f6-cb34-46ef-9e76-967790f5bf86-kube-api-access-nt6r8\") pod \"eff876f6-cb34-46ef-9e76-967790f5bf86\" (UID: \"eff876f6-cb34-46ef-9e76-967790f5bf86\") "
Jul 09 18:19:47.360703 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:47.360552    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eff876f6-cb34-46ef-9e76-967790f5bf86-kserve-provision-location\") pod \"eff876f6-cb34-46ef-9e76-967790f5bf86\" (UID: \"eff876f6-cb34-46ef-9e76-967790f5bf86\") "
Jul 09 18:19:47.360703 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:47.360569    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eff876f6-cb34-46ef-9e76-967790f5bf86-proxy-tls\") pod \"eff876f6-cb34-46ef-9e76-967790f5bf86\" (UID: \"eff876f6-cb34-46ef-9e76-967790f5bf86\") "
Jul 09 18:19:47.360893 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:47.360863    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eff876f6-cb34-46ef-9e76-967790f5bf86-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eff876f6-cb34-46ef-9e76-967790f5bf86" (UID: "eff876f6-cb34-46ef-9e76-967790f5bf86"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:19:47.360893 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:47.360879    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eff876f6-cb34-46ef-9e76-967790f5bf86-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eff876f6-cb34-46ef-9e76-967790f5bf86" (UID: "eff876f6-cb34-46ef-9e76-967790f5bf86"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:19:47.361037 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:47.360863    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff876f6-cb34-46ef-9e76-967790f5bf86-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eff876f6-cb34-46ef-9e76-967790f5bf86" (UID: "eff876f6-cb34-46ef-9e76-967790f5bf86"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:19:47.362600 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:47.362576    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff876f6-cb34-46ef-9e76-967790f5bf86-kube-api-access-nt6r8" (OuterVolumeSpecName: "kube-api-access-nt6r8") pod "eff876f6-cb34-46ef-9e76-967790f5bf86" (UID: "eff876f6-cb34-46ef-9e76-967790f5bf86"). InnerVolumeSpecName "kube-api-access-nt6r8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:19:47.362690 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:47.362639    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eff876f6-cb34-46ef-9e76-967790f5bf86-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eff876f6-cb34-46ef-9e76-967790f5bf86" (UID: "eff876f6-cb34-46ef-9e76-967790f5bf86"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:19:47.461676 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:47.461644    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eff876f6-cb34-46ef-9e76-967790f5bf86-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:19:47.461676 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:47.461671    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eff876f6-cb34-46ef-9e76-967790f5bf86-cabundle-cert\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:19:47.461676 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:47.461680    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt6r8\" (UniqueName: \"kubernetes.io/projected/eff876f6-cb34-46ef-9e76-967790f5bf86-kube-api-access-nt6r8\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:19:47.461904 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:47.461690    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eff876f6-cb34-46ef-9e76-967790f5bf86-kserve-provision-location\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:19:47.461904 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:47.461700    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eff876f6-cb34-46ef-9e76-967790f5bf86-proxy-tls\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:19:48.116558 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:48.116522    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg" event={"ID":"eff876f6-cb34-46ef-9e76-967790f5bf86","Type":"ContainerDied","Data":"39c67c57d49230c01f09c69cecefd1a42ab21f03381f19f6c97bbd36a22b8350"}
Jul 09 18:19:48.116963 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:48.116582    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg"
Jul 09 18:19:48.116963 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:48.116598    2563 scope.go:117] "RemoveContainer" containerID="8a43ab6cffc1e45f116d544f5834fd581f46d92021882afbef366598ea11de29"
Jul 09 18:19:48.125883 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:48.125865    2563 scope.go:117] "RemoveContainer" containerID="3ee13eca3092af8c99a074034e423b84dfc244a436abbda55a396c1da0b448a4"
Jul 09 18:19:48.133764 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:48.133741    2563 scope.go:117] "RemoveContainer" containerID="82475c63fa09877d38b69b5f6c2e14c51b90527d5bb5fbfb8e11858217cb89e3"
Jul 09 18:19:48.141530 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:48.141502    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg"]
Jul 09 18:19:48.143027 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:48.142995    2563 scope.go:117] "RemoveContainer" containerID="b7ed3ff56c934ce955e844920145a7981fed741ceacc14c10c1c561b86386925"
Jul 09 18:19:48.146307 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:48.146282    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5cf8dbf48c-5j5fg"]
Jul 09 18:19:48.960444 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:48.960408    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" path="/var/lib/kubelet/pods/eff876f6-cb34-46ef-9e76-967790f5bf86/volumes"
Jul 09 18:19:50.099196 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:50.099169    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf"
Jul 09 18:19:50.099846 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:50.099820    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 09 18:19:50.100152 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:19:50.100132    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:20:00.099743 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:20:00.099696    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 09 18:20:00.102206 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:20:00.100150    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:20:10.100656 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:20:10.100613    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 09 18:20:10.101235 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:20:10.101034    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:20:20.100051 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:20:20.100003    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 09 18:20:20.100508 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:20:20.100424    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:20:30.100420 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:20:30.100373    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 09 18:20:30.100876 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:20:30.100850    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:20:40.100581 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:20:40.100490    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 09 18:20:40.100942 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:20:40.100917    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:20:50.099847 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:20:50.099800    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 09 18:20:50.100513 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:20:50.100189    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:21:00.101301 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:00.101273    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf"
Jul 09 18:21:00.101737 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:00.101388    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf"
Jul 09 18:21:12.176539 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.176508    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-m8bbr_49c252a0-9096-4936-bf5a-b07e515cd43a/kserve-container/0.log"
Jul 09 18:21:12.402583 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.402553    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf"]
Jul 09 18:21:12.402866 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.402829    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="agent" containerID="cri-o://1dbe7b6f20c10c11b593e18f6bcf4b013043b95a0412ecb226ede2043887c7e5" gracePeriod=30
Jul 09 18:21:12.403020 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.402866    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="kube-rbac-proxy" containerID="cri-o://01ac2fce5aa88bb9d7554fdd32ab65e5fc7cb01f4a0810f5a8d967825c2067ab" gracePeriod=30
Jul 09 18:21:12.403020 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.402817    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="kserve-container" containerID="cri-o://c486fd7a3fc37c66d80e8296690ed146ee88784739df2b68b58027fccb2f7714" gracePeriod=30
Jul 09 18:21:12.494557 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.494464    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh"]
Jul 09 18:21:12.494891 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.494878    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="kserve-container"
Jul 09 18:21:12.494936 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.494893    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="kserve-container"
Jul 09 18:21:12.494936 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.494906    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="kube-rbac-proxy"
Jul 09 18:21:12.494936 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.494912    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="kube-rbac-proxy"
Jul 09 18:21:12.494936 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.494923    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="storage-initializer"
Jul 09 18:21:12.494936 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.494930    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="storage-initializer"
Jul 09 18:21:12.494936 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.494936    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="agent"
Jul 09 18:21:12.495130 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.494942    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="agent"
Jul 09 18:21:12.495130 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.494989    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="kserve-container"
Jul 09 18:21:12.495130 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.494996    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="kube-rbac-proxy"
Jul 09 18:21:12.495130 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.495009    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="eff876f6-cb34-46ef-9e76-967790f5bf86" containerName="agent"
Jul 09 18:21:12.498726 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.498706    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh"
Jul 09 18:21:12.500964 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.500934    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 09 18:21:12.501368 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.501346    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 09 18:21:12.518377 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.518338    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh"]
Jul 09 18:21:12.523235 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.523189    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-m8bbr"]
Jul 09 18:21:12.523587 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.523555    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-m8bbr" podUID="49c252a0-9096-4936-bf5a-b07e515cd43a" containerName="kserve-container" containerID="cri-o://59ae15fa429cc93564ce75aacb4051fa8ad63c217d0b447c199cfcd088275d4f" gracePeriod=30
Jul 09 18:21:12.523952 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.523922    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-m8bbr" podUID="49c252a0-9096-4936-bf5a-b07e515cd43a" containerName="kube-rbac-proxy" containerID="cri-o://39d3848da14876604bb3b024327126ceae1ac7e8b21c28821a816d5593f4ede3" gracePeriod=30
Jul 09 18:21:12.534637 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.534200    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gqz6\" (UniqueName: \"kubernetes.io/projected/81f64c27-d414-48a2-85a1-c937b484ea64-kube-api-access-6gqz6\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh\" (UID: \"81f64c27-d414-48a2-85a1-c937b484ea64\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh"
Jul 09 18:21:12.534637 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.534338    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81f64c27-d414-48a2-85a1-c937b484ea64-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh\" (UID: \"81f64c27-d414-48a2-85a1-c937b484ea64\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh"
Jul 09 18:21:12.534637 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.534374    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81f64c27-d414-48a2-85a1-c937b484ea64-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh\" (UID: \"81f64c27-d414-48a2-85a1-c937b484ea64\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh"
Jul 09 18:21:12.534637 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.534414    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81f64c27-d414-48a2-85a1-c937b484ea64-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh\" (UID: \"81f64c27-d414-48a2-85a1-c937b484ea64\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh"
Jul 09 18:21:12.534637 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.534455    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81f64c27-d414-48a2-85a1-c937b484ea64-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh\" (UID: \"81f64c27-d414-48a2-85a1-c937b484ea64\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh"
Jul 09 18:21:12.635724 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.635688    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gqz6\" (UniqueName: \"kubernetes.io/projected/81f64c27-d414-48a2-85a1-c937b484ea64-kube-api-access-6gqz6\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh\" (UID: \"81f64c27-d414-48a2-85a1-c937b484ea64\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh"
Jul 09 18:21:12.635898 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.635756    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81f64c27-d414-48a2-85a1-c937b484ea64-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh\" (UID: \"81f64c27-d414-48a2-85a1-c937b484ea64\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh"
Jul 09 18:21:12.635898 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.635776    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81f64c27-d414-48a2-85a1-c937b484ea64-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh\" (UID: \"81f64c27-d414-48a2-85a1-c937b484ea64\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh"
Jul 09 18:21:12.636005 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.635931    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81f64c27-d414-48a2-85a1-c937b484ea64-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh\" (UID: \"81f64c27-d414-48a2-85a1-c937b484ea64\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh"
Jul 09 18:21:12.636005 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.635985    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81f64c27-d414-48a2-85a1-c937b484ea64-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh\" (UID: \"81f64c27-d414-48a2-85a1-c937b484ea64\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh"
Jul 09 18:21:12.636197 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:21:12.636176    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v1-predictor-serving-cert: secret "isvc-autogluon-v1-predictor-serving-cert" not found
Jul 09 18:21:12.636318 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:21:12.636259    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81f64c27-d414-48a2-85a1-c937b484ea64-proxy-tls podName:81f64c27-d414-48a2-85a1-c937b484ea64 nodeName:}" failed. No retries permitted until 2026-07-09 18:21:13.136242266 +0000 UTC m=+1018.805323210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/81f64c27-d414-48a2-85a1-c937b484ea64-proxy-tls") pod "isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh" (UID: "81f64c27-d414-48a2-85a1-c937b484ea64") : secret "isvc-autogluon-v1-predictor-serving-cert" not found
Jul 09 18:21:12.636318 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.636296    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81f64c27-d414-48a2-85a1-c937b484ea64-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh\" (UID: \"81f64c27-d414-48a2-85a1-c937b484ea64\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh"
Jul 09 18:21:12.636566 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.636546    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81f64c27-d414-48a2-85a1-c937b484ea64-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh\" (UID: \"81f64c27-d414-48a2-85a1-c937b484ea64\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh"
Jul 09 18:21:12.636664 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.636640    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81f64c27-d414-48a2-85a1-c937b484ea64-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh\" (UID: \"81f64c27-d414-48a2-85a1-c937b484ea64\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh"
Jul 09 18:21:12.644721 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.644691    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gqz6\" (UniqueName: \"kubernetes.io/projected/81f64c27-d414-48a2-85a1-c937b484ea64-kube-api-access-6gqz6\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh\" (UID: \"81f64c27-d414-48a2-85a1-c937b484ea64\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh"
Jul 09 18:21:12.785486 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.785458    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-m8bbr"
Jul 09 18:21:12.838014 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.837976    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49c252a0-9096-4936-bf5a-b07e515cd43a-proxy-tls\") pod \"49c252a0-9096-4936-bf5a-b07e515cd43a\" (UID: \"49c252a0-9096-4936-bf5a-b07e515cd43a\") "
Jul 09 18:21:12.838185 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.838038    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49c252a0-9096-4936-bf5a-b07e515cd43a-kube-rbac-proxy-sar-config\") pod \"49c252a0-9096-4936-bf5a-b07e515cd43a\" (UID: \"49c252a0-9096-4936-bf5a-b07e515cd43a\") "
Jul 09 18:21:12.838185 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.838114    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n494b\" (UniqueName: \"kubernetes.io/projected/49c252a0-9096-4936-bf5a-b07e515cd43a-kube-api-access-n494b\") pod \"49c252a0-9096-4936-bf5a-b07e515cd43a\" (UID: \"49c252a0-9096-4936-bf5a-b07e515cd43a\") "
Jul 09 18:21:12.838443 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.838415    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49c252a0-9096-4936-bf5a-b07e515cd43a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "49c252a0-9096-4936-bf5a-b07e515cd43a" (UID: "49c252a0-9096-4936-bf5a-b07e515cd43a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:21:12.840131 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.840107    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49c252a0-9096-4936-bf5a-b07e515cd43a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "49c252a0-9096-4936-bf5a-b07e515cd43a" (UID: "49c252a0-9096-4936-bf5a-b07e515cd43a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:21:12.840205 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.840107    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c252a0-9096-4936-bf5a-b07e515cd43a-kube-api-access-n494b" (OuterVolumeSpecName: "kube-api-access-n494b") pod "49c252a0-9096-4936-bf5a-b07e515cd43a" (UID: "49c252a0-9096-4936-bf5a-b07e515cd43a"). InnerVolumeSpecName "kube-api-access-n494b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:21:12.939176 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.939133    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n494b\" (UniqueName: \"kubernetes.io/projected/49c252a0-9096-4936-bf5a-b07e515cd43a-kube-api-access-n494b\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:21:12.939176 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.939168    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49c252a0-9096-4936-bf5a-b07e515cd43a-proxy-tls\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:21:12.939176 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:12.939178    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49c252a0-9096-4936-bf5a-b07e515cd43a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:21:13.140803 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:13.140715    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81f64c27-d414-48a2-85a1-c937b484ea64-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh\" (UID: \"81f64c27-d414-48a2-85a1-c937b484ea64\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh"
Jul 09 18:21:13.143058 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:13.143036    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81f64c27-d414-48a2-85a1-c937b484ea64-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh\" (UID: \"81f64c27-d414-48a2-85a1-c937b484ea64\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh"
Jul 09 18:21:13.412590 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:13.412504    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh"
Jul 09 18:21:13.538688 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:13.538653    2563 generic.go:356] "Generic (PLEG): container finished" podID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerID="01ac2fce5aa88bb9d7554fdd32ab65e5fc7cb01f4a0810f5a8d967825c2067ab" exitCode=2
Jul 09 18:21:13.538858 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:13.538735    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" event={"ID":"0726cbab-f9ce-4bf2-b81c-418065d4df95","Type":"ContainerDied","Data":"01ac2fce5aa88bb9d7554fdd32ab65e5fc7cb01f4a0810f5a8d967825c2067ab"}
Jul 09 18:21:13.540293 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:13.540260    2563 generic.go:356] "Generic (PLEG): container finished" podID="49c252a0-9096-4936-bf5a-b07e515cd43a" containerID="39d3848da14876604bb3b024327126ceae1ac7e8b21c28821a816d5593f4ede3" exitCode=2
Jul 09 18:21:13.540293 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:13.540288    2563 generic.go:356] "Generic (PLEG): container finished" podID="49c252a0-9096-4936-bf5a-b07e515cd43a" containerID="59ae15fa429cc93564ce75aacb4051fa8ad63c217d0b447c199cfcd088275d4f" exitCode=2
Jul 09 18:21:13.540500 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:13.540320    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-m8bbr"
Jul 09 18:21:13.540500 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:13.540346    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-m8bbr" event={"ID":"49c252a0-9096-4936-bf5a-b07e515cd43a","Type":"ContainerDied","Data":"39d3848da14876604bb3b024327126ceae1ac7e8b21c28821a816d5593f4ede3"}
Jul 09 18:21:13.540500 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:13.540416    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-m8bbr" event={"ID":"49c252a0-9096-4936-bf5a-b07e515cd43a","Type":"ContainerDied","Data":"59ae15fa429cc93564ce75aacb4051fa8ad63c217d0b447c199cfcd088275d4f"}
Jul 09 18:21:13.540500 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:13.540449    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-m8bbr" event={"ID":"49c252a0-9096-4936-bf5a-b07e515cd43a","Type":"ContainerDied","Data":"c6fcc6cd3b42d506d73a585b2e22c103ff73e0103fe21db0214e6690c0db27f8"}
Jul 09 18:21:13.540500 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:13.540486    2563 scope.go:117] "RemoveContainer" containerID="39d3848da14876604bb3b024327126ceae1ac7e8b21c28821a816d5593f4ede3"
Jul 09 18:21:13.549101 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:13.549082    2563 scope.go:117] "RemoveContainer" containerID="59ae15fa429cc93564ce75aacb4051fa8ad63c217d0b447c199cfcd088275d4f"
Jul 09 18:21:13.552892 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:13.552789    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh"]
Jul 09 18:21:13.561026 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:13.560996    2563 scope.go:117] "RemoveContainer" containerID="39d3848da14876604bb3b024327126ceae1ac7e8b21c28821a816d5593f4ede3"
Jul 09 18:21:13.561357 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:21:13.561291    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39d3848da14876604bb3b024327126ceae1ac7e8b21c28821a816d5593f4ede3\": container with ID starting with 39d3848da14876604bb3b024327126ceae1ac7e8b21c28821a816d5593f4ede3 not found: ID does not exist" containerID="39d3848da14876604bb3b024327126ceae1ac7e8b21c28821a816d5593f4ede3"
Jul 09 18:21:13.561357 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:13.561330    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39d3848da14876604bb3b024327126ceae1ac7e8b21c28821a816d5593f4ede3"} err="failed to get container status \"39d3848da14876604bb3b024327126ceae1ac7e8b21c28821a816d5593f4ede3\": rpc error: code = NotFound desc = could not find container \"39d3848da14876604bb3b024327126ceae1ac7e8b21c28821a816d5593f4ede3\": container with ID starting with 39d3848da14876604bb3b024327126ceae1ac7e8b21c28821a816d5593f4ede3 not found: ID does not exist"
Jul 09 18:21:13.561357 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:13.561356    2563 scope.go:117] "RemoveContainer" containerID="59ae15fa429cc93564ce75aacb4051fa8ad63c217d0b447c199cfcd088275d4f"
Jul 09 18:21:13.561680 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:21:13.561661    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59ae15fa429cc93564ce75aacb4051fa8ad63c217d0b447c199cfcd088275d4f\": container with ID starting with 59ae15fa429cc93564ce75aacb4051fa8ad63c217d0b447c199cfcd088275d4f not found: ID does not exist" containerID="59ae15fa429cc93564ce75aacb4051fa8ad63c217d0b447c199cfcd088275d4f"
Jul 09 18:21:13.561753 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:13.561690    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59ae15fa429cc93564ce75aacb4051fa8ad63c217d0b447c199cfcd088275d4f"} err="failed to get container status \"59ae15fa429cc93564ce75aacb4051fa8ad63c217d0b447c199cfcd088275d4f\": rpc error: code = NotFound desc = could not find container \"59ae15fa429cc93564ce75aacb4051fa8ad63c217d0b447c199cfcd088275d4f\": container with ID starting with 59ae15fa429cc93564ce75aacb4051fa8ad63c217d0b447c199cfcd088275d4f not found: ID does not exist"
Jul 09 18:21:13.561753 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:13.561713    2563 scope.go:117] "RemoveContainer" containerID="39d3848da14876604bb3b024327126ceae1ac7e8b21c28821a816d5593f4ede3"
Jul 09 18:21:13.561986 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:13.561965    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39d3848da14876604bb3b024327126ceae1ac7e8b21c28821a816d5593f4ede3"} err="failed to get container status \"39d3848da14876604bb3b024327126ceae1ac7e8b21c28821a816d5593f4ede3\": rpc error: code = NotFound desc = could not find container \"39d3848da14876604bb3b024327126ceae1ac7e8b21c28821a816d5593f4ede3\": container with ID starting with 39d3848da14876604bb3b024327126ceae1ac7e8b21c28821a816d5593f4ede3 not found: ID does not exist"
Jul 09 18:21:13.562048 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:13.561989    2563 scope.go:117] "RemoveContainer" containerID="59ae15fa429cc93564ce75aacb4051fa8ad63c217d0b447c199cfcd088275d4f"
Jul 09 18:21:13.562280 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:13.562258    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59ae15fa429cc93564ce75aacb4051fa8ad63c217d0b447c199cfcd088275d4f"} err="failed to get container status \"59ae15fa429cc93564ce75aacb4051fa8ad63c217d0b447c199cfcd088275d4f\": rpc error: code = NotFound desc = could not find container \"59ae15fa429cc93564ce75aacb4051fa8ad63c217d0b447c199cfcd088275d4f\": container with ID starting with 59ae15fa429cc93564ce75aacb4051fa8ad63c217d0b447c199cfcd088275d4f not found: ID does not exist"
Jul 09 18:21:13.562738 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:13.562714    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-m8bbr"]
Jul 09 18:21:13.566060 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:13.566041    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-m8bbr"]
Jul 09 18:21:14.546121 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:14.546079    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh" event={"ID":"81f64c27-d414-48a2-85a1-c937b484ea64","Type":"ContainerStarted","Data":"1cf4be205c9a8b8de67e84608ae46646b220ec649bfa48c927ebe47e6da08031"}
Jul 09 18:21:14.546589 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:14.546140    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh" event={"ID":"81f64c27-d414-48a2-85a1-c937b484ea64","Type":"ContainerStarted","Data":"a99c8b0a9bb8a2ff33204a7e183bf4f0c8d62624d660662cbdd493d0a6a26686"}
Jul 09 18:21:14.960913 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:14.960869    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49c252a0-9096-4936-bf5a-b07e515cd43a" path="/var/lib/kubelet/pods/49c252a0-9096-4936-bf5a-b07e515cd43a/volumes"
Jul 09 18:21:15.095804 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:15.095763    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 09 18:21:17.563524 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:17.563430    2563 generic.go:356] "Generic (PLEG): container finished" podID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerID="c486fd7a3fc37c66d80e8296690ed146ee88784739df2b68b58027fccb2f7714" exitCode=0
Jul 09 18:21:17.563524 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:17.563499    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" event={"ID":"0726cbab-f9ce-4bf2-b81c-418065d4df95","Type":"ContainerDied","Data":"c486fd7a3fc37c66d80e8296690ed146ee88784739df2b68b58027fccb2f7714"}
Jul 09 18:21:20.095854 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:20.095809    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 09 18:21:20.100230 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:20.100191    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 09 18:21:20.100585 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:20.100562    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:21:25.095793 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:25.095747    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 09 18:21:25.096286 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:25.095889    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf"
Jul 09 18:21:30.095313 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:30.095269    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 09 18:21:30.100582 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:30.100546    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 09 18:21:30.100917 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:30.100891    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:21:35.095901 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:35.095856    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 09 18:21:40.095013 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:40.094975    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 09 18:21:40.100428 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:40.100385    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 09 18:21:40.100572 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:40.100558    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf"
Jul 09 18:21:40.100799 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:40.100778    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:21:40.100883 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:40.100869    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf"
Jul 09 18:21:42.557292 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:42.557264    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf"
Jul 09 18:21:42.683385 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:42.683352    2563 generic.go:356] "Generic (PLEG): container finished" podID="81f64c27-d414-48a2-85a1-c937b484ea64" containerID="1cf4be205c9a8b8de67e84608ae46646b220ec649bfa48c927ebe47e6da08031" exitCode=0
Jul 09 18:21:42.683563 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:42.683422    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh" event={"ID":"81f64c27-d414-48a2-85a1-c937b484ea64","Type":"ContainerDied","Data":"1cf4be205c9a8b8de67e84608ae46646b220ec649bfa48c927ebe47e6da08031"}
Jul 09 18:21:42.685808 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:42.685786    2563 generic.go:356] "Generic (PLEG): container finished" podID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerID="1dbe7b6f20c10c11b593e18f6bcf4b013043b95a0412ecb226ede2043887c7e5" exitCode=0
Jul 09 18:21:42.685890 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:42.685861    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf"
Jul 09 18:21:42.685890 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:42.685867    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" event={"ID":"0726cbab-f9ce-4bf2-b81c-418065d4df95","Type":"ContainerDied","Data":"1dbe7b6f20c10c11b593e18f6bcf4b013043b95a0412ecb226ede2043887c7e5"}
Jul 09 18:21:42.685968 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:42.685926    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf" event={"ID":"0726cbab-f9ce-4bf2-b81c-418065d4df95","Type":"ContainerDied","Data":"1d1c935ff9658135ca6355840af03c3ec33da8e36630cd1b0b614a49dab514ff"}
Jul 09 18:21:42.685968 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:42.685958    2563 scope.go:117] "RemoveContainer" containerID="1dbe7b6f20c10c11b593e18f6bcf4b013043b95a0412ecb226ede2043887c7e5"
Jul 09 18:21:42.694870 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:42.694850    2563 scope.go:117] "RemoveContainer" containerID="01ac2fce5aa88bb9d7554fdd32ab65e5fc7cb01f4a0810f5a8d967825c2067ab"
Jul 09 18:21:42.703328 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:42.703306    2563 scope.go:117] "RemoveContainer" containerID="c486fd7a3fc37c66d80e8296690ed146ee88784739df2b68b58027fccb2f7714"
Jul 09 18:21:42.711604 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:42.711585    2563 scope.go:117] "RemoveContainer" containerID="dac9ea8516ebbe20f5572e26733066bbe3167a96eeac6810f165bdbb3d1866c7"
Jul 09 18:21:42.713782 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:42.713764    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0726cbab-f9ce-4bf2-b81c-418065d4df95-proxy-tls\") pod \"0726cbab-f9ce-4bf2-b81c-418065d4df95\" (UID: \"0726cbab-f9ce-4bf2-b81c-418065d4df95\") "
Jul 09 18:21:42.713882 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:42.713804    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0726cbab-f9ce-4bf2-b81c-418065d4df95-kserve-provision-location\") pod \"0726cbab-f9ce-4bf2-b81c-418065d4df95\" (UID: \"0726cbab-f9ce-4bf2-b81c-418065d4df95\") "
Jul 09 18:21:42.713882 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:42.713831    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0726cbab-f9ce-4bf2-b81c-418065d4df95-kube-rbac-proxy-sar-config\") pod \"0726cbab-f9ce-4bf2-b81c-418065d4df95\" (UID: \"0726cbab-f9ce-4bf2-b81c-418065d4df95\") "
Jul 09 18:21:42.714022 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:42.713893    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0726cbab-f9ce-4bf2-b81c-418065d4df95-cabundle-cert\") pod \"0726cbab-f9ce-4bf2-b81c-418065d4df95\" (UID: \"0726cbab-f9ce-4bf2-b81c-418065d4df95\") "
Jul 09 18:21:42.714022 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:42.713940    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4s42l\" (UniqueName: \"kubernetes.io/projected/0726cbab-f9ce-4bf2-b81c-418065d4df95-kube-api-access-4s42l\") pod \"0726cbab-f9ce-4bf2-b81c-418065d4df95\" (UID: \"0726cbab-f9ce-4bf2-b81c-418065d4df95\") "
Jul 09 18:21:42.714180 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:42.714150    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0726cbab-f9ce-4bf2-b81c-418065d4df95-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0726cbab-f9ce-4bf2-b81c-418065d4df95" (UID: "0726cbab-f9ce-4bf2-b81c-418065d4df95"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:21:42.714294 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:42.714225    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0726cbab-f9ce-4bf2-b81c-418065d4df95-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0726cbab-f9ce-4bf2-b81c-418065d4df95" (UID: "0726cbab-f9ce-4bf2-b81c-418065d4df95"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:21:42.714395 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:42.714307    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0726cbab-f9ce-4bf2-b81c-418065d4df95-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0726cbab-f9ce-4bf2-b81c-418065d4df95" (UID: "0726cbab-f9ce-4bf2-b81c-418065d4df95"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:21:42.716005 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:42.715977    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0726cbab-f9ce-4bf2-b81c-418065d4df95-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0726cbab-f9ce-4bf2-b81c-418065d4df95" (UID: "0726cbab-f9ce-4bf2-b81c-418065d4df95"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:21:42.716100 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:42.716009    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0726cbab-f9ce-4bf2-b81c-418065d4df95-kube-api-access-4s42l" (OuterVolumeSpecName: "kube-api-access-4s42l") pod "0726cbab-f9ce-4bf2-b81c-418065d4df95" (UID: "0726cbab-f9ce-4bf2-b81c-418065d4df95"). InnerVolumeSpecName "kube-api-access-4s42l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:21:42.719870 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:42.719852    2563 scope.go:117] "RemoveContainer" containerID="1dbe7b6f20c10c11b593e18f6bcf4b013043b95a0412ecb226ede2043887c7e5"
Jul 09 18:21:42.720171 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:21:42.720152    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dbe7b6f20c10c11b593e18f6bcf4b013043b95a0412ecb226ede2043887c7e5\": container with ID starting with 1dbe7b6f20c10c11b593e18f6bcf4b013043b95a0412ecb226ede2043887c7e5 not found: ID does not exist" containerID="1dbe7b6f20c10c11b593e18f6bcf4b013043b95a0412ecb226ede2043887c7e5"
Jul 09 18:21:42.720241 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:42.720181    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dbe7b6f20c10c11b593e18f6bcf4b013043b95a0412ecb226ede2043887c7e5"} err="failed to get container status \"1dbe7b6f20c10c11b593e18f6bcf4b013043b95a0412ecb226ede2043887c7e5\": rpc error: code = NotFound desc = could not find container \"1dbe7b6f20c10c11b593e18f6bcf4b013043b95a0412ecb226ede2043887c7e5\": container with ID starting with 1dbe7b6f20c10c11b593e18f6bcf4b013043b95a0412ecb226ede2043887c7e5 not found: ID does not exist"
Jul 09 18:21:42.720241 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:42.720200    2563 scope.go:117] "RemoveContainer" containerID="01ac2fce5aa88bb9d7554fdd32ab65e5fc7cb01f4a0810f5a8d967825c2067ab"
Jul 09 18:21:42.720528 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:21:42.720502    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01ac2fce5aa88bb9d7554fdd32ab65e5fc7cb01f4a0810f5a8d967825c2067ab\": container with ID starting with 01ac2fce5aa88bb9d7554fdd32ab65e5fc7cb01f4a0810f5a8d967825c2067ab not found: ID does not exist" containerID="01ac2fce5aa88bb9d7554fdd32ab65e5fc7cb01f4a0810f5a8d967825c2067ab"
Jul 09 18:21:42.720617 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:42.720532    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01ac2fce5aa88bb9d7554fdd32ab65e5fc7cb01f4a0810f5a8d967825c2067ab"} err="failed to get container status \"01ac2fce5aa88bb9d7554fdd32ab65e5fc7cb01f4a0810f5a8d967825c2067ab\": rpc error: code = NotFound desc = could not find container \"01ac2fce5aa88bb9d7554fdd32ab65e5fc7cb01f4a0810f5a8d967825c2067ab\": container with ID starting with 01ac2fce5aa88bb9d7554fdd32ab65e5fc7cb01f4a0810f5a8d967825c2067ab not found: ID does not exist"
Jul 09 18:21:42.720617 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:42.720550    2563 scope.go:117] "RemoveContainer" containerID="c486fd7a3fc37c66d80e8296690ed146ee88784739df2b68b58027fccb2f7714"
Jul 09 18:21:42.720812 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:21:42.720791    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c486fd7a3fc37c66d80e8296690ed146ee88784739df2b68b58027fccb2f7714\": container with ID starting with c486fd7a3fc37c66d80e8296690ed146ee88784739df2b68b58027fccb2f7714 not found: ID does not exist" containerID="c486fd7a3fc37c66d80e8296690ed146ee88784739df2b68b58027fccb2f7714"
Jul 09 18:21:42.720880 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:42.720822    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c486fd7a3fc37c66d80e8296690ed146ee88784739df2b68b58027fccb2f7714"} err="failed to get container status \"c486fd7a3fc37c66d80e8296690ed146ee88784739df2b68b58027fccb2f7714\": rpc error: code = NotFound desc = could not find container \"c486fd7a3fc37c66d80e8296690ed146ee88784739df2b68b58027fccb2f7714\": container with ID starting with c486fd7a3fc37c66d80e8296690ed146ee88784739df2b68b58027fccb2f7714 not found: ID does not exist"
Jul 09 18:21:42.720880 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:42.720843    2563 scope.go:117] "RemoveContainer" containerID="dac9ea8516ebbe20f5572e26733066bbe3167a96eeac6810f165bdbb3d1866c7"
Jul 09 18:21:42.721137 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:21:42.721116    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dac9ea8516ebbe20f5572e26733066bbe3167a96eeac6810f165bdbb3d1866c7\": container with ID starting with dac9ea8516ebbe20f5572e26733066bbe3167a96eeac6810f165bdbb3d1866c7 not found: ID does not exist" containerID="dac9ea8516ebbe20f5572e26733066bbe3167a96eeac6810f165bdbb3d1866c7"
Jul 09 18:21:42.721192 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:42.721142    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dac9ea8516ebbe20f5572e26733066bbe3167a96eeac6810f165bdbb3d1866c7"} err="failed to get container status \"dac9ea8516ebbe20f5572e26733066bbe3167a96eeac6810f165bdbb3d1866c7\": rpc error: code = NotFound desc = could not find container \"dac9ea8516ebbe20f5572e26733066bbe3167a96eeac6810f165bdbb3d1866c7\": container with ID starting with dac9ea8516ebbe20f5572e26733066bbe3167a96eeac6810f165bdbb3d1866c7 not found: ID does not exist"
Jul 09 18:21:42.815580 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:42.815538    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0726cbab-f9ce-4bf2-b81c-418065d4df95-cabundle-cert\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:21:42.815580 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:42.815584    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s42l\" (UniqueName: \"kubernetes.io/projected/0726cbab-f9ce-4bf2-b81c-418065d4df95-kube-api-access-4s42l\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:21:42.815843 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:42.815609    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0726cbab-f9ce-4bf2-b81c-418065d4df95-proxy-tls\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:21:42.815843 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:42.815623    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0726cbab-f9ce-4bf2-b81c-418065d4df95-kserve-provision-location\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:21:42.815843 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:42.815636    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0726cbab-f9ce-4bf2-b81c-418065d4df95-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:21:43.007337 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:43.007305    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf"]
Jul 09 18:21:43.011512 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:43.011484    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-69476bc7f-vs7sf"]
Jul 09 18:21:44.962719 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:21:44.962680    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" path="/var/lib/kubelet/pods/0726cbab-f9ce-4bf2-b81c-418065d4df95/volumes"
Jul 09 18:24:06.408028 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:06.407935    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh" event={"ID":"81f64c27-d414-48a2-85a1-c937b484ea64","Type":"ContainerStarted","Data":"f17b8628acc92ca87229b17f9f01c2414117c2e231373a3609895f077005f970"}
Jul 09 18:24:06.408612 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:06.408039    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh" event={"ID":"81f64c27-d414-48a2-85a1-c937b484ea64","Type":"ContainerStarted","Data":"31e380a741b4560e5cf396889573a5c425b4052ab34f2fdb187acc479e519e42"}
Jul 09 18:24:06.408612 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:06.408176    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh"
Jul 09 18:24:06.439638 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:06.439580    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh" podStartSLOduration=2.231304091 podStartE2EDuration="2m54.439562552s" totalImagesPullingTime="2m23.236648435s" totalInitContainerRuntime="28.971610026s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:21:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:24:06.437762727 +0000 UTC m=+1192.106843685" watchObservedRunningTime="2026-07-09 18:24:06.439562552 +0000 UTC m=+1192.108643507"
Jul 09 18:24:07.413425 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:07.413389    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh"
Jul 09 18:24:07.414544 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:07.414508    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh" podUID="81f64c27-d414-48a2-85a1-c937b484ea64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 09 18:24:08.420178 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:08.420136    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh" podUID="81f64c27-d414-48a2-85a1-c937b484ea64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 09 18:24:13.424611 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:13.424574    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh"
Jul 09 18:24:13.425328 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:13.425292    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh" podUID="81f64c27-d414-48a2-85a1-c937b484ea64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 09 18:24:14.888357 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:14.888326    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbftk_5d463fb2-45f9-4554-9525-637e1b84351a/ovn-acl-logging/0.log"
Jul 09 18:24:14.890184 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:14.890157    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbftk_5d463fb2-45f9-4554-9525-637e1b84351a/ovn-acl-logging/0.log"
Jul 09 18:24:23.426404 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:23.426373    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh"
Jul 09 18:24:32.670017 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:32.669985    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh"]
Jul 09 18:24:32.670445 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:32.670236    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh" podUID="81f64c27-d414-48a2-85a1-c937b484ea64" containerName="kserve-container" containerID="cri-o://31e380a741b4560e5cf396889573a5c425b4052ab34f2fdb187acc479e519e42" gracePeriod=30
Jul 09 18:24:32.670445 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:32.670280    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh" podUID="81f64c27-d414-48a2-85a1-c937b484ea64" containerName="kube-rbac-proxy" containerID="cri-o://f17b8628acc92ca87229b17f9f01c2414117c2e231373a3609895f077005f970" gracePeriod=30
Jul 09 18:24:33.420475 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:33.420431    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh" podUID="81f64c27-d414-48a2-85a1-c937b484ea64" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 09 18:24:33.425813 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:33.425774    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh" podUID="81f64c27-d414-48a2-85a1-c937b484ea64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 09 18:24:33.535985 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:33.535950    2563 generic.go:356] "Generic (PLEG): container finished" podID="81f64c27-d414-48a2-85a1-c937b484ea64" containerID="f17b8628acc92ca87229b17f9f01c2414117c2e231373a3609895f077005f970" exitCode=2
Jul 09 18:24:33.535985 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:33.535981    2563 generic.go:356] "Generic (PLEG): container finished" podID="81f64c27-d414-48a2-85a1-c937b484ea64" containerID="31e380a741b4560e5cf396889573a5c425b4052ab34f2fdb187acc479e519e42" exitCode=0
Jul 09 18:24:33.536191 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:33.536020    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh" event={"ID":"81f64c27-d414-48a2-85a1-c937b484ea64","Type":"ContainerDied","Data":"f17b8628acc92ca87229b17f9f01c2414117c2e231373a3609895f077005f970"}
Jul 09 18:24:33.536191 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:33.536100    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh" event={"ID":"81f64c27-d414-48a2-85a1-c937b484ea64","Type":"ContainerDied","Data":"31e380a741b4560e5cf396889573a5c425b4052ab34f2fdb187acc479e519e42"}
Jul 09 18:24:33.646991 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:33.646957    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh"
Jul 09 18:24:33.795942 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:33.795853    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81f64c27-d414-48a2-85a1-c937b484ea64-kube-rbac-proxy-sar-config\") pod \"81f64c27-d414-48a2-85a1-c937b484ea64\" (UID: \"81f64c27-d414-48a2-85a1-c937b484ea64\") "
Jul 09 18:24:33.795942 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:33.795908    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81f64c27-d414-48a2-85a1-c937b484ea64-proxy-tls\") pod \"81f64c27-d414-48a2-85a1-c937b484ea64\" (UID: \"81f64c27-d414-48a2-85a1-c937b484ea64\") "
Jul 09 18:24:33.795942 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:33.795931    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81f64c27-d414-48a2-85a1-c937b484ea64-cabundle-cert\") pod \"81f64c27-d414-48a2-85a1-c937b484ea64\" (UID: \"81f64c27-d414-48a2-85a1-c937b484ea64\") "
Jul 09 18:24:33.796506 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:33.795950    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gqz6\" (UniqueName: \"kubernetes.io/projected/81f64c27-d414-48a2-85a1-c937b484ea64-kube-api-access-6gqz6\") pod \"81f64c27-d414-48a2-85a1-c937b484ea64\" (UID: \"81f64c27-d414-48a2-85a1-c937b484ea64\") "
Jul 09 18:24:33.796506 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:33.796013    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81f64c27-d414-48a2-85a1-c937b484ea64-kserve-provision-location\") pod \"81f64c27-d414-48a2-85a1-c937b484ea64\" (UID: \"81f64c27-d414-48a2-85a1-c937b484ea64\") "
Jul 09 18:24:33.796506 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:33.796271    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81f64c27-d414-48a2-85a1-c937b484ea64-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "81f64c27-d414-48a2-85a1-c937b484ea64" (UID: "81f64c27-d414-48a2-85a1-c937b484ea64"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:24:33.796506 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:33.796374    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81f64c27-d414-48a2-85a1-c937b484ea64-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "81f64c27-d414-48a2-85a1-c937b484ea64" (UID: "81f64c27-d414-48a2-85a1-c937b484ea64"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:24:33.798079 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:33.798056    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81f64c27-d414-48a2-85a1-c937b484ea64-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "81f64c27-d414-48a2-85a1-c937b484ea64" (UID: "81f64c27-d414-48a2-85a1-c937b484ea64"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:24:33.798165 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:33.798127    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f64c27-d414-48a2-85a1-c937b484ea64-kube-api-access-6gqz6" (OuterVolumeSpecName: "kube-api-access-6gqz6") pod "81f64c27-d414-48a2-85a1-c937b484ea64" (UID: "81f64c27-d414-48a2-85a1-c937b484ea64"). InnerVolumeSpecName "kube-api-access-6gqz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:24:33.800748 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:33.800720    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f64c27-d414-48a2-85a1-c937b484ea64-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "81f64c27-d414-48a2-85a1-c937b484ea64" (UID: "81f64c27-d414-48a2-85a1-c937b484ea64"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:24:33.896778 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:33.896738    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81f64c27-d414-48a2-85a1-c937b484ea64-kserve-provision-location\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:24:33.896778 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:33.896770    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81f64c27-d414-48a2-85a1-c937b484ea64-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:24:33.896778 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:33.896780    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81f64c27-d414-48a2-85a1-c937b484ea64-proxy-tls\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:24:33.896778 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:33.896790    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81f64c27-d414-48a2-85a1-c937b484ea64-cabundle-cert\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:24:33.897048 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:33.896799    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gqz6\" (UniqueName: \"kubernetes.io/projected/81f64c27-d414-48a2-85a1-c937b484ea64-kube-api-access-6gqz6\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:24:34.542265 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:34.542204    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh" event={"ID":"81f64c27-d414-48a2-85a1-c937b484ea64","Type":"ContainerDied","Data":"a99c8b0a9bb8a2ff33204a7e183bf4f0c8d62624d660662cbdd493d0a6a26686"}
Jul 09 18:24:34.542265 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:34.542247    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh"
Jul 09 18:24:34.542609 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:34.542299    2563 scope.go:117] "RemoveContainer" containerID="f17b8628acc92ca87229b17f9f01c2414117c2e231373a3609895f077005f970"
Jul 09 18:24:34.551714 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:34.551696    2563 scope.go:117] "RemoveContainer" containerID="31e380a741b4560e5cf396889573a5c425b4052ab34f2fdb187acc479e519e42"
Jul 09 18:24:34.559678 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:34.559662    2563 scope.go:117] "RemoveContainer" containerID="1cf4be205c9a8b8de67e84608ae46646b220ec649bfa48c927ebe47e6da08031"
Jul 09 18:24:34.565016 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:34.564991    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh"]
Jul 09 18:24:34.568778 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:34.568750    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-lhskh"]
Jul 09 18:24:34.960558 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:34.960526    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81f64c27-d414-48a2-85a1-c937b484ea64" path="/var/lib/kubelet/pods/81f64c27-d414-48a2-85a1-c937b484ea64/volumes"
Jul 09 18:24:38.044942 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.044905    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc"]
Jul 09 18:24:38.045474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.045432    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81f64c27-d414-48a2-85a1-c937b484ea64" containerName="kube-rbac-proxy"
Jul 09 18:24:38.045474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.045451    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="81f64c27-d414-48a2-85a1-c937b484ea64" containerName="kube-rbac-proxy"
Jul 09 18:24:38.045474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.045466    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="kserve-container"
Jul 09 18:24:38.045474 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.045474    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="kserve-container"
Jul 09 18:24:38.045692 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.045483    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="kube-rbac-proxy"
Jul 09 18:24:38.045692 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.045492    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="kube-rbac-proxy"
Jul 09 18:24:38.045692 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.045504    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81f64c27-d414-48a2-85a1-c937b484ea64" containerName="kserve-container"
Jul 09 18:24:38.045692 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.045512    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="81f64c27-d414-48a2-85a1-c937b484ea64" containerName="kserve-container"
Jul 09 18:24:38.045692 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.045526    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="agent"
Jul 09 18:24:38.045692 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.045536    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="agent"
Jul 09 18:24:38.045692 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.045546    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81f64c27-d414-48a2-85a1-c937b484ea64" containerName="storage-initializer"
Jul 09 18:24:38.045692 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.045555    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="81f64c27-d414-48a2-85a1-c937b484ea64" containerName="storage-initializer"
Jul 09 18:24:38.045692 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.045566    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="storage-initializer"
Jul 09 18:24:38.045692 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.045574    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="storage-initializer"
Jul 09 18:24:38.045692 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.045590    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49c252a0-9096-4936-bf5a-b07e515cd43a" containerName="kserve-container"
Jul 09 18:24:38.045692 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.045599    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="49c252a0-9096-4936-bf5a-b07e515cd43a" containerName="kserve-container"
Jul 09 18:24:38.045692 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.045609    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49c252a0-9096-4936-bf5a-b07e515cd43a" containerName="kube-rbac-proxy"
Jul 09 18:24:38.045692 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.045617    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="49c252a0-9096-4936-bf5a-b07e515cd43a" containerName="kube-rbac-proxy"
Jul 09 18:24:38.045692 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.045696    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="49c252a0-9096-4936-bf5a-b07e515cd43a" containerName="kserve-container"
Jul 09 18:24:38.046446 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.045708    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="49c252a0-9096-4936-bf5a-b07e515cd43a" containerName="kube-rbac-proxy"
Jul 09 18:24:38.046446 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.045717    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="81f64c27-d414-48a2-85a1-c937b484ea64" containerName="kserve-container"
Jul 09 18:24:38.046446 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.045726    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="kserve-container"
Jul 09 18:24:38.046446 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.045736    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="agent"
Jul 09 18:24:38.046446 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.045749    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0726cbab-f9ce-4bf2-b81c-418065d4df95" containerName="kube-rbac-proxy"
Jul 09 18:24:38.046446 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.045758    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="81f64c27-d414-48a2-85a1-c937b484ea64" containerName="kube-rbac-proxy"
Jul 09 18:24:38.084043 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.084008    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc"]
Jul 09 18:24:38.084291 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.084130    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc"
Jul 09 18:24:38.086869 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.086847    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:24:38.086993 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.086848    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:24:38.086993 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.086850    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 09 18:24:38.087719 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.087693    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 09 18:24:38.087719 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.087699    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 18:24:38.087895 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.087728    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tx5df\""
Jul 09 18:24:38.087895 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.087797    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:24:38.133342 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.133301    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfe59b41-14f8-4e60-a63b-67732ef4f07d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-shgpc\" (UID: \"cfe59b41-14f8-4e60-a63b-67732ef4f07d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc"
Jul 09 18:24:38.133525 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.133360    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8j8g\" (UniqueName: \"kubernetes.io/projected/cfe59b41-14f8-4e60-a63b-67732ef4f07d-kube-api-access-s8j8g\") pod \"isvc-autogluon-v2-predictor-67b849d456-shgpc\" (UID: \"cfe59b41-14f8-4e60-a63b-67732ef4f07d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc"
Jul 09 18:24:38.133525 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.133384    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfe59b41-14f8-4e60-a63b-67732ef4f07d-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-shgpc\" (UID: \"cfe59b41-14f8-4e60-a63b-67732ef4f07d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc"
Jul 09 18:24:38.133525 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.133403    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfe59b41-14f8-4e60-a63b-67732ef4f07d-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-shgpc\" (UID: \"cfe59b41-14f8-4e60-a63b-67732ef4f07d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc"
Jul 09 18:24:38.133525 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.133484    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfe59b41-14f8-4e60-a63b-67732ef4f07d-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-shgpc\" (UID: \"cfe59b41-14f8-4e60-a63b-67732ef4f07d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc"
Jul 09 18:24:38.234079 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.234046    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8j8g\" (UniqueName: \"kubernetes.io/projected/cfe59b41-14f8-4e60-a63b-67732ef4f07d-kube-api-access-s8j8g\") pod \"isvc-autogluon-v2-predictor-67b849d456-shgpc\" (UID: \"cfe59b41-14f8-4e60-a63b-67732ef4f07d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc"
Jul 09 18:24:38.234295 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.234087    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfe59b41-14f8-4e60-a63b-67732ef4f07d-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-shgpc\" (UID: \"cfe59b41-14f8-4e60-a63b-67732ef4f07d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc"
Jul 09 18:24:38.234295 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.234105    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfe59b41-14f8-4e60-a63b-67732ef4f07d-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-shgpc\" (UID: \"cfe59b41-14f8-4e60-a63b-67732ef4f07d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc"
Jul 09 18:24:38.234295 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.234174    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfe59b41-14f8-4e60-a63b-67732ef4f07d-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-shgpc\" (UID: \"cfe59b41-14f8-4e60-a63b-67732ef4f07d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc"
Jul 09 18:24:38.234295 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.234210    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfe59b41-14f8-4e60-a63b-67732ef4f07d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-shgpc\" (UID: \"cfe59b41-14f8-4e60-a63b-67732ef4f07d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc"
Jul 09 18:24:38.234467 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:24:38.234310    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-predictor-serving-cert: secret "isvc-autogluon-v2-predictor-serving-cert" not found
Jul 09 18:24:38.234467 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:24:38.234373    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfe59b41-14f8-4e60-a63b-67732ef4f07d-proxy-tls podName:cfe59b41-14f8-4e60-a63b-67732ef4f07d nodeName:}" failed. No retries permitted until 2026-07-09 18:24:38.734352749 +0000 UTC m=+1224.403433684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cfe59b41-14f8-4e60-a63b-67732ef4f07d-proxy-tls") pod "isvc-autogluon-v2-predictor-67b849d456-shgpc" (UID: "cfe59b41-14f8-4e60-a63b-67732ef4f07d") : secret "isvc-autogluon-v2-predictor-serving-cert" not found
Jul 09 18:24:38.234644 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.234626    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfe59b41-14f8-4e60-a63b-67732ef4f07d-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-shgpc\" (UID: \"cfe59b41-14f8-4e60-a63b-67732ef4f07d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc"
Jul 09 18:24:38.234951 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.234930    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfe59b41-14f8-4e60-a63b-67732ef4f07d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-shgpc\" (UID: \"cfe59b41-14f8-4e60-a63b-67732ef4f07d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc"
Jul 09 18:24:38.234990 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.234930    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfe59b41-14f8-4e60-a63b-67732ef4f07d-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-shgpc\" (UID: \"cfe59b41-14f8-4e60-a63b-67732ef4f07d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc"
Jul 09 18:24:38.244983 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.244958    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8j8g\" (UniqueName: \"kubernetes.io/projected/cfe59b41-14f8-4e60-a63b-67732ef4f07d-kube-api-access-s8j8g\") pod \"isvc-autogluon-v2-predictor-67b849d456-shgpc\" (UID: \"cfe59b41-14f8-4e60-a63b-67732ef4f07d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc"
Jul 09 18:24:38.737962 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.737928    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfe59b41-14f8-4e60-a63b-67732ef4f07d-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-shgpc\" (UID: \"cfe59b41-14f8-4e60-a63b-67732ef4f07d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc"
Jul 09 18:24:38.740361 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.740330    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfe59b41-14f8-4e60-a63b-67732ef4f07d-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-shgpc\" (UID: \"cfe59b41-14f8-4e60-a63b-67732ef4f07d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc"
Jul 09 18:24:38.994443 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:38.994342    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc"
Jul 09 18:24:39.142763 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:39.142730    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc"]
Jul 09 18:24:39.144776 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:24:39.144737    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfe59b41_14f8_4e60_a63b_67732ef4f07d.slice/crio-d86079787dedc9f095bd3369d313d1c8396b723eb8401fe52b862bcab8a012f0 WatchSource:0}: Error finding container d86079787dedc9f095bd3369d313d1c8396b723eb8401fe52b862bcab8a012f0: Status 404 returned error can't find the container with id d86079787dedc9f095bd3369d313d1c8396b723eb8401fe52b862bcab8a012f0
Jul 09 18:24:39.567638 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:39.567539    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc" event={"ID":"cfe59b41-14f8-4e60-a63b-67732ef4f07d","Type":"ContainerStarted","Data":"1035a56536d1d6e0450c3f1a1ec3bf872be1c3a065c4d1c1d841e53ea90c6d50"}
Jul 09 18:24:39.567638 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:39.567615    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc" event={"ID":"cfe59b41-14f8-4e60-a63b-67732ef4f07d","Type":"ContainerStarted","Data":"d86079787dedc9f095bd3369d313d1c8396b723eb8401fe52b862bcab8a012f0"}
Jul 09 18:24:44.590326 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:44.590292    2563 generic.go:356] "Generic (PLEG): container finished" podID="cfe59b41-14f8-4e60-a63b-67732ef4f07d" containerID="1035a56536d1d6e0450c3f1a1ec3bf872be1c3a065c4d1c1d841e53ea90c6d50" exitCode=0
Jul 09 18:24:44.590746 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:44.590359    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc" event={"ID":"cfe59b41-14f8-4e60-a63b-67732ef4f07d","Type":"ContainerDied","Data":"1035a56536d1d6e0450c3f1a1ec3bf872be1c3a065c4d1c1d841e53ea90c6d50"}
Jul 09 18:24:45.598085 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:45.598047    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc" event={"ID":"cfe59b41-14f8-4e60-a63b-67732ef4f07d","Type":"ContainerStarted","Data":"08fce7fba9324128466bc453e2cd03aa61b50b5a7cf3aa75d80cc52097ed4718"}
Jul 09 18:24:45.598602 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:45.598111    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc" event={"ID":"cfe59b41-14f8-4e60-a63b-67732ef4f07d","Type":"ContainerStarted","Data":"e2d9a354b7388080a73a338b76c57d4b2d6ce27611328d1a0d5e30935bcf327c"}
Jul 09 18:24:45.598602 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:45.598378    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc"
Jul 09 18:24:45.598602 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:45.598435    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc"
Jul 09 18:24:45.618328 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:45.618267    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc" podStartSLOduration=2.8689645070000003 podStartE2EDuration="7.618252451s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.749287944s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:24:45.615923009 +0000 UTC m=+1231.285003960" watchObservedRunningTime="2026-07-09 18:24:45.618252451 +0000 UTC m=+1231.287333402"
Jul 09 18:24:51.608887 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:24:51.608858    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc"
Jul 09 18:26:20.959571 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:20.959545    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc"
Jul 09 18:26:28.520911 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:28.520865    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc"]
Jul 09 18:26:28.521402 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:28.521181    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc" podUID="cfe59b41-14f8-4e60-a63b-67732ef4f07d" containerName="kserve-container" containerID="cri-o://e2d9a354b7388080a73a338b76c57d4b2d6ce27611328d1a0d5e30935bcf327c" gracePeriod=30
Jul 09 18:26:28.521402 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:28.521207    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc" podUID="cfe59b41-14f8-4e60-a63b-67732ef4f07d" containerName="kube-rbac-proxy" containerID="cri-o://08fce7fba9324128466bc453e2cd03aa61b50b5a7cf3aa75d80cc52097ed4718" gracePeriod=30
Jul 09 18:26:29.057392 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:29.057361    2563 generic.go:356] "Generic (PLEG): container finished" podID="cfe59b41-14f8-4e60-a63b-67732ef4f07d" containerID="08fce7fba9324128466bc453e2cd03aa61b50b5a7cf3aa75d80cc52097ed4718" exitCode=2
Jul 09 18:26:29.057568 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:29.057407    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc" event={"ID":"cfe59b41-14f8-4e60-a63b-67732ef4f07d","Type":"ContainerDied","Data":"08fce7fba9324128466bc453e2cd03aa61b50b5a7cf3aa75d80cc52097ed4718"}
Jul 09 18:26:29.468829 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:29.468799    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc"
Jul 09 18:26:29.508487 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:29.508454    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfe59b41-14f8-4e60-a63b-67732ef4f07d-cabundle-cert\") pod \"cfe59b41-14f8-4e60-a63b-67732ef4f07d\" (UID: \"cfe59b41-14f8-4e60-a63b-67732ef4f07d\") "
Jul 09 18:26:29.508675 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:29.508513    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8j8g\" (UniqueName: \"kubernetes.io/projected/cfe59b41-14f8-4e60-a63b-67732ef4f07d-kube-api-access-s8j8g\") pod \"cfe59b41-14f8-4e60-a63b-67732ef4f07d\" (UID: \"cfe59b41-14f8-4e60-a63b-67732ef4f07d\") "
Jul 09 18:26:29.508675 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:29.508551    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfe59b41-14f8-4e60-a63b-67732ef4f07d-proxy-tls\") pod \"cfe59b41-14f8-4e60-a63b-67732ef4f07d\" (UID: \"cfe59b41-14f8-4e60-a63b-67732ef4f07d\") "
Jul 09 18:26:29.508675 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:29.508576    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfe59b41-14f8-4e60-a63b-67732ef4f07d-kube-rbac-proxy-sar-config\") pod \"cfe59b41-14f8-4e60-a63b-67732ef4f07d\" (UID: \"cfe59b41-14f8-4e60-a63b-67732ef4f07d\") "
Jul 09 18:26:29.508675 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:29.508608    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfe59b41-14f8-4e60-a63b-67732ef4f07d-kserve-provision-location\") pod \"cfe59b41-14f8-4e60-a63b-67732ef4f07d\" (UID: \"cfe59b41-14f8-4e60-a63b-67732ef4f07d\") "
Jul 09 18:26:29.508955 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:29.508930    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfe59b41-14f8-4e60-a63b-67732ef4f07d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cfe59b41-14f8-4e60-a63b-67732ef4f07d" (UID: "cfe59b41-14f8-4e60-a63b-67732ef4f07d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:26:29.509012 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:29.508956    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfe59b41-14f8-4e60-a63b-67732ef4f07d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cfe59b41-14f8-4e60-a63b-67732ef4f07d" (UID: "cfe59b41-14f8-4e60-a63b-67732ef4f07d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:26:29.510720 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:29.510684    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfe59b41-14f8-4e60-a63b-67732ef4f07d-kube-api-access-s8j8g" (OuterVolumeSpecName: "kube-api-access-s8j8g") pod "cfe59b41-14f8-4e60-a63b-67732ef4f07d" (UID: "cfe59b41-14f8-4e60-a63b-67732ef4f07d"). InnerVolumeSpecName "kube-api-access-s8j8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:26:29.510720 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:29.510702    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfe59b41-14f8-4e60-a63b-67732ef4f07d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cfe59b41-14f8-4e60-a63b-67732ef4f07d" (UID: "cfe59b41-14f8-4e60-a63b-67732ef4f07d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:26:29.513308 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:29.513287    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe59b41-14f8-4e60-a63b-67732ef4f07d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cfe59b41-14f8-4e60-a63b-67732ef4f07d" (UID: "cfe59b41-14f8-4e60-a63b-67732ef4f07d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:26:29.609658 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:29.609570    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfe59b41-14f8-4e60-a63b-67732ef4f07d-proxy-tls\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:26:29.609658 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:29.609599    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfe59b41-14f8-4e60-a63b-67732ef4f07d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:26:29.609658 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:29.609610    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfe59b41-14f8-4e60-a63b-67732ef4f07d-kserve-provision-location\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:26:29.609658 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:29.609618    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfe59b41-14f8-4e60-a63b-67732ef4f07d-cabundle-cert\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:26:29.609658 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:29.609628    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8j8g\" (UniqueName: \"kubernetes.io/projected/cfe59b41-14f8-4e60-a63b-67732ef4f07d-kube-api-access-s8j8g\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:26:30.063902 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:30.063868    2563 generic.go:356] "Generic (PLEG): container finished" podID="cfe59b41-14f8-4e60-a63b-67732ef4f07d" containerID="e2d9a354b7388080a73a338b76c57d4b2d6ce27611328d1a0d5e30935bcf327c" exitCode=0
Jul 09 18:26:30.064087 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:30.063912    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc" event={"ID":"cfe59b41-14f8-4e60-a63b-67732ef4f07d","Type":"ContainerDied","Data":"e2d9a354b7388080a73a338b76c57d4b2d6ce27611328d1a0d5e30935bcf327c"}
Jul 09 18:26:30.064087 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:30.063926    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc"
Jul 09 18:26:30.064087 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:30.063960    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc" event={"ID":"cfe59b41-14f8-4e60-a63b-67732ef4f07d","Type":"ContainerDied","Data":"d86079787dedc9f095bd3369d313d1c8396b723eb8401fe52b862bcab8a012f0"}
Jul 09 18:26:30.064087 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:30.063988    2563 scope.go:117] "RemoveContainer" containerID="08fce7fba9324128466bc453e2cd03aa61b50b5a7cf3aa75d80cc52097ed4718"
Jul 09 18:26:30.072801 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:30.072734    2563 scope.go:117] "RemoveContainer" containerID="e2d9a354b7388080a73a338b76c57d4b2d6ce27611328d1a0d5e30935bcf327c"
Jul 09 18:26:30.080155 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:30.080134    2563 scope.go:117] "RemoveContainer" containerID="1035a56536d1d6e0450c3f1a1ec3bf872be1c3a065c4d1c1d841e53ea90c6d50"
Jul 09 18:26:30.086036 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:30.086012    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc"]
Jul 09 18:26:30.090331 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:30.090314    2563 scope.go:117] "RemoveContainer" containerID="08fce7fba9324128466bc453e2cd03aa61b50b5a7cf3aa75d80cc52097ed4718"
Jul 09 18:26:30.090412 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:30.090361    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-shgpc"]
Jul 09 18:26:30.090605 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:26:30.090588    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08fce7fba9324128466bc453e2cd03aa61b50b5a7cf3aa75d80cc52097ed4718\": container with ID starting with 08fce7fba9324128466bc453e2cd03aa61b50b5a7cf3aa75d80cc52097ed4718 not found: ID does not exist" containerID="08fce7fba9324128466bc453e2cd03aa61b50b5a7cf3aa75d80cc52097ed4718"
Jul 09 18:26:30.090645 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:30.090618    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08fce7fba9324128466bc453e2cd03aa61b50b5a7cf3aa75d80cc52097ed4718"} err="failed to get container status \"08fce7fba9324128466bc453e2cd03aa61b50b5a7cf3aa75d80cc52097ed4718\": rpc error: code = NotFound desc = could not find container \"08fce7fba9324128466bc453e2cd03aa61b50b5a7cf3aa75d80cc52097ed4718\": container with ID starting with 08fce7fba9324128466bc453e2cd03aa61b50b5a7cf3aa75d80cc52097ed4718 not found: ID does not exist"
Jul 09 18:26:30.090645 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:30.090639    2563 scope.go:117] "RemoveContainer" containerID="e2d9a354b7388080a73a338b76c57d4b2d6ce27611328d1a0d5e30935bcf327c"
Jul 09 18:26:30.090890 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:26:30.090873    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2d9a354b7388080a73a338b76c57d4b2d6ce27611328d1a0d5e30935bcf327c\": container with ID starting with e2d9a354b7388080a73a338b76c57d4b2d6ce27611328d1a0d5e30935bcf327c not found: ID does not exist" containerID="e2d9a354b7388080a73a338b76c57d4b2d6ce27611328d1a0d5e30935bcf327c"
Jul 09 18:26:30.090939 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:30.090897    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2d9a354b7388080a73a338b76c57d4b2d6ce27611328d1a0d5e30935bcf327c"} err="failed to get container status \"e2d9a354b7388080a73a338b76c57d4b2d6ce27611328d1a0d5e30935bcf327c\": rpc error: code = NotFound desc = could not find container \"e2d9a354b7388080a73a338b76c57d4b2d6ce27611328d1a0d5e30935bcf327c\": container with ID starting with e2d9a354b7388080a73a338b76c57d4b2d6ce27611328d1a0d5e30935bcf327c not found: ID does not exist"
Jul 09 18:26:30.090939 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:30.090913    2563 scope.go:117] "RemoveContainer" containerID="1035a56536d1d6e0450c3f1a1ec3bf872be1c3a065c4d1c1d841e53ea90c6d50"
Jul 09 18:26:30.091144 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:26:30.091127    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1035a56536d1d6e0450c3f1a1ec3bf872be1c3a065c4d1c1d841e53ea90c6d50\": container with ID starting with 1035a56536d1d6e0450c3f1a1ec3bf872be1c3a065c4d1c1d841e53ea90c6d50 not found: ID does not exist" containerID="1035a56536d1d6e0450c3f1a1ec3bf872be1c3a065c4d1c1d841e53ea90c6d50"
Jul 09 18:26:30.091200 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:30.091154    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1035a56536d1d6e0450c3f1a1ec3bf872be1c3a065c4d1c1d841e53ea90c6d50"} err="failed to get container status \"1035a56536d1d6e0450c3f1a1ec3bf872be1c3a065c4d1c1d841e53ea90c6d50\": rpc error: code = NotFound desc = could not find container \"1035a56536d1d6e0450c3f1a1ec3bf872be1c3a065c4d1c1d841e53ea90c6d50\": container with ID starting with 1035a56536d1d6e0450c3f1a1ec3bf872be1c3a065c4d1c1d841e53ea90c6d50 not found: ID does not exist"
Jul 09 18:26:30.960396 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:30.960360    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfe59b41-14f8-4e60-a63b-67732ef4f07d" path="/var/lib/kubelet/pods/cfe59b41-14f8-4e60-a63b-67732ef4f07d/volumes"
Jul 09 18:26:33.405734 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:33.405701    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv"]
Jul 09 18:26:33.406146 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:33.406039    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfe59b41-14f8-4e60-a63b-67732ef4f07d" containerName="kube-rbac-proxy"
Jul 09 18:26:33.406146 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:33.406050    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfe59b41-14f8-4e60-a63b-67732ef4f07d" containerName="kube-rbac-proxy"
Jul 09 18:26:33.406146 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:33.406066    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfe59b41-14f8-4e60-a63b-67732ef4f07d" containerName="storage-initializer"
Jul 09 18:26:33.406146 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:33.406071    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfe59b41-14f8-4e60-a63b-67732ef4f07d" containerName="storage-initializer"
Jul 09 18:26:33.406146 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:33.406079    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfe59b41-14f8-4e60-a63b-67732ef4f07d" containerName="kserve-container"
Jul 09 18:26:33.406146 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:33.406084    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfe59b41-14f8-4e60-a63b-67732ef4f07d" containerName="kserve-container"
Jul 09 18:26:33.406146 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:33.406139    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfe59b41-14f8-4e60-a63b-67732ef4f07d" containerName="kserve-container"
Jul 09 18:26:33.406146 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:33.406148    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfe59b41-14f8-4e60-a63b-67732ef4f07d" containerName="kube-rbac-proxy"
Jul 09 18:26:33.410390 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:33.410372    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv"
Jul 09 18:26:33.413003 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:33.412980    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:26:33.413303 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:33.413287    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:26:33.413379 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:33.413326    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 09 18:26:33.413906 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:33.413882    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 09 18:26:33.413906 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:33.413901    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 18:26:33.414044 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:33.413893    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tx5df\""
Jul 09 18:26:33.414044 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:33.413923    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:26:33.421698 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:33.421677    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv"]
Jul 09 18:26:33.442996 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:33.442964    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a33c191-2a7e-4be7-8a75-a246b06503c0-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv\" (UID: \"5a33c191-2a7e-4be7-8a75-a246b06503c0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv"
Jul 09 18:26:33.442996 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:33.442997    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a33c191-2a7e-4be7-8a75-a246b06503c0-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv\" (UID: \"5a33c191-2a7e-4be7-8a75-a246b06503c0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv"
Jul 09 18:26:33.443190 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:33.443022    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5slwc\" (UniqueName: \"kubernetes.io/projected/5a33c191-2a7e-4be7-8a75-a246b06503c0-kube-api-access-5slwc\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv\" (UID: \"5a33c191-2a7e-4be7-8a75-a246b06503c0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv"
Jul 09 18:26:33.443190 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:33.443135    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a33c191-2a7e-4be7-8a75-a246b06503c0-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv\" (UID: \"5a33c191-2a7e-4be7-8a75-a246b06503c0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv"
Jul 09 18:26:33.443190 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:33.443164    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a33c191-2a7e-4be7-8a75-a246b06503c0-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv\" (UID: \"5a33c191-2a7e-4be7-8a75-a246b06503c0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv"
Jul 09 18:26:33.543739 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:33.543704    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a33c191-2a7e-4be7-8a75-a246b06503c0-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv\" (UID: \"5a33c191-2a7e-4be7-8a75-a246b06503c0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv"
Jul 09 18:26:33.543739 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:33.543744    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a33c191-2a7e-4be7-8a75-a246b06503c0-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv\" (UID: \"5a33c191-2a7e-4be7-8a75-a246b06503c0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv"
Jul 09 18:26:33.544015 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:33.543770    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5slwc\" (UniqueName: \"kubernetes.io/projected/5a33c191-2a7e-4be7-8a75-a246b06503c0-kube-api-access-5slwc\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv\" (UID: \"5a33c191-2a7e-4be7-8a75-a246b06503c0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv"
Jul 09 18:26:33.544015 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:33.543815    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a33c191-2a7e-4be7-8a75-a246b06503c0-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv\" (UID: \"5a33c191-2a7e-4be7-8a75-a246b06503c0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv"
Jul 09 18:26:33.544015 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:33.543832    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a33c191-2a7e-4be7-8a75-a246b06503c0-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv\" (UID: \"5a33c191-2a7e-4be7-8a75-a246b06503c0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv"
Jul 09 18:26:33.544015 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:26:33.543962    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-serving-cert: secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Jul 09 18:26:33.544262 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:26:33.544032    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a33c191-2a7e-4be7-8a75-a246b06503c0-proxy-tls podName:5a33c191-2a7e-4be7-8a75-a246b06503c0 nodeName:}" failed. No retries permitted until 2026-07-09 18:26:34.044012357 +0000 UTC m=+1339.713093302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5a33c191-2a7e-4be7-8a75-a246b06503c0-proxy-tls") pod "isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv" (UID: "5a33c191-2a7e-4be7-8a75-a246b06503c0") : secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Jul 09 18:26:33.544262 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:33.544204    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a33c191-2a7e-4be7-8a75-a246b06503c0-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv\" (UID: \"5a33c191-2a7e-4be7-8a75-a246b06503c0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv"
Jul 09 18:26:33.544469 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:33.544449    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a33c191-2a7e-4be7-8a75-a246b06503c0-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv\" (UID: \"5a33c191-2a7e-4be7-8a75-a246b06503c0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv"
Jul 09 18:26:33.544507 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:33.544488    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a33c191-2a7e-4be7-8a75-a246b06503c0-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv\" (UID: \"5a33c191-2a7e-4be7-8a75-a246b06503c0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv"
Jul 09 18:26:33.553652 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:33.553627    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5slwc\" (UniqueName: \"kubernetes.io/projected/5a33c191-2a7e-4be7-8a75-a246b06503c0-kube-api-access-5slwc\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv\" (UID: \"5a33c191-2a7e-4be7-8a75-a246b06503c0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv"
Jul 09 18:26:34.049260 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:34.049182    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a33c191-2a7e-4be7-8a75-a246b06503c0-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv\" (UID: \"5a33c191-2a7e-4be7-8a75-a246b06503c0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv"
Jul 09 18:26:34.051723 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:34.051697    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a33c191-2a7e-4be7-8a75-a246b06503c0-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv\" (UID: \"5a33c191-2a7e-4be7-8a75-a246b06503c0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv"
Jul 09 18:26:34.322008 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:34.321901    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv"
Jul 09 18:26:34.456717 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:34.456690    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv"]
Jul 09 18:26:34.459262 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:26:34.459232    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a33c191_2a7e_4be7_8a75_a246b06503c0.slice/crio-41cf9cca96dd44f5dcd232a2b79dd2d43e6f770f549e70e2070237ce1d7c2c5f WatchSource:0}: Error finding container 41cf9cca96dd44f5dcd232a2b79dd2d43e6f770f549e70e2070237ce1d7c2c5f: Status 404 returned error can't find the container with id 41cf9cca96dd44f5dcd232a2b79dd2d43e6f770f549e70e2070237ce1d7c2c5f
Jul 09 18:26:35.090360 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:35.090326    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv" event={"ID":"5a33c191-2a7e-4be7-8a75-a246b06503c0","Type":"ContainerStarted","Data":"57197a232761b1d3448ec58e2539207737f836ca18738b41d2ac849d4f2469a3"}
Jul 09 18:26:35.090544 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:35.090382    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv" event={"ID":"5a33c191-2a7e-4be7-8a75-a246b06503c0","Type":"ContainerStarted","Data":"41cf9cca96dd44f5dcd232a2b79dd2d43e6f770f549e70e2070237ce1d7c2c5f"}
Jul 09 18:26:40.118126 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:40.118094    2563 generic.go:356] "Generic (PLEG): container finished" podID="5a33c191-2a7e-4be7-8a75-a246b06503c0" containerID="57197a232761b1d3448ec58e2539207737f836ca18738b41d2ac849d4f2469a3" exitCode=0
Jul 09 18:26:40.118535 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:40.118171    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv" event={"ID":"5a33c191-2a7e-4be7-8a75-a246b06503c0","Type":"ContainerDied","Data":"57197a232761b1d3448ec58e2539207737f836ca18738b41d2ac849d4f2469a3"}
Jul 09 18:26:41.125318 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:41.125287    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv" event={"ID":"5a33c191-2a7e-4be7-8a75-a246b06503c0","Type":"ContainerStarted","Data":"1f6d2de0ea7a5514d0a25e95503e440bdfa1e7467d56f08e00899a1a6d48a0f1"}
Jul 09 18:26:41.125686 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:41.125347    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv" event={"ID":"5a33c191-2a7e-4be7-8a75-a246b06503c0","Type":"ContainerStarted","Data":"2b077ff650da2f2b0b51f5d346ef6fd96bf26f37201d26605af8f8aaf64ed5bf"}
Jul 09 18:26:41.125686 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:41.125546    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv"
Jul 09 18:26:41.143921 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:41.143863    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv" podStartSLOduration=3.272331799 podStartE2EDuration="8.14384478s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.871512981s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:26:41.143635466 +0000 UTC m=+1346.812716426" watchObservedRunningTime="2026-07-09 18:26:41.14384478 +0000 UTC m=+1346.812925736"
Jul 09 18:26:42.130361 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:42.130336    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv"
Jul 09 18:26:48.140966 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:26:48.140934    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv"
Jul 09 18:28:19.959445 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:19.959410    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv"
Jul 09 18:28:23.609228 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:23.609167    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv"]
Jul 09 18:28:23.609632 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:23.609463    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv" podUID="5a33c191-2a7e-4be7-8a75-a246b06503c0" containerName="kserve-container" containerID="cri-o://2b077ff650da2f2b0b51f5d346ef6fd96bf26f37201d26605af8f8aaf64ed5bf" gracePeriod=30
Jul 09 18:28:23.609632 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:23.609513    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv" podUID="5a33c191-2a7e-4be7-8a75-a246b06503c0" containerName="kube-rbac-proxy" containerID="cri-o://1f6d2de0ea7a5514d0a25e95503e440bdfa1e7467d56f08e00899a1a6d48a0f1" gracePeriod=30
Jul 09 18:28:24.553571 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.553543    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv"
Jul 09 18:28:24.588728 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.588645    2563 generic.go:356] "Generic (PLEG): container finished" podID="5a33c191-2a7e-4be7-8a75-a246b06503c0" containerID="1f6d2de0ea7a5514d0a25e95503e440bdfa1e7467d56f08e00899a1a6d48a0f1" exitCode=2
Jul 09 18:28:24.588728 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.588668    2563 generic.go:356] "Generic (PLEG): container finished" podID="5a33c191-2a7e-4be7-8a75-a246b06503c0" containerID="2b077ff650da2f2b0b51f5d346ef6fd96bf26f37201d26605af8f8aaf64ed5bf" exitCode=0
Jul 09 18:28:24.588728 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.588717    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv"
Jul 09 18:28:24.588934 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.588726    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv" event={"ID":"5a33c191-2a7e-4be7-8a75-a246b06503c0","Type":"ContainerDied","Data":"1f6d2de0ea7a5514d0a25e95503e440bdfa1e7467d56f08e00899a1a6d48a0f1"}
Jul 09 18:28:24.588934 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.588795    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv" event={"ID":"5a33c191-2a7e-4be7-8a75-a246b06503c0","Type":"ContainerDied","Data":"2b077ff650da2f2b0b51f5d346ef6fd96bf26f37201d26605af8f8aaf64ed5bf"}
Jul 09 18:28:24.588934 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.588828    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv" event={"ID":"5a33c191-2a7e-4be7-8a75-a246b06503c0","Type":"ContainerDied","Data":"41cf9cca96dd44f5dcd232a2b79dd2d43e6f770f549e70e2070237ce1d7c2c5f"}
Jul 09 18:28:24.588934 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.588861    2563 scope.go:117] "RemoveContainer" containerID="1f6d2de0ea7a5514d0a25e95503e440bdfa1e7467d56f08e00899a1a6d48a0f1"
Jul 09 18:28:24.597245 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.597203    2563 scope.go:117] "RemoveContainer" containerID="2b077ff650da2f2b0b51f5d346ef6fd96bf26f37201d26605af8f8aaf64ed5bf"
Jul 09 18:28:24.605035 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.605012    2563 scope.go:117] "RemoveContainer" containerID="57197a232761b1d3448ec58e2539207737f836ca18738b41d2ac849d4f2469a3"
Jul 09 18:28:24.611308 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.611284    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a33c191-2a7e-4be7-8a75-a246b06503c0-proxy-tls\") pod \"5a33c191-2a7e-4be7-8a75-a246b06503c0\" (UID: \"5a33c191-2a7e-4be7-8a75-a246b06503c0\") "
Jul 09 18:28:24.611693 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.611316    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a33c191-2a7e-4be7-8a75-a246b06503c0-kserve-provision-location\") pod \"5a33c191-2a7e-4be7-8a75-a246b06503c0\" (UID: \"5a33c191-2a7e-4be7-8a75-a246b06503c0\") "
Jul 09 18:28:24.611693 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.611366    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a33c191-2a7e-4be7-8a75-a246b06503c0-cabundle-cert\") pod \"5a33c191-2a7e-4be7-8a75-a246b06503c0\" (UID: \"5a33c191-2a7e-4be7-8a75-a246b06503c0\") "
Jul 09 18:28:24.611693 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.611395    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a33c191-2a7e-4be7-8a75-a246b06503c0-kube-rbac-proxy-sar-config\") pod \"5a33c191-2a7e-4be7-8a75-a246b06503c0\" (UID: \"5a33c191-2a7e-4be7-8a75-a246b06503c0\") "
Jul 09 18:28:24.611693 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.611422    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5slwc\" (UniqueName: \"kubernetes.io/projected/5a33c191-2a7e-4be7-8a75-a246b06503c0-kube-api-access-5slwc\") pod \"5a33c191-2a7e-4be7-8a75-a246b06503c0\" (UID: \"5a33c191-2a7e-4be7-8a75-a246b06503c0\") "
Jul 09 18:28:24.611895 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.611749    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a33c191-2a7e-4be7-8a75-a246b06503c0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5a33c191-2a7e-4be7-8a75-a246b06503c0" (UID: "5a33c191-2a7e-4be7-8a75-a246b06503c0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:28:24.611895 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.611759    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a33c191-2a7e-4be7-8a75-a246b06503c0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5a33c191-2a7e-4be7-8a75-a246b06503c0" (UID: "5a33c191-2a7e-4be7-8a75-a246b06503c0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:28:24.613408 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.613383    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a33c191-2a7e-4be7-8a75-a246b06503c0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5a33c191-2a7e-4be7-8a75-a246b06503c0" (UID: "5a33c191-2a7e-4be7-8a75-a246b06503c0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:28:24.613547 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.613525    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a33c191-2a7e-4be7-8a75-a246b06503c0-kube-api-access-5slwc" (OuterVolumeSpecName: "kube-api-access-5slwc") pod "5a33c191-2a7e-4be7-8a75-a246b06503c0" (UID: "5a33c191-2a7e-4be7-8a75-a246b06503c0"). InnerVolumeSpecName "kube-api-access-5slwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:28:24.614426 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.614303    2563 scope.go:117] "RemoveContainer" containerID="1f6d2de0ea7a5514d0a25e95503e440bdfa1e7467d56f08e00899a1a6d48a0f1"
Jul 09 18:28:24.614649 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:28:24.614630    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f6d2de0ea7a5514d0a25e95503e440bdfa1e7467d56f08e00899a1a6d48a0f1\": container with ID starting with 1f6d2de0ea7a5514d0a25e95503e440bdfa1e7467d56f08e00899a1a6d48a0f1 not found: ID does not exist" containerID="1f6d2de0ea7a5514d0a25e95503e440bdfa1e7467d56f08e00899a1a6d48a0f1"
Jul 09 18:28:24.614712 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.614659    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f6d2de0ea7a5514d0a25e95503e440bdfa1e7467d56f08e00899a1a6d48a0f1"} err="failed to get container status \"1f6d2de0ea7a5514d0a25e95503e440bdfa1e7467d56f08e00899a1a6d48a0f1\": rpc error: code = NotFound desc = could not find container \"1f6d2de0ea7a5514d0a25e95503e440bdfa1e7467d56f08e00899a1a6d48a0f1\": container with ID starting with 1f6d2de0ea7a5514d0a25e95503e440bdfa1e7467d56f08e00899a1a6d48a0f1 not found: ID does not exist"
Jul 09 18:28:24.614712 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.614681    2563 scope.go:117] "RemoveContainer" containerID="2b077ff650da2f2b0b51f5d346ef6fd96bf26f37201d26605af8f8aaf64ed5bf"
Jul 09 18:28:24.614954 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:28:24.614925    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b077ff650da2f2b0b51f5d346ef6fd96bf26f37201d26605af8f8aaf64ed5bf\": container with ID starting with 2b077ff650da2f2b0b51f5d346ef6fd96bf26f37201d26605af8f8aaf64ed5bf not found: ID does not exist" containerID="2b077ff650da2f2b0b51f5d346ef6fd96bf26f37201d26605af8f8aaf64ed5bf"
Jul 09 18:28:24.615059 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.614954    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b077ff650da2f2b0b51f5d346ef6fd96bf26f37201d26605af8f8aaf64ed5bf"} err="failed to get container status \"2b077ff650da2f2b0b51f5d346ef6fd96bf26f37201d26605af8f8aaf64ed5bf\": rpc error: code = NotFound desc = could not find container \"2b077ff650da2f2b0b51f5d346ef6fd96bf26f37201d26605af8f8aaf64ed5bf\": container with ID starting with 2b077ff650da2f2b0b51f5d346ef6fd96bf26f37201d26605af8f8aaf64ed5bf not found: ID does not exist"
Jul 09 18:28:24.615059 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.614975    2563 scope.go:117] "RemoveContainer" containerID="57197a232761b1d3448ec58e2539207737f836ca18738b41d2ac849d4f2469a3"
Jul 09 18:28:24.615261 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:28:24.615240    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57197a232761b1d3448ec58e2539207737f836ca18738b41d2ac849d4f2469a3\": container with ID starting with 57197a232761b1d3448ec58e2539207737f836ca18738b41d2ac849d4f2469a3 not found: ID does not exist" containerID="57197a232761b1d3448ec58e2539207737f836ca18738b41d2ac849d4f2469a3"
Jul 09 18:28:24.615358 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.615264    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57197a232761b1d3448ec58e2539207737f836ca18738b41d2ac849d4f2469a3"} err="failed to get container status \"57197a232761b1d3448ec58e2539207737f836ca18738b41d2ac849d4f2469a3\": rpc error: code = NotFound desc = could not find container \"57197a232761b1d3448ec58e2539207737f836ca18738b41d2ac849d4f2469a3\": container with ID starting with 57197a232761b1d3448ec58e2539207737f836ca18738b41d2ac849d4f2469a3 not found: ID does not exist"
Jul 09 18:28:24.615358 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.615282    2563 scope.go:117] "RemoveContainer" containerID="1f6d2de0ea7a5514d0a25e95503e440bdfa1e7467d56f08e00899a1a6d48a0f1"
Jul 09 18:28:24.615560 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.615531    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f6d2de0ea7a5514d0a25e95503e440bdfa1e7467d56f08e00899a1a6d48a0f1"} err="failed to get container status \"1f6d2de0ea7a5514d0a25e95503e440bdfa1e7467d56f08e00899a1a6d48a0f1\": rpc error: code = NotFound desc = could not find container \"1f6d2de0ea7a5514d0a25e95503e440bdfa1e7467d56f08e00899a1a6d48a0f1\": container with ID starting with 1f6d2de0ea7a5514d0a25e95503e440bdfa1e7467d56f08e00899a1a6d48a0f1 not found: ID does not exist"
Jul 09 18:28:24.615609 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.615563    2563 scope.go:117] "RemoveContainer" containerID="2b077ff650da2f2b0b51f5d346ef6fd96bf26f37201d26605af8f8aaf64ed5bf"
Jul 09 18:28:24.615834 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.615816    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b077ff650da2f2b0b51f5d346ef6fd96bf26f37201d26605af8f8aaf64ed5bf"} err="failed to get container status \"2b077ff650da2f2b0b51f5d346ef6fd96bf26f37201d26605af8f8aaf64ed5bf\": rpc error: code = NotFound desc = could not find container \"2b077ff650da2f2b0b51f5d346ef6fd96bf26f37201d26605af8f8aaf64ed5bf\": container with ID starting with 2b077ff650da2f2b0b51f5d346ef6fd96bf26f37201d26605af8f8aaf64ed5bf not found: ID does not exist"
Jul 09 18:28:24.615887 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.615835    2563 scope.go:117] "RemoveContainer" containerID="57197a232761b1d3448ec58e2539207737f836ca18738b41d2ac849d4f2469a3"
Jul 09 18:28:24.616085 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.616063    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57197a232761b1d3448ec58e2539207737f836ca18738b41d2ac849d4f2469a3"} err="failed to get container status \"57197a232761b1d3448ec58e2539207737f836ca18738b41d2ac849d4f2469a3\": rpc error: code = NotFound desc = could not find container \"57197a232761b1d3448ec58e2539207737f836ca18738b41d2ac849d4f2469a3\": container with ID starting with 57197a232761b1d3448ec58e2539207737f836ca18738b41d2ac849d4f2469a3 not found: ID does not exist"
Jul 09 18:28:24.616949 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.616930    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a33c191-2a7e-4be7-8a75-a246b06503c0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5a33c191-2a7e-4be7-8a75-a246b06503c0" (UID: "5a33c191-2a7e-4be7-8a75-a246b06503c0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:28:24.712334 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.712298    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a33c191-2a7e-4be7-8a75-a246b06503c0-cabundle-cert\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:28:24.712334 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.712327    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a33c191-2a7e-4be7-8a75-a246b06503c0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:28:24.712334 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.712339    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5slwc\" (UniqueName: \"kubernetes.io/projected/5a33c191-2a7e-4be7-8a75-a246b06503c0-kube-api-access-5slwc\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:28:24.712574 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.712349    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a33c191-2a7e-4be7-8a75-a246b06503c0-proxy-tls\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:28:24.712574 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.712358    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a33c191-2a7e-4be7-8a75-a246b06503c0-kserve-provision-location\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:28:24.922155 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.922127    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv"]
Jul 09 18:28:24.935334 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.935303    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-h9fgv"]
Jul 09 18:28:24.961249 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:24.961198    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a33c191-2a7e-4be7-8a75-a246b06503c0" path="/var/lib/kubelet/pods/5a33c191-2a7e-4be7-8a75-a246b06503c0/volumes"
Jul 09 18:28:28.922482 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:28.922451    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh"]
Jul 09 18:28:28.922873 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:28.922858    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a33c191-2a7e-4be7-8a75-a246b06503c0" containerName="kserve-container"
Jul 09 18:28:28.922923 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:28.922875    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a33c191-2a7e-4be7-8a75-a246b06503c0" containerName="kserve-container"
Jul 09 18:28:28.922923 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:28.922890    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a33c191-2a7e-4be7-8a75-a246b06503c0" containerName="storage-initializer"
Jul 09 18:28:28.922923 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:28.922896    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a33c191-2a7e-4be7-8a75-a246b06503c0" containerName="storage-initializer"
Jul 09 18:28:28.922923 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:28.922909    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a33c191-2a7e-4be7-8a75-a246b06503c0" containerName="kube-rbac-proxy"
Jul 09 18:28:28.922923 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:28.922914    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a33c191-2a7e-4be7-8a75-a246b06503c0" containerName="kube-rbac-proxy"
Jul 09 18:28:28.923072 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:28.922978    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a33c191-2a7e-4be7-8a75-a246b06503c0" containerName="kserve-container"
Jul 09 18:28:28.923072 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:28.922987    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a33c191-2a7e-4be7-8a75-a246b06503c0" containerName="kube-rbac-proxy"
Jul 09 18:28:28.927504 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:28.927488    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh"
Jul 09 18:28:28.934229 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:28.934178    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:28:28.934229 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:28.934188    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:28:28.934464 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:28.934293    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:28:28.934464 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:28.934187    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tx5df\""
Jul 09 18:28:28.934582 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:28.934470    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 09 18:28:28.934638 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:28.934613    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 09 18:28:28.934784 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:28.934762    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 18:28:28.941942 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:28.941917    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh"]
Jul 09 18:28:29.050023 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:29.049983    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7b0c3de-2874-46cd-b0f9-6f6a9d631839-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh\" (UID: \"d7b0c3de-2874-46cd-b0f9-6f6a9d631839\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh"
Jul 09 18:28:29.050278 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:29.050182    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmh7z\" (UniqueName: \"kubernetes.io/projected/d7b0c3de-2874-46cd-b0f9-6f6a9d631839-kube-api-access-vmh7z\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh\" (UID: \"d7b0c3de-2874-46cd-b0f9-6f6a9d631839\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh"
Jul 09 18:28:29.050278 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:29.050208    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7b0c3de-2874-46cd-b0f9-6f6a9d631839-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh\" (UID: \"d7b0c3de-2874-46cd-b0f9-6f6a9d631839\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh"
Jul 09 18:28:29.050278 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:29.050272    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7b0c3de-2874-46cd-b0f9-6f6a9d631839-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh\" (UID: \"d7b0c3de-2874-46cd-b0f9-6f6a9d631839\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh"
Jul 09 18:28:29.050482 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:29.050322    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7b0c3de-2874-46cd-b0f9-6f6a9d631839-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh\" (UID: \"d7b0c3de-2874-46cd-b0f9-6f6a9d631839\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh"
Jul 09 18:28:29.151869 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:29.151830    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmh7z\" (UniqueName: \"kubernetes.io/projected/d7b0c3de-2874-46cd-b0f9-6f6a9d631839-kube-api-access-vmh7z\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh\" (UID: \"d7b0c3de-2874-46cd-b0f9-6f6a9d631839\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh"
Jul 09 18:28:29.152069 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:29.151879    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7b0c3de-2874-46cd-b0f9-6f6a9d631839-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh\" (UID: \"d7b0c3de-2874-46cd-b0f9-6f6a9d631839\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh"
Jul 09 18:28:29.152069 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:29.151919    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7b0c3de-2874-46cd-b0f9-6f6a9d631839-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh\" (UID: \"d7b0c3de-2874-46cd-b0f9-6f6a9d631839\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh"
Jul 09 18:28:29.152069 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:29.151945    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7b0c3de-2874-46cd-b0f9-6f6a9d631839-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh\" (UID: \"d7b0c3de-2874-46cd-b0f9-6f6a9d631839\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh"
Jul 09 18:28:29.152069 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:29.152011    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7b0c3de-2874-46cd-b0f9-6f6a9d631839-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh\" (UID: \"d7b0c3de-2874-46cd-b0f9-6f6a9d631839\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh"
Jul 09 18:28:29.152470 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:29.152449    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7b0c3de-2874-46cd-b0f9-6f6a9d631839-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh\" (UID: \"d7b0c3de-2874-46cd-b0f9-6f6a9d631839\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh"
Jul 09 18:28:29.152774 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:29.152751    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7b0c3de-2874-46cd-b0f9-6f6a9d631839-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh\" (UID: \"d7b0c3de-2874-46cd-b0f9-6f6a9d631839\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh"
Jul 09 18:28:29.152815 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:29.152751    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7b0c3de-2874-46cd-b0f9-6f6a9d631839-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh\" (UID: \"d7b0c3de-2874-46cd-b0f9-6f6a9d631839\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh"
Jul 09 18:28:29.154262 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:29.154243    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7b0c3de-2874-46cd-b0f9-6f6a9d631839-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh\" (UID: \"d7b0c3de-2874-46cd-b0f9-6f6a9d631839\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh"
Jul 09 18:28:29.161608 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:29.161581    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmh7z\" (UniqueName: \"kubernetes.io/projected/d7b0c3de-2874-46cd-b0f9-6f6a9d631839-kube-api-access-vmh7z\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh\" (UID: \"d7b0c3de-2874-46cd-b0f9-6f6a9d631839\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh"
Jul 09 18:28:29.244890 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:29.244784    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh"
Jul 09 18:28:29.391872 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:29.391819    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh"]
Jul 09 18:28:29.394943 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:28:29.394918    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7b0c3de_2874_46cd_b0f9_6f6a9d631839.slice/crio-6d7103e0c305cdb9d17f32fcbc959a9126ca5780b9caf56b3f2e8acfc8845afd WatchSource:0}: Error finding container 6d7103e0c305cdb9d17f32fcbc959a9126ca5780b9caf56b3f2e8acfc8845afd: Status 404 returned error can't find the container with id 6d7103e0c305cdb9d17f32fcbc959a9126ca5780b9caf56b3f2e8acfc8845afd
Jul 09 18:28:29.615685 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:29.615600    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh" event={"ID":"d7b0c3de-2874-46cd-b0f9-6f6a9d631839","Type":"ContainerStarted","Data":"e42e9066bb70dbf0b73f965437581bf9566a05360b40c9108db386be6fde6383"}
Jul 09 18:28:29.615685 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:29.615661    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh" event={"ID":"d7b0c3de-2874-46cd-b0f9-6f6a9d631839","Type":"ContainerStarted","Data":"6d7103e0c305cdb9d17f32fcbc959a9126ca5780b9caf56b3f2e8acfc8845afd"}
Jul 09 18:28:58.761555 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:58.761518    2563 generic.go:356] "Generic (PLEG): container finished" podID="d7b0c3de-2874-46cd-b0f9-6f6a9d631839" containerID="e42e9066bb70dbf0b73f965437581bf9566a05360b40c9108db386be6fde6383" exitCode=0
Jul 09 18:28:58.761965 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:58.761570    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh" event={"ID":"d7b0c3de-2874-46cd-b0f9-6f6a9d631839","Type":"ContainerDied","Data":"e42e9066bb70dbf0b73f965437581bf9566a05360b40c9108db386be6fde6383"}
Jul 09 18:28:59.769573 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:59.769534    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh" event={"ID":"d7b0c3de-2874-46cd-b0f9-6f6a9d631839","Type":"ContainerStarted","Data":"08a8ea605c7a026c2538d8a4f82ba2e81e2f6635e1fd3c0c6807a8d977ec0310"}
Jul 09 18:28:59.770068 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:59.769613    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh" event={"ID":"d7b0c3de-2874-46cd-b0f9-6f6a9d631839","Type":"ContainerStarted","Data":"df4a1de822008fffbb911918989482111d1568813af46410b3dbe46832407c67"}
Jul 09 18:28:59.770068 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:59.769829    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh"
Jul 09 18:28:59.770068 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:59.769875    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh"
Jul 09 18:28:59.790759 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:28:59.790711    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh" podStartSLOduration=2.694312538 podStartE2EDuration="31.790693293s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.096380755s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:28:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:28:59.789172741 +0000 UTC m=+1485.458253719" watchObservedRunningTime="2026-07-09 18:28:59.790693293 +0000 UTC m=+1485.459774247"
Jul 09 18:29:05.780814 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:29:05.780780    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh"
Jul 09 18:29:14.919166 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:29:14.919131    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbftk_5d463fb2-45f9-4554-9525-637e1b84351a/ovn-acl-logging/0.log"
Jul 09 18:29:14.922628 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:29:14.922608    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbftk_5d463fb2-45f9-4554-9525-637e1b84351a/ovn-acl-logging/0.log"
Jul 09 18:30:33.958536 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:33.958498    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh"
Jul 09 18:30:38.895935 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:38.895898    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh"]
Jul 09 18:30:38.901525 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:38.901481    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh" podUID="d7b0c3de-2874-46cd-b0f9-6f6a9d631839" containerName="kserve-container" containerID="cri-o://df4a1de822008fffbb911918989482111d1568813af46410b3dbe46832407c67" gracePeriod=30
Jul 09 18:30:38.901740 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:38.901703    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh" podUID="d7b0c3de-2874-46cd-b0f9-6f6a9d631839" containerName="kube-rbac-proxy" containerID="cri-o://08a8ea605c7a026c2538d8a4f82ba2e81e2f6635e1fd3c0c6807a8d977ec0310" gracePeriod=30
Jul 09 18:30:39.210424 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:39.210387    2563 generic.go:356] "Generic (PLEG): container finished" podID="d7b0c3de-2874-46cd-b0f9-6f6a9d631839" containerID="08a8ea605c7a026c2538d8a4f82ba2e81e2f6635e1fd3c0c6807a8d977ec0310" exitCode=2
Jul 09 18:30:39.210606 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:39.210456    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh" event={"ID":"d7b0c3de-2874-46cd-b0f9-6f6a9d631839","Type":"ContainerDied","Data":"08a8ea605c7a026c2538d8a4f82ba2e81e2f6635e1fd3c0c6807a8d977ec0310"}
Jul 09 18:30:39.848687 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:39.848661    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh"
Jul 09 18:30:39.911090 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:39.911004    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7b0c3de-2874-46cd-b0f9-6f6a9d631839-kube-rbac-proxy-sar-config\") pod \"d7b0c3de-2874-46cd-b0f9-6f6a9d631839\" (UID: \"d7b0c3de-2874-46cd-b0f9-6f6a9d631839\") "
Jul 09 18:30:39.911090 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:39.911038    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7b0c3de-2874-46cd-b0f9-6f6a9d631839-cabundle-cert\") pod \"d7b0c3de-2874-46cd-b0f9-6f6a9d631839\" (UID: \"d7b0c3de-2874-46cd-b0f9-6f6a9d631839\") "
Jul 09 18:30:39.911090 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:39.911062    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmh7z\" (UniqueName: \"kubernetes.io/projected/d7b0c3de-2874-46cd-b0f9-6f6a9d631839-kube-api-access-vmh7z\") pod \"d7b0c3de-2874-46cd-b0f9-6f6a9d631839\" (UID: \"d7b0c3de-2874-46cd-b0f9-6f6a9d631839\") "
Jul 09 18:30:39.911090 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:39.911088    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7b0c3de-2874-46cd-b0f9-6f6a9d631839-kserve-provision-location\") pod \"d7b0c3de-2874-46cd-b0f9-6f6a9d631839\" (UID: \"d7b0c3de-2874-46cd-b0f9-6f6a9d631839\") "
Jul 09 18:30:39.911679 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:39.911117    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7b0c3de-2874-46cd-b0f9-6f6a9d631839-proxy-tls\") pod \"d7b0c3de-2874-46cd-b0f9-6f6a9d631839\" (UID: \"d7b0c3de-2874-46cd-b0f9-6f6a9d631839\") "
Jul 09 18:30:39.911679 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:39.911458    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7b0c3de-2874-46cd-b0f9-6f6a9d631839-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d7b0c3de-2874-46cd-b0f9-6f6a9d631839" (UID: "d7b0c3de-2874-46cd-b0f9-6f6a9d631839"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:30:39.911679 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:39.911478    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7b0c3de-2874-46cd-b0f9-6f6a9d631839-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d7b0c3de-2874-46cd-b0f9-6f6a9d631839" (UID: "d7b0c3de-2874-46cd-b0f9-6f6a9d631839"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:30:39.913276 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:39.913245    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7b0c3de-2874-46cd-b0f9-6f6a9d631839-kube-api-access-vmh7z" (OuterVolumeSpecName: "kube-api-access-vmh7z") pod "d7b0c3de-2874-46cd-b0f9-6f6a9d631839" (UID: "d7b0c3de-2874-46cd-b0f9-6f6a9d631839"). InnerVolumeSpecName "kube-api-access-vmh7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:30:39.913398 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:39.913247    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7b0c3de-2874-46cd-b0f9-6f6a9d631839-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d7b0c3de-2874-46cd-b0f9-6f6a9d631839" (UID: "d7b0c3de-2874-46cd-b0f9-6f6a9d631839"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:30:39.916073 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:39.916043    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b0c3de-2874-46cd-b0f9-6f6a9d631839-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d7b0c3de-2874-46cd-b0f9-6f6a9d631839" (UID: "d7b0c3de-2874-46cd-b0f9-6f6a9d631839"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:30:40.012359 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:40.012316    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7b0c3de-2874-46cd-b0f9-6f6a9d631839-proxy-tls\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:30:40.012359 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:40.012348    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7b0c3de-2874-46cd-b0f9-6f6a9d631839-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:30:40.012359 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:40.012366    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7b0c3de-2874-46cd-b0f9-6f6a9d631839-cabundle-cert\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:30:40.012611 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:40.012451    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmh7z\" (UniqueName: \"kubernetes.io/projected/d7b0c3de-2874-46cd-b0f9-6f6a9d631839-kube-api-access-vmh7z\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:30:40.012611 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:40.012464    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7b0c3de-2874-46cd-b0f9-6f6a9d631839-kserve-provision-location\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:30:40.216096 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:40.216060    2563 generic.go:356] "Generic (PLEG): container finished" podID="d7b0c3de-2874-46cd-b0f9-6f6a9d631839" containerID="df4a1de822008fffbb911918989482111d1568813af46410b3dbe46832407c67" exitCode=0
Jul 09 18:30:40.216278 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:40.216115    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh"
Jul 09 18:30:40.216278 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:40.216145    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh" event={"ID":"d7b0c3de-2874-46cd-b0f9-6f6a9d631839","Type":"ContainerDied","Data":"df4a1de822008fffbb911918989482111d1568813af46410b3dbe46832407c67"}
Jul 09 18:30:40.216278 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:40.216208    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh" event={"ID":"d7b0c3de-2874-46cd-b0f9-6f6a9d631839","Type":"ContainerDied","Data":"6d7103e0c305cdb9d17f32fcbc959a9126ca5780b9caf56b3f2e8acfc8845afd"}
Jul 09 18:30:40.216278 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:40.216240    2563 scope.go:117] "RemoveContainer" containerID="08a8ea605c7a026c2538d8a4f82ba2e81e2f6635e1fd3c0c6807a8d977ec0310"
Jul 09 18:30:40.225426 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:40.225410    2563 scope.go:117] "RemoveContainer" containerID="df4a1de822008fffbb911918989482111d1568813af46410b3dbe46832407c67"
Jul 09 18:30:40.233125 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:40.233107    2563 scope.go:117] "RemoveContainer" containerID="e42e9066bb70dbf0b73f965437581bf9566a05360b40c9108db386be6fde6383"
Jul 09 18:30:40.238572 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:40.238549    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh"]
Jul 09 18:30:40.241517 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:40.241495    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-m2hjh"]
Jul 09 18:30:40.241897 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:40.241878    2563 scope.go:117] "RemoveContainer" containerID="08a8ea605c7a026c2538d8a4f82ba2e81e2f6635e1fd3c0c6807a8d977ec0310"
Jul 09 18:30:40.242161 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:30:40.242143    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08a8ea605c7a026c2538d8a4f82ba2e81e2f6635e1fd3c0c6807a8d977ec0310\": container with ID starting with 08a8ea605c7a026c2538d8a4f82ba2e81e2f6635e1fd3c0c6807a8d977ec0310 not found: ID does not exist" containerID="08a8ea605c7a026c2538d8a4f82ba2e81e2f6635e1fd3c0c6807a8d977ec0310"
Jul 09 18:30:40.242247 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:40.242171    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08a8ea605c7a026c2538d8a4f82ba2e81e2f6635e1fd3c0c6807a8d977ec0310"} err="failed to get container status \"08a8ea605c7a026c2538d8a4f82ba2e81e2f6635e1fd3c0c6807a8d977ec0310\": rpc error: code = NotFound desc = could not find container \"08a8ea605c7a026c2538d8a4f82ba2e81e2f6635e1fd3c0c6807a8d977ec0310\": container with ID starting with 08a8ea605c7a026c2538d8a4f82ba2e81e2f6635e1fd3c0c6807a8d977ec0310 not found: ID does not exist"
Jul 09 18:30:40.242247 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:40.242192    2563 scope.go:117] "RemoveContainer" containerID="df4a1de822008fffbb911918989482111d1568813af46410b3dbe46832407c67"
Jul 09 18:30:40.242493 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:30:40.242474    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df4a1de822008fffbb911918989482111d1568813af46410b3dbe46832407c67\": container with ID starting with df4a1de822008fffbb911918989482111d1568813af46410b3dbe46832407c67 not found: ID does not exist" containerID="df4a1de822008fffbb911918989482111d1568813af46410b3dbe46832407c67"
Jul 09 18:30:40.242548 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:40.242499    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df4a1de822008fffbb911918989482111d1568813af46410b3dbe46832407c67"} err="failed to get container status \"df4a1de822008fffbb911918989482111d1568813af46410b3dbe46832407c67\": rpc error: code = NotFound desc = could not find container \"df4a1de822008fffbb911918989482111d1568813af46410b3dbe46832407c67\": container with ID starting with df4a1de822008fffbb911918989482111d1568813af46410b3dbe46832407c67 not found: ID does not exist"
Jul 09 18:30:40.242548 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:40.242518    2563 scope.go:117] "RemoveContainer" containerID="e42e9066bb70dbf0b73f965437581bf9566a05360b40c9108db386be6fde6383"
Jul 09 18:30:40.242749 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:30:40.242727    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e42e9066bb70dbf0b73f965437581bf9566a05360b40c9108db386be6fde6383\": container with ID starting with e42e9066bb70dbf0b73f965437581bf9566a05360b40c9108db386be6fde6383 not found: ID does not exist" containerID="e42e9066bb70dbf0b73f965437581bf9566a05360b40c9108db386be6fde6383"
Jul 09 18:30:40.242791 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:40.242756    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e42e9066bb70dbf0b73f965437581bf9566a05360b40c9108db386be6fde6383"} err="failed to get container status \"e42e9066bb70dbf0b73f965437581bf9566a05360b40c9108db386be6fde6383\": rpc error: code = NotFound desc = could not find container \"e42e9066bb70dbf0b73f965437581bf9566a05360b40c9108db386be6fde6383\": container with ID starting with e42e9066bb70dbf0b73f965437581bf9566a05360b40c9108db386be6fde6383 not found: ID does not exist"
Jul 09 18:30:40.959931 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:40.959896    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7b0c3de-2874-46cd-b0f9-6f6a9d631839" path="/var/lib/kubelet/pods/d7b0c3de-2874-46cd-b0f9-6f6a9d631839/volumes"
Jul 09 18:30:44.137608 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:44.137573    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62"]
Jul 09 18:30:44.137972 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:44.137920    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7b0c3de-2874-46cd-b0f9-6f6a9d631839" containerName="storage-initializer"
Jul 09 18:30:44.137972 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:44.137933    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7b0c3de-2874-46cd-b0f9-6f6a9d631839" containerName="storage-initializer"
Jul 09 18:30:44.137972 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:44.137948    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7b0c3de-2874-46cd-b0f9-6f6a9d631839" containerName="kserve-container"
Jul 09 18:30:44.137972 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:44.137957    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7b0c3de-2874-46cd-b0f9-6f6a9d631839" containerName="kserve-container"
Jul 09 18:30:44.137972 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:44.137969    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7b0c3de-2874-46cd-b0f9-6f6a9d631839" containerName="kube-rbac-proxy"
Jul 09 18:30:44.138133 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:44.137979    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7b0c3de-2874-46cd-b0f9-6f6a9d631839" containerName="kube-rbac-proxy"
Jul 09 18:30:44.138133 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:44.138035    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7b0c3de-2874-46cd-b0f9-6f6a9d631839" containerName="kube-rbac-proxy"
Jul 09 18:30:44.138133 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:44.138042    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7b0c3de-2874-46cd-b0f9-6f6a9d631839" containerName="kserve-container"
Jul 09 18:30:44.143184 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:44.143165    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62"
Jul 09 18:30:44.146158 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:44.146128    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:30:44.146158 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:44.146155    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 18:30:44.146334 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:44.146301    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 09 18:30:44.147021 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:44.147006    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:30:44.147082 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:44.147047    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 09 18:30:44.147082 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:44.147072    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:30:44.147338 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:44.147322    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tx5df\""
Jul 09 18:30:44.155081 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:44.155059    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62"]
Jul 09 18:30:44.250442 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:44.250359    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb230b67-ec27-4b43-84e8-2d7ae5846bed-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62\" (UID: \"fb230b67-ec27-4b43-84e8-2d7ae5846bed\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62"
Jul 09 18:30:44.250442 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:44.250398    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb230b67-ec27-4b43-84e8-2d7ae5846bed-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62\" (UID: \"fb230b67-ec27-4b43-84e8-2d7ae5846bed\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62"
Jul 09 18:30:44.250442 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:44.250429    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb230b67-ec27-4b43-84e8-2d7ae5846bed-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62\" (UID: \"fb230b67-ec27-4b43-84e8-2d7ae5846bed\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62"
Jul 09 18:30:44.250660 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:44.250543    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sgpf\" (UniqueName: \"kubernetes.io/projected/fb230b67-ec27-4b43-84e8-2d7ae5846bed-kube-api-access-6sgpf\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62\" (UID: \"fb230b67-ec27-4b43-84e8-2d7ae5846bed\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62"
Jul 09 18:30:44.250660 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:44.250585    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb230b67-ec27-4b43-84e8-2d7ae5846bed-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62\" (UID: \"fb230b67-ec27-4b43-84e8-2d7ae5846bed\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62"
Jul 09 18:30:44.351411 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:44.351372    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb230b67-ec27-4b43-84e8-2d7ae5846bed-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62\" (UID: \"fb230b67-ec27-4b43-84e8-2d7ae5846bed\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62"
Jul 09 18:30:44.351411 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:44.351414    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb230b67-ec27-4b43-84e8-2d7ae5846bed-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62\" (UID: \"fb230b67-ec27-4b43-84e8-2d7ae5846bed\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62"
Jul 09 18:30:44.351663 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:44.351440    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb230b67-ec27-4b43-84e8-2d7ae5846bed-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62\" (UID: \"fb230b67-ec27-4b43-84e8-2d7ae5846bed\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62"
Jul 09 18:30:44.351663 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:44.351494    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sgpf\" (UniqueName: \"kubernetes.io/projected/fb230b67-ec27-4b43-84e8-2d7ae5846bed-kube-api-access-6sgpf\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62\" (UID: \"fb230b67-ec27-4b43-84e8-2d7ae5846bed\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62"
Jul 09 18:30:44.351663 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:44.351523    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb230b67-ec27-4b43-84e8-2d7ae5846bed-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62\" (UID: \"fb230b67-ec27-4b43-84e8-2d7ae5846bed\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62"
Jul 09 18:30:44.351846 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:44.351821    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb230b67-ec27-4b43-84e8-2d7ae5846bed-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62\" (UID: \"fb230b67-ec27-4b43-84e8-2d7ae5846bed\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62"
Jul 09 18:30:44.352170 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:44.352154    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb230b67-ec27-4b43-84e8-2d7ae5846bed-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62\" (UID: \"fb230b67-ec27-4b43-84e8-2d7ae5846bed\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62"
Jul 09 18:30:44.352230 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:44.352170    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb230b67-ec27-4b43-84e8-2d7ae5846bed-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62\" (UID: \"fb230b67-ec27-4b43-84e8-2d7ae5846bed\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62"
Jul 09 18:30:44.353836 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:44.353818    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb230b67-ec27-4b43-84e8-2d7ae5846bed-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62\" (UID: \"fb230b67-ec27-4b43-84e8-2d7ae5846bed\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62"
Jul 09 18:30:44.363011 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:44.362988    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sgpf\" (UniqueName: \"kubernetes.io/projected/fb230b67-ec27-4b43-84e8-2d7ae5846bed-kube-api-access-6sgpf\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62\" (UID: \"fb230b67-ec27-4b43-84e8-2d7ae5846bed\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62"
Jul 09 18:30:44.454192 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:44.454153    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62"
Jul 09 18:30:44.597634 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:44.597605    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62"]
Jul 09 18:30:44.599360 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:30:44.599326    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb230b67_ec27_4b43_84e8_2d7ae5846bed.slice/crio-3bde0c1a1363a1fd21a741f79a78d89505af8f37dbf47b3543b471a7581bb280 WatchSource:0}: Error finding container 3bde0c1a1363a1fd21a741f79a78d89505af8f37dbf47b3543b471a7581bb280: Status 404 returned error can't find the container with id 3bde0c1a1363a1fd21a741f79a78d89505af8f37dbf47b3543b471a7581bb280
Jul 09 18:30:45.239830 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:45.239793    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62" event={"ID":"fb230b67-ec27-4b43-84e8-2d7ae5846bed","Type":"ContainerStarted","Data":"336639e523fd04f27a487e1032fb91d9f0b9f0ffdef6fbfeae068286990213e9"}
Jul 09 18:30:45.240328 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:45.239859    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62" event={"ID":"fb230b67-ec27-4b43-84e8-2d7ae5846bed","Type":"ContainerStarted","Data":"3bde0c1a1363a1fd21a741f79a78d89505af8f37dbf47b3543b471a7581bb280"}
Jul 09 18:30:54.281753 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:54.281718    2563 generic.go:356] "Generic (PLEG): container finished" podID="fb230b67-ec27-4b43-84e8-2d7ae5846bed" containerID="336639e523fd04f27a487e1032fb91d9f0b9f0ffdef6fbfeae068286990213e9" exitCode=0
Jul 09 18:30:54.282158 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:54.281795    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62" event={"ID":"fb230b67-ec27-4b43-84e8-2d7ae5846bed","Type":"ContainerDied","Data":"336639e523fd04f27a487e1032fb91d9f0b9f0ffdef6fbfeae068286990213e9"}
Jul 09 18:30:55.289360 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:55.289317    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62" event={"ID":"fb230b67-ec27-4b43-84e8-2d7ae5846bed","Type":"ContainerStarted","Data":"75ccfaf98db46ecc21658438d9bef7bae2df3ef9ce064f03d090448f0575e173"}
Jul 09 18:30:55.289833 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:55.289390    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62" event={"ID":"fb230b67-ec27-4b43-84e8-2d7ae5846bed","Type":"ContainerStarted","Data":"c3d28a4d8472469ffd04b2e3890f328558b0d08651a1dde73c178901548b3eb1"}
Jul 09 18:30:55.289833 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:55.289670    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62"
Jul 09 18:30:55.321178 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:55.321121    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62" podStartSLOduration=2.176168814 podStartE2EDuration="11.321102662s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.144933848s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:30:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:30:55.320169099 +0000 UTC m=+1600.989250073" watchObservedRunningTime="2026-07-09 18:30:55.321102662 +0000 UTC m=+1600.990183616"
Jul 09 18:30:56.295044 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:56.295011    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62"
Jul 09 18:30:56.296332 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:56.296293    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62" podUID="fb230b67-ec27-4b43-84e8-2d7ae5846bed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 09 18:30:57.299556 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:30:57.299511    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62" podUID="fb230b67-ec27-4b43-84e8-2d7ae5846bed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 09 18:31:02.304429 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:02.304395    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62"
Jul 09 18:31:02.305023 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:02.304996    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62" podUID="fb230b67-ec27-4b43-84e8-2d7ae5846bed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 09 18:31:12.306146 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:12.306066    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62"
Jul 09 18:31:14.190527 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:14.190484    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62"]
Jul 09 18:31:14.191010 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:14.190759    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62" podUID="fb230b67-ec27-4b43-84e8-2d7ae5846bed" containerName="kserve-container" containerID="cri-o://c3d28a4d8472469ffd04b2e3890f328558b0d08651a1dde73c178901548b3eb1" gracePeriod=30
Jul 09 18:31:14.191010 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:14.190801    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62" podUID="fb230b67-ec27-4b43-84e8-2d7ae5846bed" containerName="kube-rbac-proxy" containerID="cri-o://75ccfaf98db46ecc21658438d9bef7bae2df3ef9ce064f03d090448f0575e173" gracePeriod=30
Jul 09 18:31:14.387142 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:14.387104    2563 generic.go:356] "Generic (PLEG): container finished" podID="fb230b67-ec27-4b43-84e8-2d7ae5846bed" containerID="75ccfaf98db46ecc21658438d9bef7bae2df3ef9ce064f03d090448f0575e173" exitCode=2
Jul 09 18:31:14.387374 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:14.387170    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62" event={"ID":"fb230b67-ec27-4b43-84e8-2d7ae5846bed","Type":"ContainerDied","Data":"75ccfaf98db46ecc21658438d9bef7bae2df3ef9ce064f03d090448f0575e173"}
Jul 09 18:31:15.039108 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:15.039081    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62"
Jul 09 18:31:15.130735 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:15.130702    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb230b67-ec27-4b43-84e8-2d7ae5846bed-cabundle-cert\") pod \"fb230b67-ec27-4b43-84e8-2d7ae5846bed\" (UID: \"fb230b67-ec27-4b43-84e8-2d7ae5846bed\") "
Jul 09 18:31:15.130937 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:15.130753    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb230b67-ec27-4b43-84e8-2d7ae5846bed-kube-rbac-proxy-sar-config\") pod \"fb230b67-ec27-4b43-84e8-2d7ae5846bed\" (UID: \"fb230b67-ec27-4b43-84e8-2d7ae5846bed\") "
Jul 09 18:31:15.130937 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:15.130782    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb230b67-ec27-4b43-84e8-2d7ae5846bed-kserve-provision-location\") pod \"fb230b67-ec27-4b43-84e8-2d7ae5846bed\" (UID: \"fb230b67-ec27-4b43-84e8-2d7ae5846bed\") "
Jul 09 18:31:15.130937 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:15.130800    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sgpf\" (UniqueName: \"kubernetes.io/projected/fb230b67-ec27-4b43-84e8-2d7ae5846bed-kube-api-access-6sgpf\") pod \"fb230b67-ec27-4b43-84e8-2d7ae5846bed\" (UID: \"fb230b67-ec27-4b43-84e8-2d7ae5846bed\") "
Jul 09 18:31:15.130937 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:15.130902    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb230b67-ec27-4b43-84e8-2d7ae5846bed-proxy-tls\") pod \"fb230b67-ec27-4b43-84e8-2d7ae5846bed\" (UID: \"fb230b67-ec27-4b43-84e8-2d7ae5846bed\") "
Jul 09 18:31:15.131184 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:15.131153    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb230b67-ec27-4b43-84e8-2d7ae5846bed-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fb230b67-ec27-4b43-84e8-2d7ae5846bed" (UID: "fb230b67-ec27-4b43-84e8-2d7ae5846bed"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:31:15.131184 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:15.131165    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb230b67-ec27-4b43-84e8-2d7ae5846bed-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fb230b67-ec27-4b43-84e8-2d7ae5846bed" (UID: "fb230b67-ec27-4b43-84e8-2d7ae5846bed"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:31:15.131817 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:15.131789    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb230b67-ec27-4b43-84e8-2d7ae5846bed-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fb230b67-ec27-4b43-84e8-2d7ae5846bed" (UID: "fb230b67-ec27-4b43-84e8-2d7ae5846bed"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:31:15.132884 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:15.132855    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb230b67-ec27-4b43-84e8-2d7ae5846bed-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fb230b67-ec27-4b43-84e8-2d7ae5846bed" (UID: "fb230b67-ec27-4b43-84e8-2d7ae5846bed"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:31:15.132998 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:15.132890    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb230b67-ec27-4b43-84e8-2d7ae5846bed-kube-api-access-6sgpf" (OuterVolumeSpecName: "kube-api-access-6sgpf") pod "fb230b67-ec27-4b43-84e8-2d7ae5846bed" (UID: "fb230b67-ec27-4b43-84e8-2d7ae5846bed"). InnerVolumeSpecName "kube-api-access-6sgpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:31:15.232258 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:15.232197    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb230b67-ec27-4b43-84e8-2d7ae5846bed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:31:15.232258 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:15.232250    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb230b67-ec27-4b43-84e8-2d7ae5846bed-kserve-provision-location\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:31:15.232258 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:15.232261    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sgpf\" (UniqueName: \"kubernetes.io/projected/fb230b67-ec27-4b43-84e8-2d7ae5846bed-kube-api-access-6sgpf\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:31:15.232258 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:15.232270    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb230b67-ec27-4b43-84e8-2d7ae5846bed-proxy-tls\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:31:15.232728 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:15.232279    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb230b67-ec27-4b43-84e8-2d7ae5846bed-cabundle-cert\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:31:15.394144 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:15.394043    2563 generic.go:356] "Generic (PLEG): container finished" podID="fb230b67-ec27-4b43-84e8-2d7ae5846bed" containerID="c3d28a4d8472469ffd04b2e3890f328558b0d08651a1dde73c178901548b3eb1" exitCode=0
Jul 09 18:31:15.394144 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:15.394113    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62"
Jul 09 18:31:15.394358 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:15.394132    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62" event={"ID":"fb230b67-ec27-4b43-84e8-2d7ae5846bed","Type":"ContainerDied","Data":"c3d28a4d8472469ffd04b2e3890f328558b0d08651a1dde73c178901548b3eb1"}
Jul 09 18:31:15.394358 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:15.394200    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62" event={"ID":"fb230b67-ec27-4b43-84e8-2d7ae5846bed","Type":"ContainerDied","Data":"3bde0c1a1363a1fd21a741f79a78d89505af8f37dbf47b3543b471a7581bb280"}
Jul 09 18:31:15.394358 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:15.394243    2563 scope.go:117] "RemoveContainer" containerID="75ccfaf98db46ecc21658438d9bef7bae2df3ef9ce064f03d090448f0575e173"
Jul 09 18:31:15.403342 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:15.403322    2563 scope.go:117] "RemoveContainer" containerID="c3d28a4d8472469ffd04b2e3890f328558b0d08651a1dde73c178901548b3eb1"
Jul 09 18:31:15.410964 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:15.410943    2563 scope.go:117] "RemoveContainer" containerID="336639e523fd04f27a487e1032fb91d9f0b9f0ffdef6fbfeae068286990213e9"
Jul 09 18:31:15.418669 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:15.418648    2563 scope.go:117] "RemoveContainer" containerID="75ccfaf98db46ecc21658438d9bef7bae2df3ef9ce064f03d090448f0575e173"
Jul 09 18:31:15.419023 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:31:15.418983    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75ccfaf98db46ecc21658438d9bef7bae2df3ef9ce064f03d090448f0575e173\": container with ID starting with 75ccfaf98db46ecc21658438d9bef7bae2df3ef9ce064f03d090448f0575e173 not found: ID does not exist" containerID="75ccfaf98db46ecc21658438d9bef7bae2df3ef9ce064f03d090448f0575e173"
Jul 09 18:31:15.419277 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:15.419036    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ccfaf98db46ecc21658438d9bef7bae2df3ef9ce064f03d090448f0575e173"} err="failed to get container status \"75ccfaf98db46ecc21658438d9bef7bae2df3ef9ce064f03d090448f0575e173\": rpc error: code = NotFound desc = could not find container \"75ccfaf98db46ecc21658438d9bef7bae2df3ef9ce064f03d090448f0575e173\": container with ID starting with 75ccfaf98db46ecc21658438d9bef7bae2df3ef9ce064f03d090448f0575e173 not found: ID does not exist"
Jul 09 18:31:15.419277 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:15.419058    2563 scope.go:117] "RemoveContainer" containerID="c3d28a4d8472469ffd04b2e3890f328558b0d08651a1dde73c178901548b3eb1"
Jul 09 18:31:15.419461 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:31:15.419446    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3d28a4d8472469ffd04b2e3890f328558b0d08651a1dde73c178901548b3eb1\": container with ID starting with c3d28a4d8472469ffd04b2e3890f328558b0d08651a1dde73c178901548b3eb1 not found: ID does not exist" containerID="c3d28a4d8472469ffd04b2e3890f328558b0d08651a1dde73c178901548b3eb1"
Jul 09 18:31:15.419524 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:15.419501    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3d28a4d8472469ffd04b2e3890f328558b0d08651a1dde73c178901548b3eb1"} err="failed to get container status \"c3d28a4d8472469ffd04b2e3890f328558b0d08651a1dde73c178901548b3eb1\": rpc error: code = NotFound desc = could not find container \"c3d28a4d8472469ffd04b2e3890f328558b0d08651a1dde73c178901548b3eb1\": container with ID starting with c3d28a4d8472469ffd04b2e3890f328558b0d08651a1dde73c178901548b3eb1 not found: ID does not exist"
Jul 09 18:31:15.419576 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:15.419525    2563 scope.go:117] "RemoveContainer" containerID="336639e523fd04f27a487e1032fb91d9f0b9f0ffdef6fbfeae068286990213e9"
Jul 09 18:31:15.419863 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:31:15.419844    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"336639e523fd04f27a487e1032fb91d9f0b9f0ffdef6fbfeae068286990213e9\": container with ID starting with 336639e523fd04f27a487e1032fb91d9f0b9f0ffdef6fbfeae068286990213e9 not found: ID does not exist" containerID="336639e523fd04f27a487e1032fb91d9f0b9f0ffdef6fbfeae068286990213e9"
Jul 09 18:31:15.419921 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:15.419870    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"336639e523fd04f27a487e1032fb91d9f0b9f0ffdef6fbfeae068286990213e9"} err="failed to get container status \"336639e523fd04f27a487e1032fb91d9f0b9f0ffdef6fbfeae068286990213e9\": rpc error: code = NotFound desc = could not find container \"336639e523fd04f27a487e1032fb91d9f0b9f0ffdef6fbfeae068286990213e9\": container with ID starting with 336639e523fd04f27a487e1032fb91d9f0b9f0ffdef6fbfeae068286990213e9 not found: ID does not exist"
Jul 09 18:31:15.421248 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:15.421229    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62"]
Jul 09 18:31:15.424132 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:15.424112    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gwf62"]
Jul 09 18:31:16.959792 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:16.959758    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb230b67-ec27-4b43-84e8-2d7ae5846bed" path="/var/lib/kubelet/pods/fb230b67-ec27-4b43-84e8-2d7ae5846bed/volumes"
Jul 09 18:31:20.185281 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:20.185243    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b54dt/must-gather-b7ldd"]
Jul 09 18:31:20.185684 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:20.185589    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb230b67-ec27-4b43-84e8-2d7ae5846bed" containerName="kserve-container"
Jul 09 18:31:20.185684 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:20.185600    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb230b67-ec27-4b43-84e8-2d7ae5846bed" containerName="kserve-container"
Jul 09 18:31:20.185684 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:20.185610    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb230b67-ec27-4b43-84e8-2d7ae5846bed" containerName="storage-initializer"
Jul 09 18:31:20.185684 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:20.185616    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb230b67-ec27-4b43-84e8-2d7ae5846bed" containerName="storage-initializer"
Jul 09 18:31:20.185684 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:20.185626    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb230b67-ec27-4b43-84e8-2d7ae5846bed" containerName="kube-rbac-proxy"
Jul 09 18:31:20.185684 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:20.185631    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb230b67-ec27-4b43-84e8-2d7ae5846bed" containerName="kube-rbac-proxy"
Jul 09 18:31:20.185684 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:20.185677    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb230b67-ec27-4b43-84e8-2d7ae5846bed" containerName="kserve-container"
Jul 09 18:31:20.185684 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:20.185689    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb230b67-ec27-4b43-84e8-2d7ae5846bed" containerName="kube-rbac-proxy"
Jul 09 18:31:20.190395 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:20.190375    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b54dt/must-gather-b7ldd"
Jul 09 18:31:20.192985 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:20.192959    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b54dt\"/\"openshift-service-ca.crt\""
Jul 09 18:31:20.193226 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:20.193153    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-b54dt\"/\"default-dockercfg-4t5k9\""
Jul 09 18:31:20.193335 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:20.193253    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b54dt\"/\"kube-root-ca.crt\""
Jul 09 18:31:20.201668 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:20.201645    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b54dt/must-gather-b7ldd"]
Jul 09 18:31:20.275094 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:20.275052    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/19d069db-5795-480b-984b-b58c1172a626-must-gather-output\") pod \"must-gather-b7ldd\" (UID: \"19d069db-5795-480b-984b-b58c1172a626\") " pod="openshift-must-gather-b54dt/must-gather-b7ldd"
Jul 09 18:31:20.275304 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:20.275127    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmbkb\" (UniqueName: \"kubernetes.io/projected/19d069db-5795-480b-984b-b58c1172a626-kube-api-access-rmbkb\") pod \"must-gather-b7ldd\" (UID: \"19d069db-5795-480b-984b-b58c1172a626\") " pod="openshift-must-gather-b54dt/must-gather-b7ldd"
Jul 09 18:31:20.376542 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:20.376505    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/19d069db-5795-480b-984b-b58c1172a626-must-gather-output\") pod \"must-gather-b7ldd\" (UID: \"19d069db-5795-480b-984b-b58c1172a626\") " pod="openshift-must-gather-b54dt/must-gather-b7ldd"
Jul 09 18:31:20.376706 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:20.376573    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmbkb\" (UniqueName: \"kubernetes.io/projected/19d069db-5795-480b-984b-b58c1172a626-kube-api-access-rmbkb\") pod \"must-gather-b7ldd\" (UID: \"19d069db-5795-480b-984b-b58c1172a626\") " pod="openshift-must-gather-b54dt/must-gather-b7ldd"
Jul 09 18:31:20.376853 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:20.376833    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/19d069db-5795-480b-984b-b58c1172a626-must-gather-output\") pod \"must-gather-b7ldd\" (UID: \"19d069db-5795-480b-984b-b58c1172a626\") " pod="openshift-must-gather-b54dt/must-gather-b7ldd"
Jul 09 18:31:20.384769 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:20.384741    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmbkb\" (UniqueName: \"kubernetes.io/projected/19d069db-5795-480b-984b-b58c1172a626-kube-api-access-rmbkb\") pod \"must-gather-b7ldd\" (UID: \"19d069db-5795-480b-984b-b58c1172a626\") " pod="openshift-must-gather-b54dt/must-gather-b7ldd"
Jul 09 18:31:20.514824 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:20.514734    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b54dt/must-gather-b7ldd"
Jul 09 18:31:20.646587 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:20.646565    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b54dt/must-gather-b7ldd"]
Jul 09 18:31:20.649358 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:31:20.649331    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19d069db_5795_480b_984b_b58c1172a626.slice/crio-865bf8227a6dafdee609a9a493af62f0b1dd5055c98b8ce743c06b16eb2cb389 WatchSource:0}: Error finding container 865bf8227a6dafdee609a9a493af62f0b1dd5055c98b8ce743c06b16eb2cb389: Status 404 returned error can't find the container with id 865bf8227a6dafdee609a9a493af62f0b1dd5055c98b8ce743c06b16eb2cb389
Jul 09 18:31:20.651048 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:20.651033    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:31:21.422194 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:21.422157    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b54dt/must-gather-b7ldd" event={"ID":"19d069db-5795-480b-984b-b58c1172a626","Type":"ContainerStarted","Data":"865bf8227a6dafdee609a9a493af62f0b1dd5055c98b8ce743c06b16eb2cb389"}
Jul 09 18:31:28.463453 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:28.463415    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b54dt/must-gather-b7ldd" event={"ID":"19d069db-5795-480b-984b-b58c1172a626","Type":"ContainerStarted","Data":"3720b9f1ec2120ec01a28ec7cc9436277ea6ded26d87e62d433bd46ca246f208"}
Jul 09 18:31:28.463453 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:28.463458    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b54dt/must-gather-b7ldd" event={"ID":"19d069db-5795-480b-984b-b58c1172a626","Type":"ContainerStarted","Data":"4cafebb02a7aec4dd8084a0b23929eb5b54971a3c5b2267e8436d79960a09690"}
Jul 09 18:31:28.483478 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:28.483430    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-b54dt/must-gather-b7ldd" podStartSLOduration=1.759426361 podStartE2EDuration="8.483414807s" totalImagesPullingTime="6.723988446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:31:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:31:28.480730249 +0000 UTC m=+1634.149811205" watchObservedRunningTime="2026-07-09 18:31:28.483414807 +0000 UTC m=+1634.152495761"
Jul 09 18:31:49.576031 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:49.575999    2563 generic.go:356] "Generic (PLEG): container finished" podID="19d069db-5795-480b-984b-b58c1172a626" containerID="4cafebb02a7aec4dd8084a0b23929eb5b54971a3c5b2267e8436d79960a09690" exitCode=0
Jul 09 18:31:49.576492 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:49.576059    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b54dt/must-gather-b7ldd" event={"ID":"19d069db-5795-480b-984b-b58c1172a626","Type":"ContainerDied","Data":"4cafebb02a7aec4dd8084a0b23929eb5b54971a3c5b2267e8436d79960a09690"}
Jul 09 18:31:49.576492 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:49.576411    2563 scope.go:117] "RemoveContainer" containerID="4cafebb02a7aec4dd8084a0b23929eb5b54971a3c5b2267e8436d79960a09690"
Jul 09 18:31:49.891002 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:49.890918    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-b54dt_must-gather-b7ldd_19d069db-5795-480b-984b-b58c1172a626/gather/0.log"
Jul 09 18:31:50.436371 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:50.436333    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sbwk5/must-gather-jk5vc"]
Jul 09 18:31:50.439034 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:50.438990    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sbwk5/must-gather-jk5vc"
Jul 09 18:31:50.441682 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:50.441653    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sbwk5\"/\"openshift-service-ca.crt\""
Jul 09 18:31:50.442602 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:50.442585    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sbwk5\"/\"kube-root-ca.crt\""
Jul 09 18:31:50.443511 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:50.443494    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sbwk5\"/\"default-dockercfg-w7zhs\""
Jul 09 18:31:50.449911 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:50.449887    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sbwk5/must-gather-jk5vc"]
Jul 09 18:31:50.563159 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:50.563118    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bac17e2b-97d6-4c08-846e-dbf783a191e4-must-gather-output\") pod \"must-gather-jk5vc\" (UID: \"bac17e2b-97d6-4c08-846e-dbf783a191e4\") " pod="openshift-must-gather-sbwk5/must-gather-jk5vc"
Jul 09 18:31:50.563362 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:50.563189    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9kfk\" (UniqueName: \"kubernetes.io/projected/bac17e2b-97d6-4c08-846e-dbf783a191e4-kube-api-access-k9kfk\") pod \"must-gather-jk5vc\" (UID: \"bac17e2b-97d6-4c08-846e-dbf783a191e4\") " pod="openshift-must-gather-sbwk5/must-gather-jk5vc"
Jul 09 18:31:50.664409 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:50.664362    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bac17e2b-97d6-4c08-846e-dbf783a191e4-must-gather-output\") pod \"must-gather-jk5vc\" (UID: \"bac17e2b-97d6-4c08-846e-dbf783a191e4\") " pod="openshift-must-gather-sbwk5/must-gather-jk5vc"
Jul 09 18:31:50.664880 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:50.664457    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9kfk\" (UniqueName: \"kubernetes.io/projected/bac17e2b-97d6-4c08-846e-dbf783a191e4-kube-api-access-k9kfk\") pod \"must-gather-jk5vc\" (UID: \"bac17e2b-97d6-4c08-846e-dbf783a191e4\") " pod="openshift-must-gather-sbwk5/must-gather-jk5vc"
Jul 09 18:31:50.664880 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:50.664720    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bac17e2b-97d6-4c08-846e-dbf783a191e4-must-gather-output\") pod \"must-gather-jk5vc\" (UID: \"bac17e2b-97d6-4c08-846e-dbf783a191e4\") " pod="openshift-must-gather-sbwk5/must-gather-jk5vc"
Jul 09 18:31:50.673300 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:50.673262    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9kfk\" (UniqueName: \"kubernetes.io/projected/bac17e2b-97d6-4c08-846e-dbf783a191e4-kube-api-access-k9kfk\") pod \"must-gather-jk5vc\" (UID: \"bac17e2b-97d6-4c08-846e-dbf783a191e4\") " pod="openshift-must-gather-sbwk5/must-gather-jk5vc"
Jul 09 18:31:50.748784 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:50.748688    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sbwk5/must-gather-jk5vc"
Jul 09 18:31:50.882272 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:50.882241    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sbwk5/must-gather-jk5vc"]
Jul 09 18:31:50.884805 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:31:50.884782    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbac17e2b_97d6_4c08_846e_dbf783a191e4.slice/crio-36b9903af1f8050e3bd9a6e57e99fb13d2c23a61ec6584ae6da9c3acf0ac3a59 WatchSource:0}: Error finding container 36b9903af1f8050e3bd9a6e57e99fb13d2c23a61ec6584ae6da9c3acf0ac3a59: Status 404 returned error can't find the container with id 36b9903af1f8050e3bd9a6e57e99fb13d2c23a61ec6584ae6da9c3acf0ac3a59
Jul 09 18:31:51.587855 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:51.587811    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sbwk5/must-gather-jk5vc" event={"ID":"bac17e2b-97d6-4c08-846e-dbf783a191e4","Type":"ContainerStarted","Data":"36b9903af1f8050e3bd9a6e57e99fb13d2c23a61ec6584ae6da9c3acf0ac3a59"}
Jul 09 18:31:52.596721 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:52.596679    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sbwk5/must-gather-jk5vc" event={"ID":"bac17e2b-97d6-4c08-846e-dbf783a191e4","Type":"ContainerStarted","Data":"8f9fdb7c7724b6ed904884ca0561240a056f7eb5cf0848c96664126ff202ad11"}
Jul 09 18:31:52.597118 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:52.596754    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sbwk5/must-gather-jk5vc" event={"ID":"bac17e2b-97d6-4c08-846e-dbf783a191e4","Type":"ContainerStarted","Data":"12fd6e4a3fafcab8306f4630bc3046f8750182eefdb4eaec98f5244d8041dd59"}
Jul 09 18:31:52.619086 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:52.619012    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sbwk5/must-gather-jk5vc" podStartSLOduration=1.7852500409999998 podStartE2EDuration="2.618987251s" totalImagesPullingTime="833.73721ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:31:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:31:52.617516197 +0000 UTC m=+1658.286597154" watchObservedRunningTime="2026-07-09 18:31:52.618987251 +0000 UTC m=+1658.288068207"
Jul 09 18:31:53.172948 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:53.172918    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6ng78_cddd2f06-2abf-4246-ac09-acc38c065830/global-pull-secret-syncer/0.log"
Jul 09 18:31:53.398862 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:53.398812    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tvwn2_f7fc15e3-7222-4098-8933-cd0b6cd499bd/konnectivity-agent/0.log"
Jul 09 18:31:53.427821 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:53.427749    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-181.ec2.internal_72b647e293c446e9af86e2db9bbd3ef8/haproxy/0.log"
Jul 09 18:31:55.275785 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:55.275734    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-b54dt/must-gather-b7ldd"]
Jul 09 18:31:55.276282 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:55.275998    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-b54dt/must-gather-b7ldd" podUID="19d069db-5795-480b-984b-b58c1172a626" containerName="copy" containerID="cri-o://3720b9f1ec2120ec01a28ec7cc9436277ea6ded26d87e62d433bd46ca246f208" gracePeriod=2
Jul 09 18:31:55.283995 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:55.282891    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-b54dt/must-gather-b7ldd"]
Jul 09 18:31:55.650014 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:55.649987    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-b54dt_must-gather-b7ldd_19d069db-5795-480b-984b-b58c1172a626/copy/0.log"
Jul 09 18:31:55.650683 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:55.650658    2563 generic.go:356] "Generic (PLEG): container finished" podID="19d069db-5795-480b-984b-b58c1172a626" containerID="3720b9f1ec2120ec01a28ec7cc9436277ea6ded26d87e62d433bd46ca246f208" exitCode=143
Jul 09 18:31:55.705362 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:55.705271    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-b54dt_must-gather-b7ldd_19d069db-5795-480b-984b-b58c1172a626/copy/0.log"
Jul 09 18:31:55.708090 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:55.707820    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b54dt/must-gather-b7ldd"
Jul 09 18:31:55.715464 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:31:55.712440    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-b7ldd\" is forbidden: User \"system:node:ip-10-0-129-181.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-b54dt\": no relationship found between node 'ip-10-0-129-181.ec2.internal' and this object" podUID="19d069db-5795-480b-984b-b58c1172a626" pod="openshift-must-gather-b54dt/must-gather-b7ldd"
Jul 09 18:31:55.814688 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:55.814596    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/19d069db-5795-480b-984b-b58c1172a626-must-gather-output\") pod \"19d069db-5795-480b-984b-b58c1172a626\" (UID: \"19d069db-5795-480b-984b-b58c1172a626\") "
Jul 09 18:31:55.814873 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:55.814722    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmbkb\" (UniqueName: \"kubernetes.io/projected/19d069db-5795-480b-984b-b58c1172a626-kube-api-access-rmbkb\") pod \"19d069db-5795-480b-984b-b58c1172a626\" (UID: \"19d069db-5795-480b-984b-b58c1172a626\") "
Jul 09 18:31:55.816608 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:55.816568    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19d069db-5795-480b-984b-b58c1172a626-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "19d069db-5795-480b-984b-b58c1172a626" (UID: "19d069db-5795-480b-984b-b58c1172a626"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:31:55.827596 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:55.827555    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19d069db-5795-480b-984b-b58c1172a626-kube-api-access-rmbkb" (OuterVolumeSpecName: "kube-api-access-rmbkb") pod "19d069db-5795-480b-984b-b58c1172a626" (UID: "19d069db-5795-480b-984b-b58c1172a626"). InnerVolumeSpecName "kube-api-access-rmbkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:31:55.916333 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:55.916299    2563 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/19d069db-5795-480b-984b-b58c1172a626-must-gather-output\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:31:55.916578 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:55.916562    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmbkb\" (UniqueName: \"kubernetes.io/projected/19d069db-5795-480b-984b-b58c1172a626-kube-api-access-rmbkb\") on node \"ip-10-0-129-181.ec2.internal\" DevicePath \"\""
Jul 09 18:31:56.660693 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:56.660664    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-b54dt_must-gather-b7ldd_19d069db-5795-480b-984b-b58c1172a626/copy/0.log"
Jul 09 18:31:56.662158 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:56.662120    2563 scope.go:117] "RemoveContainer" containerID="3720b9f1ec2120ec01a28ec7cc9436277ea6ded26d87e62d433bd46ca246f208"
Jul 09 18:31:56.662404 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:56.662385    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b54dt/must-gather-b7ldd"
Jul 09 18:31:56.665088 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:31:56.665040    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-b7ldd\" is forbidden: User \"system:node:ip-10-0-129-181.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-b54dt\": no relationship found between node 'ip-10-0-129-181.ec2.internal' and this object" podUID="19d069db-5795-480b-984b-b58c1172a626" pod="openshift-must-gather-b54dt/must-gather-b7ldd"
Jul 09 18:31:56.674936 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:56.674807    2563 scope.go:117] "RemoveContainer" containerID="4cafebb02a7aec4dd8084a0b23929eb5b54971a3c5b2267e8436d79960a09690"
Jul 09 18:31:56.681994 ip-10-0-129-181 kubenswrapper[2563]: E0709 18:31:56.681951    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-b7ldd\" is forbidden: User \"system:node:ip-10-0-129-181.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-b54dt\": no relationship found between node 'ip-10-0-129-181.ec2.internal' and this object" podUID="19d069db-5795-480b-984b-b58c1172a626" pod="openshift-must-gather-b54dt/must-gather-b7ldd"
Jul 09 18:31:56.963289 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:56.963245    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19d069db-5795-480b-984b-b58c1172a626" path="/var/lib/kubelet/pods/19d069db-5795-480b-984b-b58c1172a626/volumes"
Jul 09 18:31:57.815061 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:57.815024    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0ecd97df-2046-4189-a979-df898eb3faaa/alertmanager/0.log"
Jul 09 18:31:57.857864 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:57.857832    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0ecd97df-2046-4189-a979-df898eb3faaa/config-reloader/0.log"
Jul 09 18:31:57.895594 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:57.895567    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0ecd97df-2046-4189-a979-df898eb3faaa/kube-rbac-proxy-web/0.log"
Jul 09 18:31:57.930622 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:57.930523    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0ecd97df-2046-4189-a979-df898eb3faaa/kube-rbac-proxy/0.log"
Jul 09 18:31:57.983766 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:57.983514    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0ecd97df-2046-4189-a979-df898eb3faaa/kube-rbac-proxy-metric/0.log"
Jul 09 18:31:58.019582 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:58.019432    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0ecd97df-2046-4189-a979-df898eb3faaa/prom-label-proxy/0.log"
Jul 09 18:31:58.047244 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:58.047190    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0ecd97df-2046-4189-a979-df898eb3faaa/init-config-reloader/0.log"
Jul 09 18:31:58.124528 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:58.124429    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-wrs2c_b1dcaaf4-af90-4ca5-9351-33c1e2c612cf/kube-state-metrics/0.log"
Jul 09 18:31:58.153333 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:58.153300    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-wrs2c_b1dcaaf4-af90-4ca5-9351-33c1e2c612cf/kube-rbac-proxy-main/0.log"
Jul 09 18:31:58.182547 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:58.182517    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-wrs2c_b1dcaaf4-af90-4ca5-9351-33c1e2c612cf/kube-rbac-proxy-self/0.log"
Jul 09 18:31:58.215128 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:58.215094    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-578687bf9d-r7prf_cee30c90-1586-4847-a689-7c71a4fb2a5e/metrics-server/0.log"
Jul 09 18:31:58.249442 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:58.249410    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-98rsk_0049e43b-54aa-410c-9971-d89ad2d50563/monitoring-plugin/0.log"
Jul 09 18:31:58.451619 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:58.451586    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zfclm_6feeb09a-7eb5-4c46-a687-202ddba40804/node-exporter/0.log"
Jul 09 18:31:58.476329 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:58.476290    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zfclm_6feeb09a-7eb5-4c46-a687-202ddba40804/kube-rbac-proxy/0.log"
Jul 09 18:31:58.499267 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:58.499240    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zfclm_6feeb09a-7eb5-4c46-a687-202ddba40804/init-textfile/0.log"
Jul 09 18:31:58.530701 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:58.530666    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-4t62b_c61025f3-9d66-42c4-8f78-c9b828e113dc/kube-rbac-proxy-main/0.log"
Jul 09 18:31:58.554977 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:58.554927    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-4t62b_c61025f3-9d66-42c4-8f78-c9b828e113dc/kube-rbac-proxy-self/0.log"
Jul 09 18:31:58.581406 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:58.581336    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-4t62b_c61025f3-9d66-42c4-8f78-c9b828e113dc/openshift-state-metrics/0.log"
Jul 09 18:31:58.620013 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:58.619979    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_86f6b712-a9d3-45ef-bd10-a2f783e76c8f/prometheus/0.log"
Jul 09 18:31:58.647101 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:58.647069    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_86f6b712-a9d3-45ef-bd10-a2f783e76c8f/config-reloader/0.log"
Jul 09 18:31:58.676764 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:58.676727    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_86f6b712-a9d3-45ef-bd10-a2f783e76c8f/thanos-sidecar/0.log"
Jul 09 18:31:58.704599 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:58.704518    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_86f6b712-a9d3-45ef-bd10-a2f783e76c8f/kube-rbac-proxy-web/0.log"
Jul 09 18:31:58.734385 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:58.734346    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_86f6b712-a9d3-45ef-bd10-a2f783e76c8f/kube-rbac-proxy/0.log"
Jul 09 18:31:58.762864 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:58.762834    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_86f6b712-a9d3-45ef-bd10-a2f783e76c8f/kube-rbac-proxy-thanos/0.log"
Jul 09 18:31:58.793237 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:58.793173    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_86f6b712-a9d3-45ef-bd10-a2f783e76c8f/init-config-reloader/0.log"
Jul 09 18:31:58.834749 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:58.834720    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-ddwdk_9f1d9734-a399-4397-94e9-7e40770d1b0a/prometheus-operator/0.log"
Jul 09 18:31:58.867607 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:58.867582    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-ddwdk_9f1d9734-a399-4397-94e9-7e40770d1b0a/kube-rbac-proxy/0.log"
Jul 09 18:31:58.901723 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:58.901691    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-vnbxz_44fd6ea3-1676-4fd0-8b6b-94ecf269733e/prometheus-operator-admission-webhook/0.log"
Jul 09 18:31:58.946674 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:58.946642    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f77f67f4b-gkd94_e1f45c65-dd77-474d-8768-4b93049bffd9/telemeter-client/0.log"
Jul 09 18:31:58.978323 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:58.978255    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f77f67f4b-gkd94_e1f45c65-dd77-474d-8768-4b93049bffd9/reload/0.log"
Jul 09 18:31:59.006378 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:59.006343    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f77f67f4b-gkd94_e1f45c65-dd77-474d-8768-4b93049bffd9/kube-rbac-proxy/0.log"
Jul 09 18:31:59.044944 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:59.044910    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f8c49d449-vp9xh_06ca3e54-f63b-4b0b-9901-76c96741055d/thanos-query/0.log"
Jul 09 18:31:59.077899 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:59.077873    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f8c49d449-vp9xh_06ca3e54-f63b-4b0b-9901-76c96741055d/kube-rbac-proxy-web/0.log"
Jul 09 18:31:59.109292 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:59.109261    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f8c49d449-vp9xh_06ca3e54-f63b-4b0b-9901-76c96741055d/kube-rbac-proxy/0.log"
Jul 09 18:31:59.139159 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:59.139110    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f8c49d449-vp9xh_06ca3e54-f63b-4b0b-9901-76c96741055d/prom-label-proxy/0.log"
Jul 09 18:31:59.171997 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:59.171969    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f8c49d449-vp9xh_06ca3e54-f63b-4b0b-9901-76c96741055d/kube-rbac-proxy-rules/0.log"
Jul 09 18:31:59.200868 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:31:59.200838    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f8c49d449-vp9xh_06ca3e54-f63b-4b0b-9901-76c96741055d/kube-rbac-proxy-metrics/0.log"
Jul 09 18:32:01.813124 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:01.813086    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sbwk5/perf-node-gather-daemonset-wmxcx"]
Jul 09 18:32:01.813657 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:01.813637    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19d069db-5795-480b-984b-b58c1172a626" containerName="copy"
Jul 09 18:32:01.813714 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:01.813661    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="19d069db-5795-480b-984b-b58c1172a626" containerName="copy"
Jul 09 18:32:01.813714 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:01.813687    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19d069db-5795-480b-984b-b58c1172a626" containerName="gather"
Jul 09 18:32:01.813714 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:01.813695    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="19d069db-5795-480b-984b-b58c1172a626" containerName="gather"
Jul 09 18:32:01.813825 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:01.813794    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="19d069db-5795-480b-984b-b58c1172a626" containerName="gather"
Jul 09 18:32:01.813825 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:01.813806    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="19d069db-5795-480b-984b-b58c1172a626" containerName="copy"
Jul 09 18:32:01.817623 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:01.817597    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sbwk5/perf-node-gather-daemonset-wmxcx"
Jul 09 18:32:01.827713 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:01.827684    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sbwk5/perf-node-gather-daemonset-wmxcx"]
Jul 09 18:32:01.877616 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:01.877581    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/13ab8427-bd34-4be6-b076-a2ac1de7c234-lib-modules\") pod \"perf-node-gather-daemonset-wmxcx\" (UID: \"13ab8427-bd34-4be6-b076-a2ac1de7c234\") " pod="openshift-must-gather-sbwk5/perf-node-gather-daemonset-wmxcx"
Jul 09 18:32:01.877781 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:01.877634    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/13ab8427-bd34-4be6-b076-a2ac1de7c234-proc\") pod \"perf-node-gather-daemonset-wmxcx\" (UID: \"13ab8427-bd34-4be6-b076-a2ac1de7c234\") " pod="openshift-must-gather-sbwk5/perf-node-gather-daemonset-wmxcx"
Jul 09 18:32:01.877781 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:01.877693    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13ab8427-bd34-4be6-b076-a2ac1de7c234-sys\") pod \"perf-node-gather-daemonset-wmxcx\" (UID: \"13ab8427-bd34-4be6-b076-a2ac1de7c234\") " pod="openshift-must-gather-sbwk5/perf-node-gather-daemonset-wmxcx"
Jul 09 18:32:01.877781 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:01.877743    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/13ab8427-bd34-4be6-b076-a2ac1de7c234-podres\") pod \"perf-node-gather-daemonset-wmxcx\" (UID: \"13ab8427-bd34-4be6-b076-a2ac1de7c234\") " pod="openshift-must-gather-sbwk5/perf-node-gather-daemonset-wmxcx"
Jul 09 18:32:01.877889 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:01.877813    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw2br\" (UniqueName: \"kubernetes.io/projected/13ab8427-bd34-4be6-b076-a2ac1de7c234-kube-api-access-fw2br\") pod \"perf-node-gather-daemonset-wmxcx\" (UID: \"13ab8427-bd34-4be6-b076-a2ac1de7c234\") " pod="openshift-must-gather-sbwk5/perf-node-gather-daemonset-wmxcx"
Jul 09 18:32:01.978645 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:01.978608    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/13ab8427-bd34-4be6-b076-a2ac1de7c234-lib-modules\") pod \"perf-node-gather-daemonset-wmxcx\" (UID: \"13ab8427-bd34-4be6-b076-a2ac1de7c234\") " pod="openshift-must-gather-sbwk5/perf-node-gather-daemonset-wmxcx"
Jul 09 18:32:01.978850 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:01.978663    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/13ab8427-bd34-4be6-b076-a2ac1de7c234-proc\") pod \"perf-node-gather-daemonset-wmxcx\" (UID: \"13ab8427-bd34-4be6-b076-a2ac1de7c234\") " pod="openshift-must-gather-sbwk5/perf-node-gather-daemonset-wmxcx"
Jul 09 18:32:01.978850 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:01.978709    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13ab8427-bd34-4be6-b076-a2ac1de7c234-sys\") pod \"perf-node-gather-daemonset-wmxcx\" (UID: \"13ab8427-bd34-4be6-b076-a2ac1de7c234\") " pod="openshift-must-gather-sbwk5/perf-node-gather-daemonset-wmxcx"
Jul 09 18:32:01.978850 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:01.978803    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/13ab8427-bd34-4be6-b076-a2ac1de7c234-lib-modules\") pod \"perf-node-gather-daemonset-wmxcx\" (UID: \"13ab8427-bd34-4be6-b076-a2ac1de7c234\") " pod="openshift-must-gather-sbwk5/perf-node-gather-daemonset-wmxcx"
Jul 09 18:32:01.978850 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:01.978829    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/13ab8427-bd34-4be6-b076-a2ac1de7c234-proc\") pod \"perf-node-gather-daemonset-wmxcx\" (UID: \"13ab8427-bd34-4be6-b076-a2ac1de7c234\") " pod="openshift-must-gather-sbwk5/perf-node-gather-daemonset-wmxcx"
Jul 09 18:32:01.979019 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:01.978861    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/13ab8427-bd34-4be6-b076-a2ac1de7c234-podres\") pod \"perf-node-gather-daemonset-wmxcx\" (UID: \"13ab8427-bd34-4be6-b076-a2ac1de7c234\") " pod="openshift-must-gather-sbwk5/perf-node-gather-daemonset-wmxcx"
Jul 09 18:32:01.979019 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:01.978878    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13ab8427-bd34-4be6-b076-a2ac1de7c234-sys\") pod \"perf-node-gather-daemonset-wmxcx\" (UID: \"13ab8427-bd34-4be6-b076-a2ac1de7c234\") " pod="openshift-must-gather-sbwk5/perf-node-gather-daemonset-wmxcx"
Jul 09 18:32:01.979019 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:01.978904    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw2br\" (UniqueName: \"kubernetes.io/projected/13ab8427-bd34-4be6-b076-a2ac1de7c234-kube-api-access-fw2br\") pod \"perf-node-gather-daemonset-wmxcx\" (UID: \"13ab8427-bd34-4be6-b076-a2ac1de7c234\") " pod="openshift-must-gather-sbwk5/perf-node-gather-daemonset-wmxcx"
Jul 09 18:32:01.979118 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:01.979016    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/13ab8427-bd34-4be6-b076-a2ac1de7c234-podres\") pod \"perf-node-gather-daemonset-wmxcx\" (UID: \"13ab8427-bd34-4be6-b076-a2ac1de7c234\") " pod="openshift-must-gather-sbwk5/perf-node-gather-daemonset-wmxcx"
Jul 09 18:32:01.988266 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:01.988206    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw2br\" (UniqueName: \"kubernetes.io/projected/13ab8427-bd34-4be6-b076-a2ac1de7c234-kube-api-access-fw2br\") pod \"perf-node-gather-daemonset-wmxcx\" (UID: \"13ab8427-bd34-4be6-b076-a2ac1de7c234\") " pod="openshift-must-gather-sbwk5/perf-node-gather-daemonset-wmxcx"
Jul 09 18:32:02.132189 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:02.132091    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sbwk5/perf-node-gather-daemonset-wmxcx"
Jul 09 18:32:02.302326 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:02.302289    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sbwk5/perf-node-gather-daemonset-wmxcx"]
Jul 09 18:32:02.307143 ip-10-0-129-181 kubenswrapper[2563]: W0709 18:32:02.307113    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod13ab8427_bd34_4be6_b076_a2ac1de7c234.slice/crio-e29045752388cf7668304c598a556eb99436efc573f8b6562dfae538515d4189 WatchSource:0}: Error finding container e29045752388cf7668304c598a556eb99436efc573f8b6562dfae538515d4189: Status 404 returned error can't find the container with id e29045752388cf7668304c598a556eb99436efc573f8b6562dfae538515d4189
Jul 09 18:32:02.703066 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:02.703025    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sbwk5/perf-node-gather-daemonset-wmxcx" event={"ID":"13ab8427-bd34-4be6-b076-a2ac1de7c234","Type":"ContainerStarted","Data":"c16e444d408c621490ef28bdad38387ef3c40591dba6b1e963cf4a18905f4153"}
Jul 09 18:32:02.703345 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:02.703095    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sbwk5/perf-node-gather-daemonset-wmxcx"
Jul 09 18:32:02.703345 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:02.703122    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sbwk5/perf-node-gather-daemonset-wmxcx" event={"ID":"13ab8427-bd34-4be6-b076-a2ac1de7c234","Type":"ContainerStarted","Data":"e29045752388cf7668304c598a556eb99436efc573f8b6562dfae538515d4189"}
Jul 09 18:32:02.721245 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:02.721174    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sbwk5/perf-node-gather-daemonset-wmxcx" podStartSLOduration=1.7211562759999999 podStartE2EDuration="1.721156276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:32:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:32:02.718661109 +0000 UTC m=+1668.387742067" watchObservedRunningTime="2026-07-09 18:32:02.721156276 +0000 UTC m=+1668.390237230"
Jul 09 18:32:03.384288 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:03.384258    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pnvtm_996150c6-fdce-44a0-963e-d607d34fdc1a/dns/0.log"
Jul 09 18:32:03.410983 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:03.410956    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pnvtm_996150c6-fdce-44a0-963e-d607d34fdc1a/kube-rbac-proxy/0.log"
Jul 09 18:32:03.491002 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:03.490974    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rx2rz_399b7ee4-7292-4fb5-836c-11acc593fa62/dns-node-resolver/0.log"
Jul 09 18:32:04.045340 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:04.045303    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9xkxg_67e30329-228a-455d-886a-626b46e438d4/node-ca/0.log"
Jul 09 18:32:05.347672 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:05.347644    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5nkqz_51ac9338-1662-42ce-8b4b-069afc8fcd55/serve-healthcheck-canary/0.log"
Jul 09 18:32:05.974487 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:05.974459    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q6mvh_d8df3e9b-66dd-45af-9829-41d518142d58/kube-rbac-proxy/0.log"
Jul 09 18:32:06.000878 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:06.000843    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q6mvh_d8df3e9b-66dd-45af-9829-41d518142d58/exporter/0.log"
Jul 09 18:32:06.027643 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:06.027618    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q6mvh_d8df3e9b-66dd-45af-9829-41d518142d58/extractor/0.log"
Jul 09 18:32:08.305573 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:08.305541    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-ftnd9_d8c3dc45-9521-401c-a070-bb3c2114fbbc/manager/0.log"
Jul 09 18:32:08.340748 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:08.340718    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-czmts_8ff484ff-f431-4f3d-9df8-a318387eb715/s3-init/0.log"
Jul 09 18:32:08.359654 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:08.359627    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-czmts_8ff484ff-f431-4f3d-9df8-a318387eb715/download-hf-model/0.log"
Jul 09 18:32:08.393877 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:08.393848    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-45wf6_2795f7e9-d85b-49de-aa43-a20d2caf463c/s3-tls-init-custom/0.log"
Jul 09 18:32:08.412413 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:08.412383    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-45wf6_2795f7e9-d85b-49de-aa43-a20d2caf463c/download-hf-model/0.log"
Jul 09 18:32:08.444042 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:08.444010    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-njsm4_8c6bed38-10d2-4e48-9db3-42fd880f9543/s3-tls-init-serving/0.log"
Jul 09 18:32:08.463078 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:08.463050    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-njsm4_8c6bed38-10d2-4e48-9db3-42fd880f9543/download-hf-model/0.log"
Jul 09 18:32:08.491716 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:08.491676    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-gbdw7_0abbe120-9dc5-4092-87c0-b0d9bec20c5c/seaweedfs/0.log"
Jul 09 18:32:08.726670 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:08.726643    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sbwk5/perf-node-gather-daemonset-wmxcx"
Jul 09 18:32:14.579608 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:14.579570    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-7xtpt_30ba50d7-581d-4b1b-9e63-7e1fe0ea9137/kube-multus/0.log"
Jul 09 18:32:14.608161 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:14.608134    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lljj8_7f2265cf-194a-43a6-b5d4-4fe400c2f429/kube-multus-additional-cni-plugins/0.log"
Jul 09 18:32:14.633106 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:14.633075    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lljj8_7f2265cf-194a-43a6-b5d4-4fe400c2f429/egress-router-binary-copy/0.log"
Jul 09 18:32:14.661306 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:14.661277    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lljj8_7f2265cf-194a-43a6-b5d4-4fe400c2f429/cni-plugins/0.log"
Jul 09 18:32:14.690909 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:14.690839    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lljj8_7f2265cf-194a-43a6-b5d4-4fe400c2f429/bond-cni-plugin/0.log"
Jul 09 18:32:14.716737 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:14.716712    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lljj8_7f2265cf-194a-43a6-b5d4-4fe400c2f429/routeoverride-cni/0.log"
Jul 09 18:32:14.742123 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:14.742096    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lljj8_7f2265cf-194a-43a6-b5d4-4fe400c2f429/whereabouts-cni-bincopy/0.log"
Jul 09 18:32:14.768091 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:14.768068    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lljj8_7f2265cf-194a-43a6-b5d4-4fe400c2f429/whereabouts-cni/0.log"
Jul 09 18:32:15.376589 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:15.376500    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wqgvh_d5de2134-56a4-4318-a22f-90937385dc5b/network-metrics-daemon/0.log"
Jul 09 18:32:15.403963 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:15.403935    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wqgvh_d5de2134-56a4-4318-a22f-90937385dc5b/kube-rbac-proxy/0.log"
Jul 09 18:32:16.588273 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:16.588248    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbftk_5d463fb2-45f9-4554-9525-637e1b84351a/ovn-controller/0.log"
Jul 09 18:32:16.609168 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:16.609140    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbftk_5d463fb2-45f9-4554-9525-637e1b84351a/ovn-acl-logging/0.log"
Jul 09 18:32:16.609168 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:16.609161    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbftk_5d463fb2-45f9-4554-9525-637e1b84351a/ovn-acl-logging/1.log"
Jul 09 18:32:16.627693 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:16.627667    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbftk_5d463fb2-45f9-4554-9525-637e1b84351a/kube-rbac-proxy-node/0.log"
Jul 09 18:32:16.650830 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:16.650796    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbftk_5d463fb2-45f9-4554-9525-637e1b84351a/kube-rbac-proxy-ovn-metrics/0.log"
Jul 09 18:32:16.674164 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:16.674120    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbftk_5d463fb2-45f9-4554-9525-637e1b84351a/northd/0.log"
Jul 09 18:32:16.697897 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:16.697864    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbftk_5d463fb2-45f9-4554-9525-637e1b84351a/nbdb/0.log"
Jul 09 18:32:16.721807 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:16.721782    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbftk_5d463fb2-45f9-4554-9525-637e1b84351a/sbdb/0.log"
Jul 09 18:32:16.766500 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:16.766465    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbftk_5d463fb2-45f9-4554-9525-637e1b84351a/ovnkube-controller/0.log"
Jul 09 18:32:18.277310 ip-10-0-129-181 kubenswrapper[2563]: I0709 18:32:18.277281    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-sf462_52ba7983-15e4-437e-a86f-36f0048b0614/network-check-target-container/0.log"