Jul 07 11:38:04.564229 ip-10-0-138-183 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 07 11:38:04.564238 ip-10-0-138-183 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 07 11:38:04.564245 ip-10-0-138-183 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 07 11:38:04.564451 ip-10-0-138-183 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 07 11:38:14.609797 ip-10-0-138-183 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 07 11:38:14.609815 ip-10-0-138-183 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot e875c89973de400f952b70c128a43e31 --
Jul 07 11:40:09.526306 ip-10-0-138-183 systemd[1]: Starting Kubernetes Kubelet...
Jul 07 11:40:11.713192 ip-10-0-138-183 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 11:40:11.713192 ip-10-0-138-183 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 07 11:40:11.713192 ip-10-0-138-183 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 11:40:11.713192 ip-10-0-138-183 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 07 11:40:11.713192 ip-10-0-138-183 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 11:40:11.713192 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.655729    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 07 11:40:11.713192 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662316    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 11:40:11.713192 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662328    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 11:40:11.713192 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662332    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 11:40:11.713192 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662337    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 11:40:11.713192 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662341    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 11:40:11.713192 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662344    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 11:40:11.713192 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662347    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 11:40:11.713192 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662350    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 11:40:11.716746 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662353    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 11:40:11.716746 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662356    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 11:40:11.716746 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662359    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 11:40:11.716746 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662362    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 11:40:11.716746 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662368    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 11:40:11.716746 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662371    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 11:40:11.716746 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662374    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 11:40:11.716746 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662377    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 11:40:11.716746 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662380    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 11:40:11.716746 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662384    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 11:40:11.716746 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662389    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 11:40:11.716746 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662392    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 11:40:11.716746 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662396    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 11:40:11.716746 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662399    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 11:40:11.716746 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662402    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 11:40:11.716746 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662408    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 11:40:11.716746 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662415    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 11:40:11.716746 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662419    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 11:40:11.716746 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662422    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 11:40:11.717621 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662425    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 11:40:11.717621 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662428    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 11:40:11.717621 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662432    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 11:40:11.717621 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662434    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 11:40:11.717621 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662437    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 11:40:11.717621 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662440    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 11:40:11.717621 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662443    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 11:40:11.717621 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662446    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 11:40:11.717621 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662449    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 11:40:11.717621 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662452    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 11:40:11.717621 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662458    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 11:40:11.717621 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662460    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 11:40:11.717621 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662463    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 11:40:11.717621 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662466    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 11:40:11.717621 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662469    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 11:40:11.717621 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662471    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 11:40:11.717621 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662474    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 11:40:11.717621 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662477    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 11:40:11.717621 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662480    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 11:40:11.717621 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662486    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 11:40:11.837680 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662490    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 11:40:11.837680 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662493    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 11:40:11.837680 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662499    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 11:40:11.837680 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662502    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 11:40:11.837680 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662506    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 11:40:11.837680 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662508    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 11:40:11.837680 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662511    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 11:40:11.837680 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662515    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 11:40:11.837680 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662518    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 11:40:11.837680 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662521    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 11:40:11.837680 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662524    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 11:40:11.837680 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662527    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 11:40:11.837680 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662530    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 11:40:11.837680 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662533    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 11:40:11.837680 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662539    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 11:40:11.837680 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662542    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 11:40:11.837680 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662545    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 11:40:11.837680 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662548    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 11:40:11.837680 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662550    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 11:40:11.837680 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662553    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 11:40:11.839026 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662556    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 11:40:11.839026 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662559    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 11:40:11.839026 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662561    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 11:40:11.839026 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662564    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 11:40:11.839026 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662568    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 11:40:11.839026 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662571    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 11:40:11.839026 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662577    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 11:40:11.839026 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662579    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 11:40:11.839026 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662583    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 11:40:11.839026 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662586    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 11:40:11.839026 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662588    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 11:40:11.839026 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662591    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 11:40:11.839026 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662594    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 11:40:11.839026 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662596    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 11:40:11.839026 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662599    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 11:40:11.839026 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662602    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 11:40:11.839026 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662605    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 11:40:11.839026 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662608    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 11:40:11.839026 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662614    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 11:40:11.839026 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662617    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 11:40:11.839969 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662620    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 11:40:11.839969 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662623    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 11:40:11.839969 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662626    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 11:40:11.839969 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662629    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 11:40:11.839969 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662632    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 11:40:11.839969 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662635    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 11:40:11.839969 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662638    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 11:40:11.839969 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662641    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 11:40:11.839969 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662644    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 11:40:11.839969 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662647    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 11:40:11.839969 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662650    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 11:40:11.839969 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.662655    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 11:40:11.839969 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663828    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 11:40:11.839969 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663835    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 11:40:11.839969 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663839    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 11:40:11.839969 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663845    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 11:40:11.839969 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663847    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 11:40:11.839969 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663850    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 11:40:11.839969 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663853    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 11:40:11.842239 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663858    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 11:40:11.842239 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663862    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 11:40:11.842239 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663866    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 11:40:11.842239 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663869    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 11:40:11.842239 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663873    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 11:40:11.842239 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663876    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 11:40:11.842239 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663879    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 11:40:11.842239 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663882    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 11:40:11.842239 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663887    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 11:40:11.842239 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663891    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 11:40:11.842239 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663894    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 11:40:11.842239 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663898    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 11:40:11.842239 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663902    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 11:40:11.842239 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663906    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 11:40:11.842239 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663909    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 11:40:11.842239 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663912    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 11:40:11.842239 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663915    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 11:40:11.842239 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663921    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 11:40:11.842239 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663925    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 11:40:11.843196 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663931    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 11:40:11.843196 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663934    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 11:40:11.843196 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663937    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 11:40:11.843196 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663941    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 11:40:11.843196 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663944    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 11:40:11.843196 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663947    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 11:40:11.843196 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663950    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 11:40:11.843196 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663952    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 11:40:11.843196 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663956    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 11:40:11.843196 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663958    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 11:40:11.843196 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663961    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 11:40:11.843196 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663964    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 11:40:11.843196 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663969    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 11:40:11.843196 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663972    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 11:40:11.843196 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663975    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 11:40:11.843196 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663978    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 11:40:11.843196 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663981    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 11:40:11.843196 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663984    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 11:40:11.843196 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663986    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 11:40:11.843196 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663989    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 11:40:11.844529 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663992    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 11:40:11.844529 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663995    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 11:40:11.844529 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.663997    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 11:40:11.844529 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664000    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 11:40:11.844529 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664003    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 11:40:11.844529 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664008    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 11:40:11.844529 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664011    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 11:40:11.844529 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664014    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 11:40:11.844529 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664017    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 11:40:11.844529 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664020    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 11:40:11.844529 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664023    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 11:40:11.844529 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664025    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 11:40:11.844529 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664029    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 11:40:11.844529 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664032    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 11:40:11.844529 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664038    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 11:40:11.844529 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664041    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 11:40:11.844529 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664045    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 11:40:11.844529 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664050    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 11:40:11.844529 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664053    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 11:40:11.845469 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664056    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 11:40:11.845469 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664059    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 11:40:11.845469 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664062    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 11:40:11.845469 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664065    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 11:40:11.845469 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664067    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 11:40:11.845469 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664070    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 11:40:11.845469 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664073    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 11:40:11.845469 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664076    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 11:40:11.845469 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664079    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 11:40:11.845469 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664097    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 11:40:11.845469 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664103    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 11:40:11.845469 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664106    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 11:40:11.845469 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664109    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 11:40:11.845469 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664112    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 11:40:11.845469 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664115    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 11:40:11.845469 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664118    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 11:40:11.845469 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664121    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 11:40:11.845469 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664123    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 11:40:11.845469 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664126    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 11:40:11.845469 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664129    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 11:40:11.846980 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664132    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 11:40:11.846980 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664135    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 11:40:11.846980 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664138    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 11:40:11.846980 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664144    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 11:40:11.846980 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664147    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 11:40:11.846980 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664150    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 11:40:11.846980 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664154    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 11:40:11.846980 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664156    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 11:40:11.846980 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664159    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 11:40:11.846980 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664162    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 11:40:11.846980 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664165    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 11:40:11.846980 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664167    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 11:40:11.846980 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664170    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 11:40:11.846980 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664192    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 11:40:11.846980 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664828    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 11:40:11.846980 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664833    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 11:40:11.846980 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664836    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 11:40:11.846980 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664840    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 11:40:11.846980 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664843    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 11:40:11.846980 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664846    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 11:40:11.849250 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664852    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 11:40:11.849250 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664855    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 11:40:11.849250 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664858    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 11:40:11.849250 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664861    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 11:40:11.849250 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664863    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 11:40:11.849250 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664866    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 11:40:11.849250 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664869    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 11:40:11.849250 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664872    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 11:40:11.849250 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664875    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 11:40:11.849250 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664878    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 11:40:11.849250 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664881    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 11:40:11.849250 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664884    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 11:40:11.849250 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664889    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 11:40:11.849250 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664892    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 11:40:11.849250 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664895    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 11:40:11.849250 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664900    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 11:40:11.849250 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664903    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 11:40:11.849250 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664906    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 11:40:11.849250 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664910    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 11:40:11.849250 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664913    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 11:40:11.849830 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664916    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 11:40:11.849830 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664919    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 11:40:11.849830 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664922    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 11:40:11.849830 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664925    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 11:40:11.849830 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664928    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 11:40:11.849830 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664933    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 11:40:11.849830 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664936    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 11:40:11.849830 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664941    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 11:40:11.849830 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664944    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 11:40:11.849830 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664947    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 11:40:11.849830 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664949    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 11:40:11.849830 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664952    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 11:40:11.849830 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664955    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 11:40:11.849830 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664958    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 11:40:11.849830 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664961    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 11:40:11.849830 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664963    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 11:40:11.849830 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664966    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 11:40:11.849830 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664969    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 11:40:11.849830 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664975    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 11:40:11.850350 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664978    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 11:40:11.850350 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664981    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 11:40:11.850350 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664984    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 11:40:11.850350 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664987    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 11:40:11.850350 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664990    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 11:40:11.850350 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664992    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 11:40:11.850350 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664995    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 11:40:11.850350 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.664998    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 11:40:11.850350 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665004    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 11:40:11.850350 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665007    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 11:40:11.850350 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665010    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 11:40:11.850350 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665015    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 11:40:11.850350 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665018    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 11:40:11.850350 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665021    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 11:40:11.850350 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665024    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 11:40:11.850350 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665027    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 11:40:11.850350 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665030    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 11:40:11.850350 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665032    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 11:40:11.850350 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665035    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 11:40:11.850350 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665038    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 11:40:11.850859 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665042    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 11:40:11.850859 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665045    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 11:40:11.850859 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665048    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 11:40:11.850859 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665053    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 11:40:11.850859 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665056    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 11:40:11.850859 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665059    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 11:40:11.850859 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665062    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 11:40:11.850859 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665065    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 11:40:11.850859 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665067    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 11:40:11.850859 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665070    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 11:40:11.850859 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665074    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 11:40:11.850859 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665078    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 11:40:11.850859 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665100    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 11:40:11.850859 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665104    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 11:40:11.850859 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665109    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 11:40:11.850859 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665114    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 11:40:11.850859 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665117    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 11:40:11.850859 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665120    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 11:40:11.850859 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665123    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 11:40:11.850859 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665126    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 11:40:11.851386 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665131    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 11:40:11.851386 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665133    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 11:40:11.851386 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665136    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 11:40:11.851386 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665139    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 11:40:11.851386 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665142    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 11:40:11.851386 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665146    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 11:40:11.851386 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665149    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 11:40:11.851386 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665154    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 11:40:11.851386 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665157    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 11:40:11.851386 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665161    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 11:40:11.851386 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665165    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 11:40:11.851386 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665168    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 11:40:11.851386 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665173    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 11:40:11.851386 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.665176    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 11:40:11.851386 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665271    2581 flags.go:64] FLAG: --address="0.0.0.0"
Jul 07 11:40:11.851386 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665278    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 07 11:40:11.851386 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665285    2581 flags.go:64] FLAG: --anonymous-auth="true"
Jul 07 11:40:11.851386 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665292    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 07 11:40:11.851386 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665296    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 07 11:40:11.851386 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665300    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 07 11:40:11.851386 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665304    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 07 11:40:11.851914 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665312    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 07 11:40:11.851914 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665316    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 07 11:40:11.851914 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665319    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 07 11:40:11.851914 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665323    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 07 11:40:11.851914 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665326    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 07 11:40:11.851914 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665330    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 07 11:40:11.851914 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665333    2581 flags.go:64] FLAG: --cgroup-root=""
Jul 07 11:40:11.851914 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665336    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 07 11:40:11.851914 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665340    2581 flags.go:64] FLAG: --client-ca-file=""
Jul 07 11:40:11.851914 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665345    2581 flags.go:64] FLAG: --cloud-provider="external"
Jul 07 11:40:11.851914 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665349    2581 flags.go:64] FLAG: --cluster-dns="[]"
Jul 07 11:40:11.851914 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665355    2581 flags.go:64] FLAG: --cluster-domain=""
Jul 07 11:40:11.851914 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665358    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 07 11:40:11.851914 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665362    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 07 11:40:11.851914 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665365    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 07 11:40:11.851914 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665369    2581 flags.go:64] FLAG: --container-log-max-files="5"
Jul 07 11:40:11.851914 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665373    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 07 11:40:11.851914 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665380    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 07 11:40:11.851914 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665383    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 07 11:40:11.851914 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665387    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 07 11:40:11.851914 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665390    2581 flags.go:64] FLAG: --contention-profiling="false"
Jul 07 11:40:11.851914 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665394    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 07 11:40:11.851914 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665398    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 07 11:40:11.851914 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665402    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 07 11:40:11.852542 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665406    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 07 11:40:11.852542 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665410    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 07 11:40:11.852542 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665416    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 07 11:40:11.852542 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665419    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 07 11:40:11.852542 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665422    2581 flags.go:64] FLAG: --enable-load-reader="false"
Jul 07 11:40:11.852542 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665425    2581 flags.go:64] FLAG: --enable-server="true"
Jul 07 11:40:11.852542 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665428    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 07 11:40:11.852542 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665433    2581 flags.go:64] FLAG: --event-burst="100"
Jul 07 11:40:11.852542 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665436    2581 flags.go:64] FLAG: --event-qps="50"
Jul 07 11:40:11.852542 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665440    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 07 11:40:11.852542 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665445    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 07 11:40:11.852542 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665449    2581 flags.go:64] FLAG: --eviction-hard=""
Jul 07 11:40:11.852542 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665453    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 07 11:40:11.852542 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665456    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 07 11:40:11.852542 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665459    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 07 11:40:11.852542 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665463    2581 flags.go:64] FLAG: --eviction-soft=""
Jul 07 11:40:11.852542 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665466    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 07 11:40:11.852542 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665469    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 07 11:40:11.852542 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665474    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 07 11:40:11.852542 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665479    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 07 11:40:11.852542 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665483    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 07 11:40:11.852542 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665486    2581 flags.go:64] FLAG: --fail-swap-on="true"
Jul 07 11:40:11.852542 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665489    2581 flags.go:64] FLAG: --feature-gates=""
Jul 07 11:40:11.852542 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665493    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 07 11:40:11.852542 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665496    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 07 11:40:11.853248 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665500    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 07 11:40:11.853248 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665504    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 07 11:40:11.853248 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665509    2581 flags.go:64] FLAG: --healthz-port="10248"
Jul 07 11:40:11.853248 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665513    2581 flags.go:64] FLAG: --help="false"
Jul 07 11:40:11.853248 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665516    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-138-183.ec2.internal"
Jul 07 11:40:11.853248 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665519    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 07 11:40:11.853248 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665524    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 07 11:40:11.853248 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665527    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 07 11:40:11.853248 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665531    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 07 11:40:11.853248 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665534    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 07 11:40:11.853248 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665540    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 07 11:40:11.853248 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665544    2581 flags.go:64] FLAG: --image-service-endpoint=""
Jul 07 11:40:11.853248 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665547    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 07 11:40:11.853248 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665550    2581 flags.go:64] FLAG: --kube-api-burst="100"
Jul 07 11:40:11.853248 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665553    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 07 11:40:11.853248 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665557    2581 flags.go:64] FLAG: --kube-api-qps="50"
Jul 07 11:40:11.853248 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665560    2581 flags.go:64] FLAG: --kube-reserved=""
Jul 07 11:40:11.853248 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665563    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 07 11:40:11.853248 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665566    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 07 11:40:11.853248 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665572    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 07 11:40:11.853248 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665575    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 07 11:40:11.853248 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665579    2581 flags.go:64] FLAG: --lock-file=""
Jul 07 11:40:11.853248 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665582    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 07 11:40:11.853248 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665585    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 07 11:40:11.853859 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665588    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 07 11:40:11.853859 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665594    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 07 11:40:11.853859 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665599    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 07 11:40:11.853859 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665604    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 07 11:40:11.853859 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665608    2581 flags.go:64] FLAG: --logging-format="text"
Jul 07 11:40:11.853859 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665611    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 07 11:40:11.853859 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665615    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 07 11:40:11.853859 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665618    2581 flags.go:64] FLAG: --manifest-url=""
Jul 07 11:40:11.853859 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665621    2581 flags.go:64] FLAG: --manifest-url-header=""
Jul 07 11:40:11.853859 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665626    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 07 11:40:11.853859 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665630    2581 flags.go:64] FLAG: --max-open-files="1000000"
Jul 07 11:40:11.853859 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665634    2581 flags.go:64] FLAG: --max-pods="110"
Jul 07 11:40:11.853859 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665640    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 07 11:40:11.853859 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665643    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 07 11:40:11.853859 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665648    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 07 11:40:11.853859 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665651    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 07 11:40:11.853859 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665654    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 07 11:40:11.853859 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665658    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 07 11:40:11.853859 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665661    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 07 11:40:11.853859 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665671    2581 flags.go:64] FLAG: --node-status-max-images="50"
Jul 07 11:40:11.853859 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665675    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 07 11:40:11.853859 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665678    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 07 11:40:11.853859 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665682    2581 flags.go:64] FLAG: --pod-cidr=""
Jul 07 11:40:11.854492 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665685    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 07 11:40:11.854492 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665690    2581 flags.go:64] FLAG: --pod-manifest-path=""
Jul 07 11:40:11.854492 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665694    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 07 11:40:11.854492 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665697    2581 flags.go:64] FLAG: --pods-per-core="0"
Jul 07 11:40:11.854492 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665703    2581 flags.go:64] FLAG: --port="10250"
Jul 07 11:40:11.854492 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665706    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 07 11:40:11.854492 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665709    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0cbcdfa6bf1dee86e"
Jul 07 11:40:11.854492 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665713    2581 flags.go:64] FLAG: --qos-reserved=""
Jul 07 11:40:11.854492 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665716    2581 flags.go:64] FLAG: --read-only-port="10255"
Jul 07 11:40:11.854492 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665720    2581 flags.go:64] FLAG: --register-node="true"
Jul 07 11:40:11.854492 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665723    2581 flags.go:64] FLAG: --register-with-taints=""
Jul 07 11:40:11.854492 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665727    2581 flags.go:64] FLAG: --registry-burst="10"
Jul 07 11:40:11.854492 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665732    2581 flags.go:64] FLAG: --registry-qps="5"
Jul 07 11:40:11.854492 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665737    2581 flags.go:64] FLAG: --reserved-cpus=""
Jul 07 11:40:11.854492 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665740    2581 flags.go:64] FLAG: --reserved-memory=""
Jul 07 11:40:11.854492 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665744    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 07 11:40:11.854492 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665748    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 07 11:40:11.854492 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665751    2581 flags.go:64] FLAG: --rotate-certificates="false"
Jul 07 11:40:11.854492 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665754    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 07 11:40:11.854492 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665757    2581 flags.go:64] FLAG: --runonce="false"
Jul 07 11:40:11.854492 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665761    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 07 11:40:11.854492 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665767    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 07 11:40:11.854492 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665770    2581 flags.go:64] FLAG: --seccomp-default="false"
Jul 07 11:40:11.854492 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665774    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 07 11:40:11.854492 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665778    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 07 11:40:11.855153 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665782    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 07 11:40:11.855153 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665785    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 07 11:40:11.855153 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665789    2581 flags.go:64] FLAG: --storage-driver-password="root"
Jul 07 11:40:11.855153 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665792    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 07 11:40:11.855153 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665795    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 07 11:40:11.855153 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665801    2581 flags.go:64] FLAG: --storage-driver-user="root"
Jul 07 11:40:11.855153 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665804    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 07 11:40:11.855153 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665807    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 07 11:40:11.855153 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665811    2581 flags.go:64] FLAG: --system-cgroups=""
Jul 07 11:40:11.855153 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665814    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 07 11:40:11.855153 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665820    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 07 11:40:11.855153 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665823    2581 flags.go:64] FLAG: --tls-cert-file=""
Jul 07 11:40:11.855153 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665829    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 07 11:40:11.855153 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665833    2581 flags.go:64] FLAG: --tls-min-version=""
Jul 07 11:40:11.855153 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665836    2581 flags.go:64] FLAG: --tls-private-key-file=""
Jul 07 11:40:11.855153 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665839    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 07 11:40:11.855153 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665842    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 07 11:40:11.855153 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665845    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 07 11:40:11.855153 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665849    2581 flags.go:64] FLAG: --v="2"
Jul 07 11:40:11.855153 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665855    2581 flags.go:64] FLAG: --version="false"
Jul 07 11:40:11.855153 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665861    2581 flags.go:64] FLAG: --vmodule=""
Jul 07 11:40:11.855153 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665865    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 07 11:40:11.855153 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.665869    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 07 11:40:11.855153 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666077    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 11:40:11.855153 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666097    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 11:40:11.855782 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666102    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 11:40:11.855782 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666106    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 11:40:11.855782 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666109    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 11:40:11.855782 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666115    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 11:40:11.855782 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666119    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 11:40:11.855782 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666125    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 11:40:11.855782 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666129    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 11:40:11.855782 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666133    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 11:40:11.855782 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666139    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 11:40:11.855782 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666143    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 11:40:11.855782 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666146    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 11:40:11.855782 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666149    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 11:40:11.855782 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666153    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 11:40:11.855782 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666156    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 11:40:11.855782 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666159    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 11:40:11.855782 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666162    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 11:40:11.855782 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666166    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 11:40:11.855782 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666169    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 11:40:11.855782 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666172    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 11:40:11.856290 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666175    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 11:40:11.856290 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666180    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 11:40:11.856290 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666183    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 11:40:11.856290 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666186    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 11:40:11.856290 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666189    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 11:40:11.856290 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666192    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 11:40:11.856290 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666195    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 11:40:11.856290 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666199    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 11:40:11.856290 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666202    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 11:40:11.856290 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666205    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 11:40:11.856290 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666208    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 11:40:11.856290 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666211    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 11:40:11.856290 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666214    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 11:40:11.856290 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666217    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 11:40:11.856290 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666222    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 11:40:11.856290 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666225    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 11:40:11.856290 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666228    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 11:40:11.856290 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666232    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 11:40:11.856290 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666237    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 11:40:11.856290 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666239    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 11:40:11.856807 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666242    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 11:40:11.856807 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666245    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 11:40:11.856807 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666248    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 11:40:11.856807 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666251    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 11:40:11.856807 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666254    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 11:40:11.856807 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666257    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 11:40:11.856807 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666262    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 11:40:11.856807 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666265    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 11:40:11.856807 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666268    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 11:40:11.856807 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666271    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 11:40:11.856807 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666273    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 11:40:11.856807 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666276    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 11:40:11.856807 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666279    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 11:40:11.856807 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666282    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 11:40:11.856807 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666285    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 11:40:11.856807 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666288    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 11:40:11.856807 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666290    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 11:40:11.856807 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666293    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 11:40:11.856807 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666296    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 11:40:11.856807 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666304    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 11:40:11.857341 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666307    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 11:40:11.857341 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666310    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 11:40:11.857341 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666313    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 11:40:11.857341 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666315    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 11:40:11.857341 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666318    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 11:40:11.857341 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666321    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 11:40:11.857341 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666324    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 11:40:11.857341 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666327    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 11:40:11.857341 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666329    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 11:40:11.857341 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666332    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 11:40:11.857341 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666337    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 11:40:11.857341 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666343    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 11:40:11.857341 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666345    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 11:40:11.857341 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666348    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 11:40:11.857341 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666351    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 11:40:11.857341 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666356    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 11:40:11.857341 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666360    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 11:40:11.857341 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666363    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 11:40:11.857341 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666367    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 11:40:11.857822 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666370    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 11:40:11.857822 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666373    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 11:40:11.857822 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666376    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 11:40:11.857822 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666379    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 11:40:11.857822 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666382    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 11:40:11.857822 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666387    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 11:40:11.857822 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666390    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 11:40:11.857822 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666393    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 11:40:11.857822 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666396    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 11:40:11.857822 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666399    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 11:40:11.857822 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666401    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 11:40:11.857822 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666404    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 11:40:11.857822 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666409    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 11:40:11.857822 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666412    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 11:40:11.857822 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666414    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 11:40:11.857822 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666417    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 11:40:11.857822 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666420    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 11:40:11.857822 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666423    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 11:40:11.857822 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.666428    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 11:40:11.858338 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.667070    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 11:40:11.858338 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.675497    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 07 11:40:11.858338 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.675513    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 07 11:40:11.858338 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.675535    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 07 11:40:11.858338 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.675542    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 07 11:40:11.858338 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675680    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 11:40:11.858338 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675687    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 11:40:11.858338 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675691    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 11:40:11.858338 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675695    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 11:40:11.858338 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675699    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 11:40:11.858338 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675703    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 11:40:11.858338 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675706    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 11:40:11.858338 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675709    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 11:40:11.858338 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675715    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 11:40:11.858757 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675718    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 11:40:11.858757 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675722    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 11:40:11.858757 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675727    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 11:40:11.858757 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675732    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 11:40:11.858757 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675741    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 11:40:11.858757 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675745    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 11:40:11.858757 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675748    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 11:40:11.858757 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675751    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 11:40:11.858757 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675754    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 11:40:11.858757 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675758    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 11:40:11.858757 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675761    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 11:40:11.858757 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675766    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 11:40:11.858757 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675769    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 11:40:11.858757 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675772    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 11:40:11.858757 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675776    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 11:40:11.858757 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675780    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 11:40:11.858757 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675784    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 11:40:11.858757 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675788    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 11:40:11.858757 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675790    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 11:40:11.859264 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675793    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 11:40:11.859264 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675813    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 11:40:11.859264 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675817    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 11:40:11.859264 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675821    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 11:40:11.859264 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675827    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 11:40:11.859264 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675831    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 11:40:11.859264 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675834    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 11:40:11.859264 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675837    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 11:40:11.859264 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675840    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 11:40:11.859264 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675843    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 11:40:11.859264 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675846    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 11:40:11.859264 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675849    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 11:40:11.859264 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675852    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 11:40:11.859264 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675855    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 11:40:11.859264 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675858    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 11:40:11.859264 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675861    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 11:40:11.859264 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675864    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 11:40:11.859264 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675870    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 11:40:11.859264 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675873    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 11:40:11.859749 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675876    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 11:40:11.859749 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675879    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 11:40:11.859749 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675882    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 11:40:11.859749 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675885    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 11:40:11.859749 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675888    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 11:40:11.859749 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675891    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 11:40:11.859749 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675894    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 11:40:11.859749 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675898    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 11:40:11.859749 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675903    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 11:40:11.859749 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675908    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 11:40:11.859749 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675921    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 11:40:11.859749 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675924    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 11:40:11.859749 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675927    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 11:40:11.859749 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675945    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 11:40:11.859749 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675956    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 11:40:11.859749 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675966    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 11:40:11.859749 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675970    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 11:40:11.859749 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675973    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 11:40:11.859749 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675976    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 11:40:11.859749 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675979    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 11:40:11.860279 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675982    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 11:40:11.860279 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675985    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 11:40:11.860279 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675990    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 11:40:11.860279 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675993    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 11:40:11.860279 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675996    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 11:40:11.860279 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.675999    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 11:40:11.860279 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676002    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 11:40:11.860279 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676005    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 11:40:11.860279 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676008    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 11:40:11.860279 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676011    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 11:40:11.860279 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676014    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 11:40:11.860279 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676018    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 11:40:11.860279 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676022    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 11:40:11.860279 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676027    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 11:40:11.860279 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676032    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 11:40:11.860279 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676044    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 11:40:11.860279 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676048    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 11:40:11.860279 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676051    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 11:40:11.860279 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676054    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 11:40:11.860279 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676056    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 11:40:11.860788 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676060    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 11:40:11.860788 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676062    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 11:40:11.860788 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676065    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 11:40:11.860788 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676068    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 11:40:11.860788 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676071    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 11:40:11.860788 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676073    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 11:40:11.860788 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676079    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 11:40:11.860788 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676097    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 11:40:11.860788 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676111    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 11:40:11.860788 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676114    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 11:40:11.860788 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676117    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 11:40:11.860788 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676120    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 11:40:11.860788 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.676125    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 11:40:11.860788 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676434    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 11:40:11.861177 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676442    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 11:40:11.861177 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676447    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 11:40:11.861177 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676450    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 11:40:11.861177 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676454    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 11:40:11.861177 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676457    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 11:40:11.861177 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676461    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 11:40:11.861177 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676465    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 11:40:11.861177 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676468    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 11:40:11.861177 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676472    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 11:40:11.861177 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676478    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 11:40:11.861177 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676481    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 11:40:11.861177 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676484    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 11:40:11.861177 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676487    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 11:40:11.861177 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676490    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 11:40:11.861177 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676492    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 11:40:11.861177 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676495    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 11:40:11.861177 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676498    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 11:40:11.861177 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676501    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 11:40:11.861177 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676503    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 11:40:11.861656 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676506    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 11:40:11.861656 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676509    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 11:40:11.861656 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676512    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 11:40:11.861656 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676517    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 11:40:11.861656 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676520    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 11:40:11.861656 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676525    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 11:40:11.861656 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676529    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 11:40:11.861656 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676532    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 11:40:11.861656 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676541    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 11:40:11.861656 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676545    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 11:40:11.861656 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676548    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 11:40:11.861656 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676552    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 11:40:11.861656 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676557    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 11:40:11.861656 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676562    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 11:40:11.861656 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676566    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 11:40:11.861656 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676569    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 11:40:11.861656 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676575    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 11:40:11.861656 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676578    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 11:40:11.861656 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676581    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 11:40:11.861656 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676584    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 11:40:11.862194 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676587    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 11:40:11.862194 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676590    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 11:40:11.862194 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676593    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 11:40:11.862194 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676595    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 11:40:11.862194 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676598    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 11:40:11.862194 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676601    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 11:40:11.862194 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676604    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 11:40:11.862194 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676608    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 11:40:11.862194 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676613    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 11:40:11.862194 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676616    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 11:40:11.862194 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676619    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 11:40:11.862194 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676622    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 11:40:11.862194 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676625    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 11:40:11.862194 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676628    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 11:40:11.862194 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676630    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 11:40:11.862194 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676633    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 11:40:11.862194 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676636    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 11:40:11.862194 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676640    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 11:40:11.862194 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676647    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 11:40:11.862684 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676650    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 11:40:11.862684 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676663    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 11:40:11.862684 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676674    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 11:40:11.862684 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676678    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 11:40:11.862684 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676681    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 11:40:11.862684 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676684    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 11:40:11.862684 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676687    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 11:40:11.862684 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676690    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 11:40:11.862684 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676693    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 11:40:11.862684 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676696    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 11:40:11.862684 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676699    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 11:40:11.862684 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676702    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 11:40:11.862684 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676705    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 11:40:11.862684 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676707    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 11:40:11.862684 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676713    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 11:40:11.862684 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676716    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 11:40:11.862684 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676719    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 11:40:11.862684 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676723    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 11:40:11.862684 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676728    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 11:40:11.862684 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676733    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 11:40:11.863228 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676742    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 11:40:11.863228 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676747    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 11:40:11.863228 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676749    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 11:40:11.863228 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676752    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 11:40:11.863228 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676755    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 11:40:11.863228 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676758    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 11:40:11.863228 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676761    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 11:40:11.863228 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676766    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 11:40:11.863228 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676769    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 11:40:11.863228 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676772    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 11:40:11.863228 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676776    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 11:40:11.863228 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676782    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 11:40:11.863228 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676785    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 11:40:11.863228 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676788    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 11:40:11.863228 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676791    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 11:40:11.863228 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676800    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 11:40:11.863228 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676803    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 11:40:11.863228 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676806    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 11:40:11.863228 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676809    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 11:40:11.863228 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:11.676815    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 11:40:11.863745 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.676821    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 11:40:11.863745 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.677047    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 07 11:40:11.916123 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.916105    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 07 11:40:11.918944 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.918931    2581 server.go:1019] "Starting client certificate rotation"
Jul 07 11:40:11.919745 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.919732    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 11:40:11.919782 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.919772    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 11:40:11.928647 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.928622    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 11:40:11.929986 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.929958    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 11:40:11.941402 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.941383    2581 log.go:25] "Validated CRI v1 runtime API"
Jul 07 11:40:11.943728 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.943711    2581 log.go:25] "Validated CRI v1 image API"
Jul 07 11:40:11.944581 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.944561    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 07 11:40:11.946823 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.946803    2581 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 e8b55c5a-7cf3-4306-9079-e2f5c77a2aaa:/dev/nvme0n1p3 efe137ae-166c-43f9-a95b-2117010912ec:/dev/nvme0n1p4]
Jul 07 11:40:11.946879 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.946823    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 07 11:40:11.954225 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.954112    2581 manager.go:216] Machine: {Timestamp:2026-07-07 11:40:11.952783368 +0000 UTC m=+2.299996984 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108945 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec234dcc3a5d76f1930df5a868453efd SystemUUID:ec234dcc-3a5d-76f1-930d-f5a868453efd BootID:e875c899-73de-400f-952b-70c128a43e31 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:cd:b3:8c:6e:8b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:cd:b3:8c:6e:8b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4a:b1:04:7b:99:5e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 07 11:40:11.954225 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.954219    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 07 11:40:11.954354 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.954342    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 07 11:40:11.957440 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.957413    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 07 11:40:11.957589 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.957444    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-183.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 07 11:40:11.958363 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.958350    2581 topology_manager.go:138] "Creating topology manager with none policy"
Jul 07 11:40:11.958398 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.958365    2581 container_manager_linux.go:309] "Creating device plugin manager"
Jul 07 11:40:11.958398 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.958373    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 11:40:11.958927 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.958916    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 11:40:11.960029 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.960017    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 07 11:40:11.963670 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.963653    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 11:40:11.964163 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.964151    2581 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 11:40:11.964292 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.964282    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 07 11:40:11.964324 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.964312    2581 kubelet.go:486] "Attempting to sync node with API server"
Jul 07 11:40:11.964324 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.964323    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 07 11:40:11.964378 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.964340    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 07 11:40:11.964378 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.964350    2581 kubelet.go:398] "Adding apiserver pod source"
Jul 07 11:40:11.964378 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.964359    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 07 11:40:11.964901 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.964854    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 11:40:11.964901 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.964899    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 11:40:11.967206 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.967186    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 07 11:40:11.969292 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.969267    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 07 11:40:11.969355 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.969323    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 07 11:40:11.970343 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.970330    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 07 11:40:11.970384 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.970348    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 07 11:40:11.970384 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.970356    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 07 11:40:11.970384 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.970372    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 07 11:40:11.970384 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.970382    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 07 11:40:11.970496 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.970388    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 07 11:40:11.970496 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.970396    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 07 11:40:11.970496 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.970402    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 07 11:40:11.970496 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.970408    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 07 11:40:11.970496 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.970414    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 07 11:40:11.970496 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.970429    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 07 11:40:11.970496 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.970434    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 07 11:40:11.971691 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.971679    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 07 11:40:11.971691 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.971692    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 07 11:40:11.974212 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.974192    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-183.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 07 11:40:11.974212 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:11.974200    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 07 11:40:11.974346 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:11.974195    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-183.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 07 11:40:11.975557 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.975542    2581 server.go:1268] "Started kubelet"
Jul 07 11:40:11.975664 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.975628    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 07 11:40:11.975770 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.975705    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 07 11:40:11.975812 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.975798    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 07 11:40:11.979119 ip-10-0-138-183 systemd[1]: Started Kubernetes Kubelet.
Jul 07 11:40:11.980211 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.980165    2581 server.go:310] "Adding debug handlers to kubelet server"
Jul 07 11:40:11.980290 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.980244    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 07 11:40:11.984619 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:11.984595    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 07 11:40:11.986500 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:11.985509    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-183.ec2.internal.18bfff0dbf8f6151  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-183.ec2.internal,UID:ip-10-0-138-183.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-183.ec2.internal,},FirstTimestamp:2026-07-07 11:40:11.975508305 +0000 UTC m=+2.322721922,LastTimestamp:2026-07-07 11:40:11.975508305 +0000 UTC m=+2.322721922,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-183.ec2.internal,}"
Jul 07 11:40:11.986861 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.986840    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 07 11:40:11.986861 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.986852    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 07 11:40:11.986993 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:11.986938    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-183.ec2.internal\" not found"
Jul 07 11:40:11.987443 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.987424    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 07 11:40:11.987443 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.987424    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 07 11:40:11.987572 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.987454    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 07 11:40:11.987572 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.987543    2581 reconstruct.go:97] "Volume reconstruction finished"
Jul 07 11:40:11.987572 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.987553    2581 reconciler.go:29] "Reconciler: start to sync state"
Jul 07 11:40:11.988121 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.988103    2581 factory.go:55] Registering systemd factory
Jul 07 11:40:11.988121 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.988123    2581 factory.go:223] Registration of the systemd container factory successfully
Jul 07 11:40:11.989400 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.988921    2581 factory.go:153] Registering CRI-O factory
Jul 07 11:40:11.989400 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.988938    2581 factory.go:223] Registration of the crio container factory successfully
Jul 07 11:40:11.989400 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.988991    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 07 11:40:11.989400 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.989033    2581 factory.go:103] Registering Raw factory
Jul 07 11:40:11.989400 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.989048    2581 manager.go:1196] Started watching for new ooms in manager
Jul 07 11:40:11.989736 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:11.989498    2581 manager.go:349] Starting recovery of all containers
Jul 07 11:40:11.996796 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:11.996771    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 07 11:40:11.996877 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:11.996768    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-183.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 07 11:40:12.001407 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.001389    2581 manager.go:354] Recovery completed
Jul 07 11:40:12.003192 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:12.003162    2581 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 07 11:40:12.006437 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.006418    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 11:40:12.008324 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.008299    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-183.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 11:40:12.008391 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.008345    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-183.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 11:40:12.008391 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.008365    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-183.ec2.internal" event="NodeHasSufficientPID"
Jul 07 11:40:12.008622 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.008605    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wxtnh"
Jul 07 11:40:12.009272 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.009252    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 07 11:40:12.009272 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.009273    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 07 11:40:12.009396 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.009277    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 11:40:12.009396 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.009293    2581 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 11:40:12.010262 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.010241    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-183.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 11:40:12.010374 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.010283    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-183.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 11:40:12.010374 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.010301    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-183.ec2.internal" event="NodeHasSufficientPID"
Jul 07 11:40:12.010374 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:12.010264    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-183.ec2.internal.18bfff0dc18420f7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-183.ec2.internal,UID:ip-10-0-138-183.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-183.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-183.ec2.internal,},FirstTimestamp:2026-07-07 11:40:12.008325367 +0000 UTC m=+2.355538989,LastTimestamp:2026-07-07 11:40:12.008325367 +0000 UTC m=+2.355538989,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-183.ec2.internal,}"
Jul 07 11:40:12.011857 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.011839    2581 policy_none.go:49] "None policy: Start"
Jul 07 11:40:12.011923 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.011867    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 07 11:40:12.011923 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.011879    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 07 11:40:12.013941 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.013929    2581 policy_none.go:47] "Start"
Jul 07 11:40:12.018303 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.018287    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wxtnh"
Jul 07 11:40:12.022314 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:12.022151    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-183.ec2.internal.18bfff0dc184901b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-183.ec2.internal,UID:ip-10-0-138-183.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-183.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-183.ec2.internal,},FirstTimestamp:2026-07-07 11:40:12.008353819 +0000 UTC m=+2.355567437,LastTimestamp:2026-07-07 11:40:12.008353819 +0000 UTC m=+2.355567437,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-183.ec2.internal,}"
Jul 07 11:40:12.048692 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.048676    2581 manager.go:337] "Starting Device Plugin manager"
Jul 07 11:40:12.048787 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:12.048713    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 07 11:40:12.048787 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.048727    2581 server.go:87] "Starting device plugin registration server"
Jul 07 11:40:12.048982 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.048960    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 07 11:40:12.049127 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.048976    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 07 11:40:12.049185 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.049146    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 07 11:40:12.049231 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.049217    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 07 11:40:12.049277 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.049232    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 07 11:40:12.049608 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:12.049587    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 07 11:40:12.049697 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:12.049636    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-183.ec2.internal\" not found"
Jul 07 11:40:12.152228 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.152157    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 11:40:12.154401 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.153302    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-183.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 11:40:12.154401 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.153335    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-183.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 11:40:12.154401 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.153347    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-183.ec2.internal" event="NodeHasSufficientPID"
Jul 07 11:40:12.154401 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.153371    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-183.ec2.internal"
Jul 07 11:40:12.163355 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.163336    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-183.ec2.internal"
Jul 07 11:40:12.163459 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:12.163363    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-183.ec2.internal\": node \"ip-10-0-138-183.ec2.internal\" not found"
Jul 07 11:40:12.164613 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.164580    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 07 11:40:12.165842 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.165822    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 07 11:40:12.165933 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.165863    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 07 11:40:12.165933 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.165882    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 07 11:40:12.165933 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:12.165913    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 07 11:40:12.168726 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.168708    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 11:40:12.198981 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:12.198963    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-183.ec2.internal\" not found"
Jul 07 11:40:12.266716 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.266678    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-183.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-183.ec2.internal"]
Jul 07 11:40:12.266810 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.266756    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 11:40:12.268234 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.268217    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-183.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 11:40:12.268321 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.268248    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-183.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 11:40:12.268321 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.268260    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-183.ec2.internal" event="NodeHasSufficientPID"
Jul 07 11:40:12.270472 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.270459    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 11:40:12.270653 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.270640    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-183.ec2.internal"
Jul 07 11:40:12.270718 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.270670    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 11:40:12.271166 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.271148    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-183.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 11:40:12.271166 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.271159    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-183.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 11:40:12.271270 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.271179    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-183.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 11:40:12.271270 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.271180    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-183.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 11:40:12.271270 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.271191    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-183.ec2.internal" event="NodeHasSufficientPID"
Jul 07 11:40:12.271270 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.271198    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-183.ec2.internal" event="NodeHasSufficientPID"
Jul 07 11:40:12.273464 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.273447    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-183.ec2.internal"
Jul 07 11:40:12.273560 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.273476    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 11:40:12.274217 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.274201    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-183.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 11:40:12.274301 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.274228    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-183.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 11:40:12.274301 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.274238    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-183.ec2.internal" event="NodeHasSufficientPID"
Jul 07 11:40:12.286643 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:12.286622    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-183.ec2.internal\" not found" node="ip-10-0-138-183.ec2.internal"
Jul 07 11:40:12.288674 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.288651    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/30f4a1734807f8863906bb6425570933-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-183.ec2.internal\" (UID: \"30f4a1734807f8863906bb6425570933\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-183.ec2.internal"
Jul 07 11:40:12.288793 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.288679    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30f4a1734807f8863906bb6425570933-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-183.ec2.internal\" (UID: \"30f4a1734807f8863906bb6425570933\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-183.ec2.internal"
Jul 07 11:40:12.288793 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.288698    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f35e0bd4cbf1174e802ed0ff417ef039-config\") pod \"kube-apiserver-proxy-ip-10-0-138-183.ec2.internal\" (UID: \"f35e0bd4cbf1174e802ed0ff417ef039\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-183.ec2.internal"
Jul 07 11:40:12.290802 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:12.290784    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-183.ec2.internal\" not found" node="ip-10-0-138-183.ec2.internal"
Jul 07 11:40:12.299952 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:12.299931    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-183.ec2.internal\" not found"
Jul 07 11:40:12.389092 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.389068    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/30f4a1734807f8863906bb6425570933-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-183.ec2.internal\" (UID: \"30f4a1734807f8863906bb6425570933\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-183.ec2.internal"
Jul 07 11:40:12.389175 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.389100    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30f4a1734807f8863906bb6425570933-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-183.ec2.internal\" (UID: \"30f4a1734807f8863906bb6425570933\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-183.ec2.internal"
Jul 07 11:40:12.389175 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.389117    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f35e0bd4cbf1174e802ed0ff417ef039-config\") pod \"kube-apiserver-proxy-ip-10-0-138-183.ec2.internal\" (UID: \"f35e0bd4cbf1174e802ed0ff417ef039\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-183.ec2.internal"
Jul 07 11:40:12.389175 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.389160    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f35e0bd4cbf1174e802ed0ff417ef039-config\") pod \"kube-apiserver-proxy-ip-10-0-138-183.ec2.internal\" (UID: \"f35e0bd4cbf1174e802ed0ff417ef039\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-183.ec2.internal"
Jul 07 11:40:12.389276 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.389188    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30f4a1734807f8863906bb6425570933-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-183.ec2.internal\" (UID: \"30f4a1734807f8863906bb6425570933\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-183.ec2.internal"
Jul 07 11:40:12.389276 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.389166    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/30f4a1734807f8863906bb6425570933-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-183.ec2.internal\" (UID: \"30f4a1734807f8863906bb6425570933\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-183.ec2.internal"
Jul 07 11:40:12.400123 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:12.400100    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-183.ec2.internal\" not found"
Jul 07 11:40:12.500904 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:12.500837    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-183.ec2.internal\" not found"
Jul 07 11:40:12.594219 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.594203    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-183.ec2.internal"
Jul 07 11:40:12.594219 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.594218    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-183.ec2.internal"
Jul 07 11:40:12.601578 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:12.601554    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-183.ec2.internal\" not found"
Jul 07 11:40:12.702107 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:12.702078    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-183.ec2.internal\" not found"
Jul 07 11:40:12.802994 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:12.802939    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-183.ec2.internal\" not found"
Jul 07 11:40:12.903425 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:12.903391    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-183.ec2.internal\" not found"
Jul 07 11:40:12.917181 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.917157    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 07 11:40:12.917319 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:12.917302    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 11:40:13.003574 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:13.003547    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-183.ec2.internal\" not found"
Jul 07 11:40:13.021272 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.021243    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 11:35:12 +0000 UTC" deadline="2028-02-21 14:13:04.078809438 +0000 UTC"
Jul 07 11:40:13.021355 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.021274    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14258h32m51.057542543s"
Jul 07 11:40:13.090925 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.090876    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 11:40:13.187918 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.187888    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-183.ec2.internal"
Jul 07 11:40:13.206650 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.206626    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 11:40:13.208379 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.208361    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-183.ec2.internal"
Jul 07 11:40:13.216112 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.216096    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 11:40:13.220037 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:13.219998    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f4a1734807f8863906bb6425570933.slice/crio-8889a8da34db43a5a0a4d663d5f1d83f9b278bdc7a1778ddd3b5855240ee6da9 WatchSource:0}: Error finding container 8889a8da34db43a5a0a4d663d5f1d83f9b278bdc7a1778ddd3b5855240ee6da9: Status 404 returned error can't find the container with id 8889a8da34db43a5a0a4d663d5f1d83f9b278bdc7a1778ddd3b5855240ee6da9
Jul 07 11:40:13.225139 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.225120    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 11:40:13.312458 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.312435    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 11:40:13.573932 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.573861    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 11:40:13.771481 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.771439    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 11:40:13.968092 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.968064    2581 apiserver.go:52] "Watching apiserver"
Jul 07 11:40:13.978059 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.978033    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 07 11:40:13.978586 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.978558    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-86xkz","openshift-network-operator/iptables-alerter-rmjgp","openshift-ovn-kubernetes/ovnkube-node-kf48j","kube-system/konnectivity-agent-8c7wn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl9lc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-183.ec2.internal","openshift-multus/multus-9j6l9","openshift-multus/multus-additional-cni-plugins-p5l2w","openshift-multus/network-metrics-daemon-mt9fg","kube-system/global-pull-secret-syncer-v289k","kube-system/kube-apiserver-proxy-ip-10-0-138-183.ec2.internal","openshift-cluster-node-tuning-operator/tuned-n49lw","openshift-dns/node-resolver-r47kl","openshift-image-registry/node-ca-gpk8v"]
Jul 07 11:40:13.981409 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.981380    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:13.984210 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.984186    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:13.984314 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:13.984275    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt9fg" podUID="fd7d46ac-c068-44b2-adca-77ddc1da152e"
Jul 07 11:40:13.984821 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.984793    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 07 11:40:13.985586 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.985564    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 07 11:40:13.985741 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.985720    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ml5xv\""
Jul 07 11:40:13.985803 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.985781    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 07 11:40:13.985872 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.985564    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 07 11:40:13.986358 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.986339    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8c7wn"
Jul 07 11:40:13.988223 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.988202    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 07 11:40:13.988823 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.988804    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:40:13.988908 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:13.988868    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86xkz" podUID="9f060945-568c-484f-affd-2bbaa59faa07"
Jul 07 11:40:13.989066 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.989020    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 07 11:40:13.990457 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.990093    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 07 11:40:13.990457 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.990312    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8sfkw\""
Jul 07 11:40:13.991038 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.990991    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl9lc"
Jul 07 11:40:13.993899 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.993580    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:13.994481 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.994224    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 07 11:40:13.994481 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.994318    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 07 11:40:13.995488 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.995226    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-k77vh\""
Jul 07 11:40:13.995488 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.995315    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 07 11:40:13.998717 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.996203    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p5l2w"
Jul 07 11:40:13.998717 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.997652    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 07 11:40:13.998717 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.998245    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 07 11:40:13.998717 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.998383    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-v9stq\""
Jul 07 11:40:13.998717 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.998469    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 07 11:40:13.998717 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:13.998601    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 07 11:40:14.000200 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.000180    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rmjgp"
Jul 07 11:40:14.003445 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.000697    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 07 11:40:14.003445 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.001745    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd7d46ac-c068-44b2-adca-77ddc1da152e-metrics-certs\") pod \"network-metrics-daemon-mt9fg\" (UID: \"fd7d46ac-c068-44b2-adca-77ddc1da152e\") " pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:14.003445 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.001787    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f1745e00-65cf-4d21-9fd1-4ff342a9bf55-konnectivity-ca\") pod \"konnectivity-agent-8c7wn\" (UID: \"f1745e00-65cf-4d21-9fd1-4ff342a9bf55\") " pod="kube-system/konnectivity-agent-8c7wn"
Jul 07 11:40:14.003445 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.001816    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9dzb\" (UniqueName: \"kubernetes.io/projected/9f060945-568c-484f-affd-2bbaa59faa07-kube-api-access-c9dzb\") pod \"network-check-target-86xkz\" (UID: \"9f060945-568c-484f-affd-2bbaa59faa07\") " pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:40:14.003445 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.001851    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-multus-cni-dir\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.003445 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.001886    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-host-run-netns\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.003445 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.001920    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-host-var-lib-cni-bin\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.003445 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.001953    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-host-var-lib-kubelet\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.003445 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.002006    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8zzr\" (UniqueName: \"kubernetes.io/projected/fd7d46ac-c068-44b2-adca-77ddc1da152e-kube-api-access-k8zzr\") pod \"network-metrics-daemon-mt9fg\" (UID: \"fd7d46ac-c068-44b2-adca-77ddc1da152e\") " pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:14.003445 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.002042    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f1745e00-65cf-4d21-9fd1-4ff342a9bf55-agent-certs\") pod \"konnectivity-agent-8c7wn\" (UID: \"f1745e00-65cf-4d21-9fd1-4ff342a9bf55\") " pod="kube-system/konnectivity-agent-8c7wn"
Jul 07 11:40:14.003445 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.002076    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-cnibin\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.003445 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.002133    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-hostroot\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.003445 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.002165    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 07 11:40:14.003445 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.002481    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 07 11:40:14.003445 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.002169    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-etc-kubernetes\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.003445 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.002654    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-system-cni-dir\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.003445 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.002684    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70d0c362-29de-4978-b557-ce4181aabb9a-cni-binary-copy\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.003445 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.002719    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-multus-socket-dir-parent\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.004473 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.002803    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-host-run-k8s-cni-cncf-io\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.004473 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.002833    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-multus-conf-dir\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.004473 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.002859    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/70d0c362-29de-4978-b557-ce4181aabb9a-multus-daemon-config\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.004473 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.002884    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:14.004473 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.002892    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-host-run-multus-certs\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.004473 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.002927    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-os-release\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.004473 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:14.002974    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v289k" podUID="140115cd-5270-4373-b7eb-3db479036e81"
Jul 07 11:40:14.004473 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.002958    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-host-var-lib-cni-multus\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.004473 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.003009    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 07 11:40:14.004473 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.003037    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wk7g\" (UniqueName: \"kubernetes.io/projected/70d0c362-29de-4978-b557-ce4181aabb9a-kube-api-access-2wk7g\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.005687 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.005666    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.006275 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.006255    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 07 11:40:14.006376 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.006321    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 07 11:40:14.006861 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.006843    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-f4fkw\""
Jul 07 11:40:14.007061 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.007038    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-djd8f\""
Jul 07 11:40:14.007141 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.007114    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 07 11:40:14.007441 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.007421    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 11:40:14.008708 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.008687    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vg67c\""
Jul 07 11:40:14.009833 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.009567    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r47kl"
Jul 07 11:40:14.010643 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.010622    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 07 11:40:14.010859 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.010833    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 07 11:40:14.013276 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.013021    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 07 11:40:14.013276 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.013093    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-462vm\""
Jul 07 11:40:14.013438 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.013351    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 07 11:40:14.014049 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.014031    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gpk8v"
Jul 07 11:40:14.019129 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.018403    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 07 11:40:14.019129 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.018627    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 07 11:40:14.019129 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.018710    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xs4br\""
Jul 07 11:40:14.019129 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.018921    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 07 11:40:14.039252 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.039186    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-77m97"
Jul 07 11:40:14.053114 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.053056    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-77m97"
Jul 07 11:40:14.088814 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.088794    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 07 11:40:14.103215 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.103191    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f1745e00-65cf-4d21-9fd1-4ff342a9bf55-agent-certs\") pod \"konnectivity-agent-8c7wn\" (UID: \"f1745e00-65cf-4d21-9fd1-4ff342a9bf55\") " pod="kube-system/konnectivity-agent-8c7wn"
Jul 07 11:40:14.103332 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.103222    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-hostroot\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.103332 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.103244    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/245c8810-46ea-4bd3-97d8-34e8b93e0312-sys-fs\") pod \"aws-ebs-csi-driver-node-rl9lc\" (UID: \"245c8810-46ea-4bd3-97d8-34e8b93e0312\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl9lc"
Jul 07 11:40:14.103332 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.103263    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8l9x\" (UniqueName: \"kubernetes.io/projected/7b7d6bea-562a-423c-bd1c-4b12eb5103ef-kube-api-access-v8l9x\") pod \"node-ca-gpk8v\" (UID: \"7b7d6bea-562a-423c-bd1c-4b12eb5103ef\") " pod="openshift-image-registry/node-ca-gpk8v"
Jul 07 11:40:14.103332 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.103289    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-host-slash\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.103332 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.103313    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-host-run-netns\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.103602 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.103340    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-hostroot\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.103602 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.103346    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/17e097fb-7012-4f55-a0c1-8cc356fc093d-etc-sysconfig\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.103602 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.103406    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-host-run-k8s-cni-cncf-io\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.103602 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.103431    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/140115cd-5270-4373-b7eb-3db479036e81-kubelet-config\") pod \"global-pull-secret-syncer-v289k\" (UID: \"140115cd-5270-4373-b7eb-3db479036e81\") " pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:14.103602 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.103456    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7b7d6bea-562a-423c-bd1c-4b12eb5103ef-serviceca\") pod \"node-ca-gpk8v\" (UID: \"7b7d6bea-562a-423c-bd1c-4b12eb5103ef\") " pod="openshift-image-registry/node-ca-gpk8v"
Jul 07 11:40:14.103602 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.103481    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-systemd-units\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.103602 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.103497    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-host-run-k8s-cni-cncf-io\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.103602 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.103505    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-var-lib-openvswitch\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.103602 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.103562    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-host-run-ovn-kubernetes\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.103602 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.103593    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5d7ce38f-f030-42d2-a486-84f6a8b6ebc4-cni-binary-copy\") pod \"multus-additional-cni-plugins-p5l2w\" (UID: \"5d7ce38f-f030-42d2-a486-84f6a8b6ebc4\") " pod="openshift-multus/multus-additional-cni-plugins-p5l2w"
Jul 07 11:40:14.104049 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.103621    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/245c8810-46ea-4bd3-97d8-34e8b93e0312-socket-dir\") pod \"aws-ebs-csi-driver-node-rl9lc\" (UID: \"245c8810-46ea-4bd3-97d8-34e8b93e0312\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl9lc"
Jul 07 11:40:14.104049 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.103629    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 07 11:40:14.104049 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.103653    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b7d6bea-562a-423c-bd1c-4b12eb5103ef-host\") pod \"node-ca-gpk8v\" (UID: \"7b7d6bea-562a-423c-bd1c-4b12eb5103ef\") " pod="openshift-image-registry/node-ca-gpk8v"
Jul 07 11:40:14.104049 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.103695    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-run-systemd\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.104049 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.103721    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d7ce38f-f030-42d2-a486-84f6a8b6ebc4-system-cni-dir\") pod \"multus-additional-cni-plugins-p5l2w\" (UID: \"5d7ce38f-f030-42d2-a486-84f6a8b6ebc4\") " pod="openshift-multus/multus-additional-cni-plugins-p5l2w"
Jul 07 11:40:14.104049 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.103746    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz4rg\" (UniqueName: \"kubernetes.io/projected/5d7ce38f-f030-42d2-a486-84f6a8b6ebc4-kube-api-access-vz4rg\") pod \"multus-additional-cni-plugins-p5l2w\" (UID: \"5d7ce38f-f030-42d2-a486-84f6a8b6ebc4\") " pod="openshift-multus/multus-additional-cni-plugins-p5l2w"
Jul 07 11:40:14.104049 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.103771    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17e097fb-7012-4f55-a0c1-8cc356fc093d-sys\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.104049 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.103803    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/17e097fb-7012-4f55-a0c1-8cc356fc093d-lib-modules\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.104049 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.103844    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f1745e00-65cf-4d21-9fd1-4ff342a9bf55-konnectivity-ca\") pod \"konnectivity-agent-8c7wn\" (UID: \"f1745e00-65cf-4d21-9fd1-4ff342a9bf55\") " pod="kube-system/konnectivity-agent-8c7wn"
Jul 07 11:40:14.104049 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.103908    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67bvn\" (UniqueName: \"kubernetes.io/projected/245c8810-46ea-4bd3-97d8-34e8b93e0312-kube-api-access-67bvn\") pod \"aws-ebs-csi-driver-node-rl9lc\" (UID: \"245c8810-46ea-4bd3-97d8-34e8b93e0312\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl9lc"
Jul 07 11:40:14.104049 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.103944    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-host-kubelet\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.104049 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.103968    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-ovnkube-config\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.104049 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.103992    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/17e097fb-7012-4f55-a0c1-8cc356fc093d-etc-tuned\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.104049 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104030    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlq6p\" (UniqueName: \"kubernetes.io/projected/04b0713d-434f-4a0a-af55-d93a2a9e9804-kube-api-access-xlq6p\") pod \"node-resolver-r47kl\" (UID: \"04b0713d-434f-4a0a-af55-d93a2a9e9804\") " pod="openshift-dns/node-resolver-r47kl"
Jul 07 11:40:14.104049 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104062    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8zzr\" (UniqueName: \"kubernetes.io/projected/fd7d46ac-c068-44b2-adca-77ddc1da152e-kube-api-access-k8zzr\") pod \"network-metrics-daemon-mt9fg\" (UID: \"fd7d46ac-c068-44b2-adca-77ddc1da152e\") " pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:14.104952 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104087    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/245c8810-46ea-4bd3-97d8-34e8b93e0312-etc-selinux\") pod \"aws-ebs-csi-driver-node-rl9lc\" (UID: \"245c8810-46ea-4bd3-97d8-34e8b93e0312\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl9lc"
Jul 07 11:40:14.104952 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104117    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.104952 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104144    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5d7ce38f-f030-42d2-a486-84f6a8b6ebc4-cnibin\") pod \"multus-additional-cni-plugins-p5l2w\" (UID: \"5d7ce38f-f030-42d2-a486-84f6a8b6ebc4\") " pod="openshift-multus/multus-additional-cni-plugins-p5l2w"
Jul 07 11:40:14.104952 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104168    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17e097fb-7012-4f55-a0c1-8cc356fc093d-var-lib-kubelet\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.104952 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104217    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-system-cni-dir\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.104952 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104241    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70d0c362-29de-4978-b557-ce4181aabb9a-cni-binary-copy\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.104952 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104269    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/70d0c362-29de-4978-b557-ce4181aabb9a-multus-daemon-config\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.104952 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104298    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-host-run-multus-certs\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.104952 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104324    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-etc-openvswitch\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.104952 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104351    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5d7ce38f-f030-42d2-a486-84f6a8b6ebc4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p5l2w\" (UID: \"5d7ce38f-f030-42d2-a486-84f6a8b6ebc4\") " pod="openshift-multus/multus-additional-cni-plugins-p5l2w"
Jul 07 11:40:14.104952 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104377    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17e097fb-7012-4f55-a0c1-8cc356fc093d-host\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.104952 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104404    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wk7g\" (UniqueName: \"kubernetes.io/projected/70d0c362-29de-4978-b557-ce4181aabb9a-kube-api-access-2wk7g\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.104952 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104449    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-run-ovn\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.104952 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104454    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-host-run-multus-certs\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.104952 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104480    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-env-overrides\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.104952 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104507    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5d7ce38f-f030-42d2-a486-84f6a8b6ebc4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p5l2w\" (UID: \"5d7ce38f-f030-42d2-a486-84f6a8b6ebc4\") " pod="openshift-multus/multus-additional-cni-plugins-p5l2w"
Jul 07 11:40:14.104952 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104526    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-system-cni-dir\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.105858 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104532    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1937f0c-df84-4f64-b4be-678b7e871e8c-host-slash\") pod \"iptables-alerter-rmjgp\" (UID: \"f1937f0c-df84-4f64-b4be-678b7e871e8c\") " pod="openshift-network-operator/iptables-alerter-rmjgp"
Jul 07 11:40:14.105858 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104557    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/04b0713d-434f-4a0a-af55-d93a2a9e9804-hosts-file\") pod \"node-resolver-r47kl\" (UID: \"04b0713d-434f-4a0a-af55-d93a2a9e9804\") " pod="openshift-dns/node-resolver-r47kl"
Jul 07 11:40:14.105858 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104581    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/245c8810-46ea-4bd3-97d8-34e8b93e0312-device-dir\") pod \"aws-ebs-csi-driver-node-rl9lc\" (UID: \"245c8810-46ea-4bd3-97d8-34e8b93e0312\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl9lc"
Jul 07 11:40:14.105858 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104608    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-ovn-node-metrics-cert\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.105858 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104633    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-host-cni-bin\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.105858 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104657    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-cnibin\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.105858 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104682    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/245c8810-46ea-4bd3-97d8-34e8b93e0312-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rl9lc\" (UID: \"245c8810-46ea-4bd3-97d8-34e8b93e0312\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl9lc"
Jul 07 11:40:14.105858 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104708    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-host-cni-netd\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.105858 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104735    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-multus-conf-dir\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.105858 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104758    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-ovnkube-script-lib\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.105858 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104782    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/17e097fb-7012-4f55-a0c1-8cc356fc093d-etc-modprobe-d\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.105858 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104809    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m4dm\" (UniqueName: \"kubernetes.io/projected/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-kube-api-access-7m4dm\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.105858 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104840    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5d7ce38f-f030-42d2-a486-84f6a8b6ebc4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p5l2w\" (UID: \"5d7ce38f-f030-42d2-a486-84f6a8b6ebc4\") " pod="openshift-multus/multus-additional-cni-plugins-p5l2w"
Jul 07 11:40:14.105858 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104867    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfjlr\" (UniqueName: \"kubernetes.io/projected/f1937f0c-df84-4f64-b4be-678b7e871e8c-kube-api-access-zfjlr\") pod \"iptables-alerter-rmjgp\" (UID: \"f1937f0c-df84-4f64-b4be-678b7e871e8c\") " pod="openshift-network-operator/iptables-alerter-rmjgp"
Jul 07 11:40:14.105858 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104892    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/17e097fb-7012-4f55-a0c1-8cc356fc093d-run\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.105858 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104920    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17e097fb-7012-4f55-a0c1-8cc356fc093d-tmp\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.105858 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104950    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-cnibin\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.106672 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104951    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd7d46ac-c068-44b2-adca-77ddc1da152e-metrics-certs\") pod \"network-metrics-daemon-mt9fg\" (UID: \"fd7d46ac-c068-44b2-adca-77ddc1da152e\") " pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:14.106672 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.105028    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-multus-conf-dir\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.106672 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:14.105049    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:14.106672 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.104890    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f1745e00-65cf-4d21-9fd1-4ff342a9bf55-konnectivity-ca\") pod \"konnectivity-agent-8c7wn\" (UID: \"f1745e00-65cf-4d21-9fd1-4ff342a9bf55\") " pod="kube-system/konnectivity-agent-8c7wn"
Jul 07 11:40:14.106672 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.105079    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9dzb\" (UniqueName: \"kubernetes.io/projected/9f060945-568c-484f-affd-2bbaa59faa07-kube-api-access-c9dzb\") pod \"network-check-target-86xkz\" (UID: \"9f060945-568c-484f-affd-2bbaa59faa07\") " pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:40:14.106672 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.105138    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70d0c362-29de-4978-b557-ce4181aabb9a-cni-binary-copy\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.106672 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.105233    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-multus-cni-dir\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.106672 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.105261    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/70d0c362-29de-4978-b557-ce4181aabb9a-multus-daemon-config\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.106672 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.105421    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-host-var-lib-kubelet\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.106672 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.105493    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-multus-cni-dir\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.106672 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:14.105612    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd7d46ac-c068-44b2-adca-77ddc1da152e-metrics-certs podName:fd7d46ac-c068-44b2-adca-77ddc1da152e nodeName:}" failed. No retries permitted until 2026-07-07 11:40:14.605585862 +0000 UTC m=+4.952799469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd7d46ac-c068-44b2-adca-77ddc1da152e-metrics-certs") pod "network-metrics-daemon-mt9fg" (UID: "fd7d46ac-c068-44b2-adca-77ddc1da152e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:14.106672 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.105264    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-host-var-lib-kubelet\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.106672 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.105722    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-run-openvswitch\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.106672 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.105748    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f1937f0c-df84-4f64-b4be-678b7e871e8c-iptables-alerter-script\") pod \"iptables-alerter-rmjgp\" (UID: \"f1937f0c-df84-4f64-b4be-678b7e871e8c\") " pod="openshift-network-operator/iptables-alerter-rmjgp"
Jul 07 11:40:14.106672 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.105774    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-etc-kubernetes\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.106672 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.105810    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/245c8810-46ea-4bd3-97d8-34e8b93e0312-registration-dir\") pod \"aws-ebs-csi-driver-node-rl9lc\" (UID: \"245c8810-46ea-4bd3-97d8-34e8b93e0312\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl9lc"
Jul 07 11:40:14.106672 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.105846    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-etc-kubernetes\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.107360 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.105882    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/140115cd-5270-4373-b7eb-3db479036e81-original-pull-secret\") pod \"global-pull-secret-syncer-v289k\" (UID: \"140115cd-5270-4373-b7eb-3db479036e81\") " pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:14.107360 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.105910    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/17e097fb-7012-4f55-a0c1-8cc356fc093d-etc-sysctl-d\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.107360 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.105948    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/17e097fb-7012-4f55-a0c1-8cc356fc093d-etc-systemd\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.107360 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.105972    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgcnd\" (UniqueName: \"kubernetes.io/projected/17e097fb-7012-4f55-a0c1-8cc356fc093d-kube-api-access-cgcnd\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.107360 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.106003    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-multus-socket-dir-parent\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.107360 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.106036    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/140115cd-5270-4373-b7eb-3db479036e81-dbus\") pod \"global-pull-secret-syncer-v289k\" (UID: \"140115cd-5270-4373-b7eb-3db479036e81\") " pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:14.107360 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.106059    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-os-release\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.107360 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.106127    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-os-release\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.107360 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.106132    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-multus-socket-dir-parent\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.107360 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.106151    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-host-var-lib-cni-multus\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.107360 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.106177    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-node-log\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.107360 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.106198    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-log-socket\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.107360 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.106214    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17e097fb-7012-4f55-a0c1-8cc356fc093d-etc-kubernetes\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.107360 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.106232    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/17e097fb-7012-4f55-a0c1-8cc356fc093d-etc-sysctl-conf\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.107360 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.106268    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04b0713d-434f-4a0a-af55-d93a2a9e9804-tmp-dir\") pod \"node-resolver-r47kl\" (UID: \"04b0713d-434f-4a0a-af55-d93a2a9e9804\") " pod="openshift-dns/node-resolver-r47kl"
Jul 07 11:40:14.107360 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.106274    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-host-var-lib-cni-multus\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.107360 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.106286    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-host-run-netns\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.108112 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.106318    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-host-var-lib-cni-bin\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.108112 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.106349    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5d7ce38f-f030-42d2-a486-84f6a8b6ebc4-os-release\") pod \"multus-additional-cni-plugins-p5l2w\" (UID: \"5d7ce38f-f030-42d2-a486-84f6a8b6ebc4\") " pod="openshift-multus/multus-additional-cni-plugins-p5l2w"
Jul 07 11:40:14.108112 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.106355    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-host-run-netns\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.108112 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.106427    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70d0c362-29de-4978-b557-ce4181aabb9a-host-var-lib-cni-bin\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.109996 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.109974    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f1745e00-65cf-4d21-9fd1-4ff342a9bf55-agent-certs\") pod \"konnectivity-agent-8c7wn\" (UID: \"f1745e00-65cf-4d21-9fd1-4ff342a9bf55\") " pod="kube-system/konnectivity-agent-8c7wn"
Jul 07 11:40:14.112929 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:14.112907    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 11:40:14.112929 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:14.112937    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 11:40:14.113109 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:14.112953    2581 projected.go:196] Error preparing data for projected volume kube-api-access-c9dzb for pod openshift-network-diagnostics/network-check-target-86xkz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:14.113109 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:14.113015    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f060945-568c-484f-affd-2bbaa59faa07-kube-api-access-c9dzb podName:9f060945-568c-484f-affd-2bbaa59faa07 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:14.612997892 +0000 UTC m=+4.960211498 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-c9dzb" (UniqueName: "kubernetes.io/projected/9f060945-568c-484f-affd-2bbaa59faa07-kube-api-access-c9dzb") pod "network-check-target-86xkz" (UID: "9f060945-568c-484f-affd-2bbaa59faa07") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:14.115576 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.115498    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8zzr\" (UniqueName: \"kubernetes.io/projected/fd7d46ac-c068-44b2-adca-77ddc1da152e-kube-api-access-k8zzr\") pod \"network-metrics-daemon-mt9fg\" (UID: \"fd7d46ac-c068-44b2-adca-77ddc1da152e\") " pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:14.115818 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.115795    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wk7g\" (UniqueName: \"kubernetes.io/projected/70d0c362-29de-4978-b557-ce4181aabb9a-kube-api-access-2wk7g\") pod \"multus-9j6l9\" (UID: \"70d0c362-29de-4978-b557-ce4181aabb9a\") " pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.171611 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.171544    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-183.ec2.internal" event={"ID":"f35e0bd4cbf1174e802ed0ff417ef039","Type":"ContainerStarted","Data":"3f01d7c6e9cb1828e1efce67a27311eaae2c68f13c6652f98f9648d3825a3b3c"}
Jul 07 11:40:14.173091 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.173066    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-183.ec2.internal" event={"ID":"30f4a1734807f8863906bb6425570933","Type":"ContainerStarted","Data":"8889a8da34db43a5a0a4d663d5f1d83f9b278bdc7a1778ddd3b5855240ee6da9"}
Jul 07 11:40:14.207463 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.207392    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-run-openvswitch\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.207463 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.207436    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f1937f0c-df84-4f64-b4be-678b7e871e8c-iptables-alerter-script\") pod \"iptables-alerter-rmjgp\" (UID: \"f1937f0c-df84-4f64-b4be-678b7e871e8c\") " pod="openshift-network-operator/iptables-alerter-rmjgp"
Jul 07 11:40:14.207648 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.207468    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/245c8810-46ea-4bd3-97d8-34e8b93e0312-registration-dir\") pod \"aws-ebs-csi-driver-node-rl9lc\" (UID: \"245c8810-46ea-4bd3-97d8-34e8b93e0312\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl9lc"
Jul 07 11:40:14.207648 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.207496    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/140115cd-5270-4373-b7eb-3db479036e81-original-pull-secret\") pod \"global-pull-secret-syncer-v289k\" (UID: \"140115cd-5270-4373-b7eb-3db479036e81\") " pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:14.207648 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.207523    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/17e097fb-7012-4f55-a0c1-8cc356fc093d-etc-sysctl-d\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.207648 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.207548    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/17e097fb-7012-4f55-a0c1-8cc356fc093d-etc-systemd\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.207648 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.207573    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgcnd\" (UniqueName: \"kubernetes.io/projected/17e097fb-7012-4f55-a0c1-8cc356fc093d-kube-api-access-cgcnd\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.207648 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.207599    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/140115cd-5270-4373-b7eb-3db479036e81-dbus\") pod \"global-pull-secret-syncer-v289k\" (UID: \"140115cd-5270-4373-b7eb-3db479036e81\") " pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:14.207648 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.207625    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-node-log\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.207648 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.207632    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/245c8810-46ea-4bd3-97d8-34e8b93e0312-registration-dir\") pod \"aws-ebs-csi-driver-node-rl9lc\" (UID: \"245c8810-46ea-4bd3-97d8-34e8b93e0312\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl9lc"
Jul 07 11:40:14.207648 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.207648    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-log-socket\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.208155 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.207688    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17e097fb-7012-4f55-a0c1-8cc356fc093d-etc-kubernetes\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.208155 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.207720    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/17e097fb-7012-4f55-a0c1-8cc356fc093d-etc-sysctl-conf\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.208155 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.207749    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04b0713d-434f-4a0a-af55-d93a2a9e9804-tmp-dir\") pod \"node-resolver-r47kl\" (UID: \"04b0713d-434f-4a0a-af55-d93a2a9e9804\") " pod="openshift-dns/node-resolver-r47kl"
Jul 07 11:40:14.208155 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.207777    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5d7ce38f-f030-42d2-a486-84f6a8b6ebc4-os-release\") pod \"multus-additional-cni-plugins-p5l2w\" (UID: \"5d7ce38f-f030-42d2-a486-84f6a8b6ebc4\") " pod="openshift-multus/multus-additional-cni-plugins-p5l2w"
Jul 07 11:40:14.208155 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.207807    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/245c8810-46ea-4bd3-97d8-34e8b93e0312-sys-fs\") pod \"aws-ebs-csi-driver-node-rl9lc\" (UID: \"245c8810-46ea-4bd3-97d8-34e8b93e0312\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl9lc"
Jul 07 11:40:14.208155 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.207827    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8l9x\" (UniqueName: \"kubernetes.io/projected/7b7d6bea-562a-423c-bd1c-4b12eb5103ef-kube-api-access-v8l9x\") pod \"node-ca-gpk8v\" (UID: \"7b7d6bea-562a-423c-bd1c-4b12eb5103ef\") " pod="openshift-image-registry/node-ca-gpk8v"
Jul 07 11:40:14.208155 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.207836    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/17e097fb-7012-4f55-a0c1-8cc356fc093d-etc-sysctl-d\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.208155 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.207843    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-host-slash\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.208155 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.207862    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-host-run-netns\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.208155 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.207881    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/17e097fb-7012-4f55-a0c1-8cc356fc093d-etc-sysconfig\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.208155 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.207899    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/17e097fb-7012-4f55-a0c1-8cc356fc093d-etc-systemd\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.208155 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.207909    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/140115cd-5270-4373-b7eb-3db479036e81-kubelet-config\") pod \"global-pull-secret-syncer-v289k\" (UID: \"140115cd-5270-4373-b7eb-3db479036e81\") " pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:14.208155 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.207933    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7b7d6bea-562a-423c-bd1c-4b12eb5103ef-serviceca\") pod \"node-ca-gpk8v\" (UID: \"7b7d6bea-562a-423c-bd1c-4b12eb5103ef\") " pod="openshift-image-registry/node-ca-gpk8v"
Jul 07 11:40:14.208155 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.207957    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-systemd-units\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.208155 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.207976    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-var-lib-openvswitch\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.208155 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.207991    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-host-run-ovn-kubernetes\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.208155 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208008    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5d7ce38f-f030-42d2-a486-84f6a8b6ebc4-cni-binary-copy\") pod \"multus-additional-cni-plugins-p5l2w\" (UID: \"5d7ce38f-f030-42d2-a486-84f6a8b6ebc4\") " pod="openshift-multus/multus-additional-cni-plugins-p5l2w"
Jul 07 11:40:14.209062 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208026    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/245c8810-46ea-4bd3-97d8-34e8b93e0312-socket-dir\") pod \"aws-ebs-csi-driver-node-rl9lc\" (UID: \"245c8810-46ea-4bd3-97d8-34e8b93e0312\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl9lc"
Jul 07 11:40:14.209062 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208045    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b7d6bea-562a-423c-bd1c-4b12eb5103ef-host\") pod \"node-ca-gpk8v\" (UID: \"7b7d6bea-562a-423c-bd1c-4b12eb5103ef\") " pod="openshift-image-registry/node-ca-gpk8v"
Jul 07 11:40:14.209062 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208087    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-run-systemd\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.209062 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208115    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d7ce38f-f030-42d2-a486-84f6a8b6ebc4-system-cni-dir\") pod \"multus-additional-cni-plugins-p5l2w\" (UID: \"5d7ce38f-f030-42d2-a486-84f6a8b6ebc4\") " pod="openshift-multus/multus-additional-cni-plugins-p5l2w"
Jul 07 11:40:14.209062 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208136    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz4rg\" (UniqueName: \"kubernetes.io/projected/5d7ce38f-f030-42d2-a486-84f6a8b6ebc4-kube-api-access-vz4rg\") pod \"multus-additional-cni-plugins-p5l2w\" (UID: \"5d7ce38f-f030-42d2-a486-84f6a8b6ebc4\") " pod="openshift-multus/multus-additional-cni-plugins-p5l2w"
Jul 07 11:40:14.209062 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208153    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17e097fb-7012-4f55-a0c1-8cc356fc093d-sys\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.209062 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208169    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/17e097fb-7012-4f55-a0c1-8cc356fc093d-lib-modules\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.209062 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208193    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67bvn\" (UniqueName: \"kubernetes.io/projected/245c8810-46ea-4bd3-97d8-34e8b93e0312-kube-api-access-67bvn\") pod \"aws-ebs-csi-driver-node-rl9lc\" (UID: \"245c8810-46ea-4bd3-97d8-34e8b93e0312\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl9lc"
Jul 07 11:40:14.209062 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208196    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/140115cd-5270-4373-b7eb-3db479036e81-dbus\") pod \"global-pull-secret-syncer-v289k\" (UID: \"140115cd-5270-4373-b7eb-3db479036e81\") " pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:14.209062 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208216    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-host-kubelet\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.209062 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208243    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-ovnkube-config\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.209062 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208250    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-node-log\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.209062 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208264    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/17e097fb-7012-4f55-a0c1-8cc356fc093d-etc-tuned\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.209062 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208290    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlq6p\" (UniqueName: \"kubernetes.io/projected/04b0713d-434f-4a0a-af55-d93a2a9e9804-kube-api-access-xlq6p\") pod \"node-resolver-r47kl\" (UID: \"04b0713d-434f-4a0a-af55-d93a2a9e9804\") " pod="openshift-dns/node-resolver-r47kl"
Jul 07 11:40:14.209062 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208307    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/140115cd-5270-4373-b7eb-3db479036e81-kubelet-config\") pod \"global-pull-secret-syncer-v289k\" (UID: \"140115cd-5270-4373-b7eb-3db479036e81\") " pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:14.209062 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208317    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/245c8810-46ea-4bd3-97d8-34e8b93e0312-etc-selinux\") pod \"aws-ebs-csi-driver-node-rl9lc\" (UID: \"245c8810-46ea-4bd3-97d8-34e8b93e0312\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl9lc"
Jul 07 11:40:14.209062 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208344    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.209764 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208349    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-run-openvswitch\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.209764 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208371    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5d7ce38f-f030-42d2-a486-84f6a8b6ebc4-cnibin\") pod \"multus-additional-cni-plugins-p5l2w\" (UID: \"5d7ce38f-f030-42d2-a486-84f6a8b6ebc4\") " pod="openshift-multus/multus-additional-cni-plugins-p5l2w"
Jul 07 11:40:14.209764 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208388    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17e097fb-7012-4f55-a0c1-8cc356fc093d-var-lib-kubelet\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.209764 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208408    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-etc-openvswitch\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.209764 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208425    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5d7ce38f-f030-42d2-a486-84f6a8b6ebc4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p5l2w\" (UID: \"5d7ce38f-f030-42d2-a486-84f6a8b6ebc4\") " pod="openshift-multus/multus-additional-cni-plugins-p5l2w"
Jul 07 11:40:14.209764 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208447    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17e097fb-7012-4f55-a0c1-8cc356fc093d-host\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.209764 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208473    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-run-ovn\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.209764 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208498    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-env-overrides\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.209764 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208524    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5d7ce38f-f030-42d2-a486-84f6a8b6ebc4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p5l2w\" (UID: \"5d7ce38f-f030-42d2-a486-84f6a8b6ebc4\") " pod="openshift-multus/multus-additional-cni-plugins-p5l2w"
Jul 07 11:40:14.209764 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1937f0c-df84-4f64-b4be-678b7e871e8c-host-slash\") pod \"iptables-alerter-rmjgp\" (UID: \"f1937f0c-df84-4f64-b4be-678b7e871e8c\") " pod="openshift-network-operator/iptables-alerter-rmjgp"
Jul 07 11:40:14.209764 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208576    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/04b0713d-434f-4a0a-af55-d93a2a9e9804-hosts-file\") pod \"node-resolver-r47kl\" (UID: \"04b0713d-434f-4a0a-af55-d93a2a9e9804\") " pod="openshift-dns/node-resolver-r47kl"
Jul 07 11:40:14.209764 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208602    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/245c8810-46ea-4bd3-97d8-34e8b93e0312-device-dir\") pod \"aws-ebs-csi-driver-node-rl9lc\" (UID: \"245c8810-46ea-4bd3-97d8-34e8b93e0312\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl9lc"
Jul 07 11:40:14.209764 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208631    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-ovn-node-metrics-cert\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.209764 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208657    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-host-cni-bin\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.209764 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208684    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/245c8810-46ea-4bd3-97d8-34e8b93e0312-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rl9lc\" (UID: \"245c8810-46ea-4bd3-97d8-34e8b93e0312\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl9lc"
Jul 07 11:40:14.209764 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208711    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-host-cni-netd\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.209764 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208749    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-ovnkube-script-lib\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.210446 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208773    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/17e097fb-7012-4f55-a0c1-8cc356fc093d-etc-modprobe-d\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.210446 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208804    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m4dm\" (UniqueName: \"kubernetes.io/projected/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-kube-api-access-7m4dm\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.210446 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208825    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5d7ce38f-f030-42d2-a486-84f6a8b6ebc4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p5l2w\" (UID: \"5d7ce38f-f030-42d2-a486-84f6a8b6ebc4\") " pod="openshift-multus/multus-additional-cni-plugins-p5l2w"
Jul 07 11:40:14.210446 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208852    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfjlr\" (UniqueName: \"kubernetes.io/projected/f1937f0c-df84-4f64-b4be-678b7e871e8c-kube-api-access-zfjlr\") pod \"iptables-alerter-rmjgp\" (UID: \"f1937f0c-df84-4f64-b4be-678b7e871e8c\") " pod="openshift-network-operator/iptables-alerter-rmjgp"
Jul 07 11:40:14.210446 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208878    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/17e097fb-7012-4f55-a0c1-8cc356fc093d-run\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.210446 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208902    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17e097fb-7012-4f55-a0c1-8cc356fc093d-tmp\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.210446 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208903    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f1937f0c-df84-4f64-b4be-678b7e871e8c-iptables-alerter-script\") pod \"iptables-alerter-rmjgp\" (UID: \"f1937f0c-df84-4f64-b4be-678b7e871e8c\") " pod="openshift-network-operator/iptables-alerter-rmjgp"
Jul 07 11:40:14.210446 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.208961    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-log-socket\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.210446 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.209011    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17e097fb-7012-4f55-a0c1-8cc356fc093d-etc-kubernetes\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.210446 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:14.209073    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:14.210446 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:14.209148    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/140115cd-5270-4373-b7eb-3db479036e81-original-pull-secret podName:140115cd-5270-4373-b7eb-3db479036e81 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:14.709130483 +0000 UTC m=+5.056344096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/140115cd-5270-4373-b7eb-3db479036e81-original-pull-secret") pod "global-pull-secret-syncer-v289k" (UID: "140115cd-5270-4373-b7eb-3db479036e81") : object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:14.210446 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.209314    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5d7ce38f-f030-42d2-a486-84f6a8b6ebc4-cnibin\") pod \"multus-additional-cni-plugins-p5l2w\" (UID: \"5d7ce38f-f030-42d2-a486-84f6a8b6ebc4\") " pod="openshift-multus/multus-additional-cni-plugins-p5l2w"
Jul 07 11:40:14.210446 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.209375    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-etc-openvswitch\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.210446 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.209431    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17e097fb-7012-4f55-a0c1-8cc356fc093d-var-lib-kubelet\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.210446 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.209445    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/17e097fb-7012-4f55-a0c1-8cc356fc093d-etc-sysctl-conf\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.210446 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.209797    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04b0713d-434f-4a0a-af55-d93a2a9e9804-tmp-dir\") pod \"node-resolver-r47kl\" (UID: \"04b0713d-434f-4a0a-af55-d93a2a9e9804\") " pod="openshift-dns/node-resolver-r47kl"
Jul 07 11:40:14.210446 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.209814    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7b7d6bea-562a-423c-bd1c-4b12eb5103ef-serviceca\") pod \"node-ca-gpk8v\" (UID: \"7b7d6bea-562a-423c-bd1c-4b12eb5103ef\") " pod="openshift-image-registry/node-ca-gpk8v"
Jul 07 11:40:14.211333 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.209882    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-systemd-units\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.211333 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.209920    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5d7ce38f-f030-42d2-a486-84f6a8b6ebc4-os-release\") pod \"multus-additional-cni-plugins-p5l2w\" (UID: \"5d7ce38f-f030-42d2-a486-84f6a8b6ebc4\") " pod="openshift-multus/multus-additional-cni-plugins-p5l2w"
Jul 07 11:40:14.211333 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.209933    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-var-lib-openvswitch\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.211333 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.209983    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/245c8810-46ea-4bd3-97d8-34e8b93e0312-sys-fs\") pod \"aws-ebs-csi-driver-node-rl9lc\" (UID: \"245c8810-46ea-4bd3-97d8-34e8b93e0312\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl9lc"
Jul 07 11:40:14.211333 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.210003    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5d7ce38f-f030-42d2-a486-84f6a8b6ebc4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p5l2w\" (UID: \"5d7ce38f-f030-42d2-a486-84f6a8b6ebc4\") " pod="openshift-multus/multus-additional-cni-plugins-p5l2w"
Jul 07 11:40:14.211333 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.210068    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17e097fb-7012-4f55-a0c1-8cc356fc093d-host\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.211333 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.210099    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/245c8810-46ea-4bd3-97d8-34e8b93e0312-socket-dir\") pod \"aws-ebs-csi-driver-node-rl9lc\" (UID: \"245c8810-46ea-4bd3-97d8-34e8b93e0312\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl9lc"
Jul 07 11:40:14.211333 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.210128    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-run-ovn\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.211333 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.210283    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-host-slash\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.211333 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.210362    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-host-run-netns\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.211333 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.210429    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/17e097fb-7012-4f55-a0c1-8cc356fc093d-etc-sysconfig\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.211333 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.209983    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-host-run-ovn-kubernetes\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.211333 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.210543    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-env-overrides\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.211333 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.210557    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5d7ce38f-f030-42d2-a486-84f6a8b6ebc4-cni-binary-copy\") pod \"multus-additional-cni-plugins-p5l2w\" (UID: \"5d7ce38f-f030-42d2-a486-84f6a8b6ebc4\") " pod="openshift-multus/multus-additional-cni-plugins-p5l2w"
Jul 07 11:40:14.211333 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.210613    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b7d6bea-562a-423c-bd1c-4b12eb5103ef-host\") pod \"node-ca-gpk8v\" (UID: \"7b7d6bea-562a-423c-bd1c-4b12eb5103ef\") " pod="openshift-image-registry/node-ca-gpk8v"
Jul 07 11:40:14.211333 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.210657    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-run-systemd\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.211333 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.210686    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5d7ce38f-f030-42d2-a486-84f6a8b6ebc4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p5l2w\" (UID: \"5d7ce38f-f030-42d2-a486-84f6a8b6ebc4\") " pod="openshift-multus/multus-additional-cni-plugins-p5l2w"
Jul 07 11:40:14.212221 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.210702    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d7ce38f-f030-42d2-a486-84f6a8b6ebc4-system-cni-dir\") pod \"multus-additional-cni-plugins-p5l2w\" (UID: \"5d7ce38f-f030-42d2-a486-84f6a8b6ebc4\") " pod="openshift-multus/multus-additional-cni-plugins-p5l2w"
Jul 07 11:40:14.212221 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.210740    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1937f0c-df84-4f64-b4be-678b7e871e8c-host-slash\") pod \"iptables-alerter-rmjgp\" (UID: \"f1937f0c-df84-4f64-b4be-678b7e871e8c\") " pod="openshift-network-operator/iptables-alerter-rmjgp"
Jul 07 11:40:14.212221 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.210790    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/04b0713d-434f-4a0a-af55-d93a2a9e9804-hosts-file\") pod \"node-resolver-r47kl\" (UID: \"04b0713d-434f-4a0a-af55-d93a2a9e9804\") " pod="openshift-dns/node-resolver-r47kl"
Jul 07 11:40:14.212221 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.211289    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-ovnkube-config\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.212221 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.211359    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17e097fb-7012-4f55-a0c1-8cc356fc093d-sys\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.212221 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.211492    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/17e097fb-7012-4f55-a0c1-8cc356fc093d-lib-modules\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.212221 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.211774    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-host-kubelet\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.212221 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.211901    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/245c8810-46ea-4bd3-97d8-34e8b93e0312-etc-selinux\") pod \"aws-ebs-csi-driver-node-rl9lc\" (UID: \"245c8810-46ea-4bd3-97d8-34e8b93e0312\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl9lc"
Jul 07 11:40:14.212221 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.211935    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.212221 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.211998    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-host-cni-bin\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.212221 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.212011    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/245c8810-46ea-4bd3-97d8-34e8b93e0312-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rl9lc\" (UID: \"245c8810-46ea-4bd3-97d8-34e8b93e0312\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl9lc"
Jul 07 11:40:14.212221 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.212127    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-host-cni-netd\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.212221 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.212166    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/17e097fb-7012-4f55-a0c1-8cc356fc093d-etc-modprobe-d\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.212221 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.212183    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/17e097fb-7012-4f55-a0c1-8cc356fc093d-run\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.212221 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.212226    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/245c8810-46ea-4bd3-97d8-34e8b93e0312-device-dir\") pod \"aws-ebs-csi-driver-node-rl9lc\" (UID: \"245c8810-46ea-4bd3-97d8-34e8b93e0312\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl9lc"
Jul 07 11:40:14.213006 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.212313    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17e097fb-7012-4f55-a0c1-8cc356fc093d-tmp\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.213006 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.212648    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-ovnkube-script-lib\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.213006 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.212693    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5d7ce38f-f030-42d2-a486-84f6a8b6ebc4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p5l2w\" (UID: \"5d7ce38f-f030-42d2-a486-84f6a8b6ebc4\") " pod="openshift-multus/multus-additional-cni-plugins-p5l2w"
Jul 07 11:40:14.214275 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.214042    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/17e097fb-7012-4f55-a0c1-8cc356fc093d-etc-tuned\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.214823 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.214776    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-ovn-node-metrics-cert\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.227525 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.227500    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8l9x\" (UniqueName: \"kubernetes.io/projected/7b7d6bea-562a-423c-bd1c-4b12eb5103ef-kube-api-access-v8l9x\") pod \"node-ca-gpk8v\" (UID: \"7b7d6bea-562a-423c-bd1c-4b12eb5103ef\") " pod="openshift-image-registry/node-ca-gpk8v"
Jul 07 11:40:14.228490 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.228459    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz4rg\" (UniqueName: \"kubernetes.io/projected/5d7ce38f-f030-42d2-a486-84f6a8b6ebc4-kube-api-access-vz4rg\") pod \"multus-additional-cni-plugins-p5l2w\" (UID: \"5d7ce38f-f030-42d2-a486-84f6a8b6ebc4\") " pod="openshift-multus/multus-additional-cni-plugins-p5l2w"
Jul 07 11:40:14.229279 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.229254    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgcnd\" (UniqueName: \"kubernetes.io/projected/17e097fb-7012-4f55-a0c1-8cc356fc093d-kube-api-access-cgcnd\") pod \"tuned-n49lw\" (UID: \"17e097fb-7012-4f55-a0c1-8cc356fc093d\") " pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.240252 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.240221    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlq6p\" (UniqueName: \"kubernetes.io/projected/04b0713d-434f-4a0a-af55-d93a2a9e9804-kube-api-access-xlq6p\") pod \"node-resolver-r47kl\" (UID: \"04b0713d-434f-4a0a-af55-d93a2a9e9804\") " pod="openshift-dns/node-resolver-r47kl"
Jul 07 11:40:14.242350 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.242307    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m4dm\" (UniqueName: \"kubernetes.io/projected/4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801-kube-api-access-7m4dm\") pod \"ovnkube-node-kf48j\" (UID: \"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.242515 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.242496    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfjlr\" (UniqueName: \"kubernetes.io/projected/f1937f0c-df84-4f64-b4be-678b7e871e8c-kube-api-access-zfjlr\") pod \"iptables-alerter-rmjgp\" (UID: \"f1937f0c-df84-4f64-b4be-678b7e871e8c\") " pod="openshift-network-operator/iptables-alerter-rmjgp"
Jul 07 11:40:14.243668 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.243645    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67bvn\" (UniqueName: \"kubernetes.io/projected/245c8810-46ea-4bd3-97d8-34e8b93e0312-kube-api-access-67bvn\") pod \"aws-ebs-csi-driver-node-rl9lc\" (UID: \"245c8810-46ea-4bd3-97d8-34e8b93e0312\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl9lc"
Jul 07 11:40:14.298757 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.298729    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9j6l9"
Jul 07 11:40:14.318257 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.318233    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8c7wn"
Jul 07 11:40:14.325486 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.325462    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl9lc"
Jul 07 11:40:14.333611 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.333591    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:14.343608 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.343589    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p5l2w"
Jul 07 11:40:14.351437 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.351415    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rmjgp"
Jul 07 11:40:14.361262 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.361235    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n49lw"
Jul 07 11:40:14.374352 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.374266    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r47kl"
Jul 07 11:40:14.382095 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.382064    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gpk8v"
Jul 07 11:40:14.611424 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.611389    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd7d46ac-c068-44b2-adca-77ddc1da152e-metrics-certs\") pod \"network-metrics-daemon-mt9fg\" (UID: \"fd7d46ac-c068-44b2-adca-77ddc1da152e\") " pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:14.611604 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:14.611569    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:14.611674 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:14.611646    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd7d46ac-c068-44b2-adca-77ddc1da152e-metrics-certs podName:fd7d46ac-c068-44b2-adca-77ddc1da152e nodeName:}" failed. No retries permitted until 2026-07-07 11:40:15.611624683 +0000 UTC m=+5.958838286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd7d46ac-c068-44b2-adca-77ddc1da152e-metrics-certs") pod "network-metrics-daemon-mt9fg" (UID: "fd7d46ac-c068-44b2-adca-77ddc1da152e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:14.711770 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.711732    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9dzb\" (UniqueName: \"kubernetes.io/projected/9f060945-568c-484f-affd-2bbaa59faa07-kube-api-access-c9dzb\") pod \"network-check-target-86xkz\" (UID: \"9f060945-568c-484f-affd-2bbaa59faa07\") " pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:40:14.711943 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:14.711884    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 11:40:14.711943 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:14.711908    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 11:40:14.711943 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:14.711921    2581 projected.go:196] Error preparing data for projected volume kube-api-access-c9dzb for pod openshift-network-diagnostics/network-check-target-86xkz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:14.712132 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:14.711977    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f060945-568c-484f-affd-2bbaa59faa07-kube-api-access-c9dzb podName:9f060945-568c-484f-affd-2bbaa59faa07 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:15.711958772 +0000 UTC m=+6.059172383 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-c9dzb" (UniqueName: "kubernetes.io/projected/9f060945-568c-484f-affd-2bbaa59faa07-kube-api-access-c9dzb") pod "network-check-target-86xkz" (UID: "9f060945-568c-484f-affd-2bbaa59faa07") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:14.712132 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:14.711989    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:14.712132 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:14.711885    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/140115cd-5270-4373-b7eb-3db479036e81-original-pull-secret\") pod \"global-pull-secret-syncer-v289k\" (UID: \"140115cd-5270-4373-b7eb-3db479036e81\") " pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:14.712132 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:14.712066    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/140115cd-5270-4373-b7eb-3db479036e81-original-pull-secret podName:140115cd-5270-4373-b7eb-3db479036e81 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:15.712049306 +0000 UTC m=+6.059262911 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/140115cd-5270-4373-b7eb-3db479036e81-original-pull-secret") pod "global-pull-secret-syncer-v289k" (UID: "140115cd-5270-4373-b7eb-3db479036e81") : object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:15.056923 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:15.056884    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 11:35:14 +0000 UTC" deadline="2027-12-07 07:46:30.202089344 +0000 UTC"
Jul 07 11:40:15.056923 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:15.056918    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12428h6m15.145176286s"
Jul 07 11:40:15.166614 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:15.166585    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:15.166796 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:15.166723    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt9fg" podUID="fd7d46ac-c068-44b2-adca-77ddc1da152e"
Jul 07 11:40:15.260062 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:15.260033    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b7d6bea_562a_423c_bd1c_4b12eb5103ef.slice/crio-2d325534ebadf30cdb6a0ed68259fc32c301e88560e2e580e1a23d41e099162f WatchSource:0}: Error finding container 2d325534ebadf30cdb6a0ed68259fc32c301e88560e2e580e1a23d41e099162f: Status 404 returned error can't find the container with id 2d325534ebadf30cdb6a0ed68259fc32c301e88560e2e580e1a23d41e099162f
Jul 07 11:40:15.262405 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:15.262364    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d0cbd46_ebd5_4b15_9588_4c6ae7ddd801.slice/crio-ef69dd1eddd5fc55951e9f944d11830f50e9019034b24656903281dcde4802fd WatchSource:0}: Error finding container ef69dd1eddd5fc55951e9f944d11830f50e9019034b24656903281dcde4802fd: Status 404 returned error can't find the container with id ef69dd1eddd5fc55951e9f944d11830f50e9019034b24656903281dcde4802fd
Jul 07 11:40:15.267467 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:15.267383    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod245c8810_46ea_4bd3_97d8_34e8b93e0312.slice/crio-8311598491dabca5601f9cefd8ace70455d1f06d7192c64d78812edcb4377fc3 WatchSource:0}: Error finding container 8311598491dabca5601f9cefd8ace70455d1f06d7192c64d78812edcb4377fc3: Status 404 returned error can't find the container with id 8311598491dabca5601f9cefd8ace70455d1f06d7192c64d78812edcb4377fc3
Jul 07 11:40:15.268486 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:15.268464    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e097fb_7012_4f55_a0c1_8cc356fc093d.slice/crio-6a6b7383ec8a8ad025aa971f91e16158498154084cd96a3a48d78f45671e8ee1 WatchSource:0}: Error finding container 6a6b7383ec8a8ad025aa971f91e16158498154084cd96a3a48d78f45671e8ee1: Status 404 returned error can't find the container with id 6a6b7383ec8a8ad025aa971f91e16158498154084cd96a3a48d78f45671e8ee1
Jul 07 11:40:15.270955 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:40:15.269523    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70d0c362_29de_4978_b557_ce4181aabb9a.slice/crio-7e91104846e30098c13cb88c801599f3aa0290f7b2b40aa1bae4b22c66e94e57 WatchSource:0}: Error finding container 7e91104846e30098c13cb88c801599f3aa0290f7b2b40aa1bae4b22c66e94e57: Status 404 returned error can't find the container with id 7e91104846e30098c13cb88c801599f3aa0290f7b2b40aa1bae4b22c66e94e57
Jul 07 11:40:15.618505 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:15.618257    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd7d46ac-c068-44b2-adca-77ddc1da152e-metrics-certs\") pod \"network-metrics-daemon-mt9fg\" (UID: \"fd7d46ac-c068-44b2-adca-77ddc1da152e\") " pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:15.618621 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:15.618417    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:15.618621 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:15.618583    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd7d46ac-c068-44b2-adca-77ddc1da152e-metrics-certs podName:fd7d46ac-c068-44b2-adca-77ddc1da152e nodeName:}" failed. No retries permitted until 2026-07-07 11:40:17.618566566 +0000 UTC m=+7.965780173 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd7d46ac-c068-44b2-adca-77ddc1da152e-metrics-certs") pod "network-metrics-daemon-mt9fg" (UID: "fd7d46ac-c068-44b2-adca-77ddc1da152e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:15.720323 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:15.719434    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/140115cd-5270-4373-b7eb-3db479036e81-original-pull-secret\") pod \"global-pull-secret-syncer-v289k\" (UID: \"140115cd-5270-4373-b7eb-3db479036e81\") " pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:15.720323 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:15.719538    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9dzb\" (UniqueName: \"kubernetes.io/projected/9f060945-568c-484f-affd-2bbaa59faa07-kube-api-access-c9dzb\") pod \"network-check-target-86xkz\" (UID: \"9f060945-568c-484f-affd-2bbaa59faa07\") " pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:40:15.720323 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:15.719704    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 11:40:15.720323 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:15.719725    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 11:40:15.720323 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:15.719739    2581 projected.go:196] Error preparing data for projected volume kube-api-access-c9dzb for pod openshift-network-diagnostics/network-check-target-86xkz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:15.720323 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:15.719797    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f060945-568c-484f-affd-2bbaa59faa07-kube-api-access-c9dzb podName:9f060945-568c-484f-affd-2bbaa59faa07 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:17.719779286 +0000 UTC m=+8.066992893 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-c9dzb" (UniqueName: "kubernetes.io/projected/9f060945-568c-484f-affd-2bbaa59faa07-kube-api-access-c9dzb") pod "network-check-target-86xkz" (UID: "9f060945-568c-484f-affd-2bbaa59faa07") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:15.720323 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:15.720230    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:15.720323 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:15.720286    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/140115cd-5270-4373-b7eb-3db479036e81-original-pull-secret podName:140115cd-5270-4373-b7eb-3db479036e81 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:17.720269728 +0000 UTC m=+8.067483335 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/140115cd-5270-4373-b7eb-3db479036e81-original-pull-secret") pod "global-pull-secret-syncer-v289k" (UID: "140115cd-5270-4373-b7eb-3db479036e81") : object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:16.057899 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:16.057776    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 11:35:14 +0000 UTC" deadline="2028-04-21 22:00:08.235869566 +0000 UTC"
Jul 07 11:40:16.057899 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:16.057845    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15706h19m52.178028434s"
Jul 07 11:40:16.168664 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:16.168631    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:40:16.168880 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:16.168764    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86xkz" podUID="9f060945-568c-484f-affd-2bbaa59faa07"
Jul 07 11:40:16.169379 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:16.169202    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:16.169379 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:16.169301    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v289k" podUID="140115cd-5270-4373-b7eb-3db479036e81"
Jul 07 11:40:16.182428 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:16.182355    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8c7wn" event={"ID":"f1745e00-65cf-4d21-9fd1-4ff342a9bf55","Type":"ContainerStarted","Data":"8d84ce93f1b27d84240f26d107a0132fb4f4346cf0d800dce8406d8f6a4ee321"}
Jul 07 11:40:16.186153 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:16.186050    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl9lc" event={"ID":"245c8810-46ea-4bd3-97d8-34e8b93e0312","Type":"ContainerStarted","Data":"8311598491dabca5601f9cefd8ace70455d1f06d7192c64d78812edcb4377fc3"}
Jul 07 11:40:16.193713 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:16.193614    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kf48j" event={"ID":"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801","Type":"ContainerStarted","Data":"ef69dd1eddd5fc55951e9f944d11830f50e9019034b24656903281dcde4802fd"}
Jul 07 11:40:16.219766 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:16.217309    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gpk8v" event={"ID":"7b7d6bea-562a-423c-bd1c-4b12eb5103ef","Type":"ContainerStarted","Data":"2d325534ebadf30cdb6a0ed68259fc32c301e88560e2e580e1a23d41e099162f"}
Jul 07 11:40:16.223160 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:16.221980    2581 generic.go:356] "Generic (PLEG): container finished" podID="30f4a1734807f8863906bb6425570933" containerID="1ba9393e8b50c7b12bd63f54d96adcc75059ded1842e577c0a9e60258afb8bfb" exitCode=0
Jul 07 11:40:16.223160 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:16.222057    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-183.ec2.internal" event={"ID":"30f4a1734807f8863906bb6425570933","Type":"ContainerDied","Data":"1ba9393e8b50c7b12bd63f54d96adcc75059ded1842e577c0a9e60258afb8bfb"}
Jul 07 11:40:16.230412 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:16.230346    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r47kl" event={"ID":"04b0713d-434f-4a0a-af55-d93a2a9e9804","Type":"ContainerStarted","Data":"e9f7f292df208cc4986bb81d9487e0cd87a6ae44c8844b5dacaaf06ce0f500b4"}
Jul 07 11:40:16.258944 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:16.258916    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p5l2w" event={"ID":"5d7ce38f-f030-42d2-a486-84f6a8b6ebc4","Type":"ContainerStarted","Data":"3c4b8515cfe18f2cdfe3b703263327a8972cd224d5d6ca32a0226d1d7297e011"}
Jul 07 11:40:16.265879 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:16.265833    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rmjgp" event={"ID":"f1937f0c-df84-4f64-b4be-678b7e871e8c","Type":"ContainerStarted","Data":"eab86b7dca67880db65e3e163bcfea78f2a862b3b903138ed04b3985d62e108b"}
Jul 07 11:40:16.294123 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:16.294079    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-183.ec2.internal" event={"ID":"f35e0bd4cbf1174e802ed0ff417ef039","Type":"ContainerStarted","Data":"f37a5537d964baf8d00296465bd30af7e64f7d01a754255001054204b974e496"}
Jul 07 11:40:16.305937 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:16.305912    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n49lw" event={"ID":"17e097fb-7012-4f55-a0c1-8cc356fc093d","Type":"ContainerStarted","Data":"6a6b7383ec8a8ad025aa971f91e16158498154084cd96a3a48d78f45671e8ee1"}
Jul 07 11:40:16.311378 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:16.311322    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9j6l9" event={"ID":"70d0c362-29de-4978-b557-ce4181aabb9a","Type":"ContainerStarted","Data":"7e91104846e30098c13cb88c801599f3aa0290f7b2b40aa1bae4b22c66e94e57"}
Jul 07 11:40:17.166171 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:17.166138    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:17.166662 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:17.166291    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt9fg" podUID="fd7d46ac-c068-44b2-adca-77ddc1da152e"
Jul 07 11:40:17.351548 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:17.348231    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-183.ec2.internal" event={"ID":"30f4a1734807f8863906bb6425570933","Type":"ContainerStarted","Data":"399bc2b589f5ab3741294d0492a319d5101e03924b9adedd0ab11760c0f4d52e"}
Jul 07 11:40:17.368346 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:17.368287    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-183.ec2.internal" podStartSLOduration=4.368267992 podStartE2EDuration="4.368267992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:40:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:40:17.367894372 +0000 UTC m=+7.715108095" watchObservedRunningTime="2026-07-07 11:40:17.368267992 +0000 UTC m=+7.715481620"
Jul 07 11:40:17.368737 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:17.368704    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-183.ec2.internal" podStartSLOduration=4.36869642 podStartE2EDuration="4.36869642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:40:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:40:16.317564588 +0000 UTC m=+6.664778216" watchObservedRunningTime="2026-07-07 11:40:17.36869642 +0000 UTC m=+7.715910049"
Jul 07 11:40:17.632755 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:17.632618    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd7d46ac-c068-44b2-adca-77ddc1da152e-metrics-certs\") pod \"network-metrics-daemon-mt9fg\" (UID: \"fd7d46ac-c068-44b2-adca-77ddc1da152e\") " pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:17.632920 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:17.632818    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:17.632920 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:17.632897    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd7d46ac-c068-44b2-adca-77ddc1da152e-metrics-certs podName:fd7d46ac-c068-44b2-adca-77ddc1da152e nodeName:}" failed. No retries permitted until 2026-07-07 11:40:21.632875503 +0000 UTC m=+11.980089129 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd7d46ac-c068-44b2-adca-77ddc1da152e-metrics-certs") pod "network-metrics-daemon-mt9fg" (UID: "fd7d46ac-c068-44b2-adca-77ddc1da152e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:17.734696 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:17.733901    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9dzb\" (UniqueName: \"kubernetes.io/projected/9f060945-568c-484f-affd-2bbaa59faa07-kube-api-access-c9dzb\") pod \"network-check-target-86xkz\" (UID: \"9f060945-568c-484f-affd-2bbaa59faa07\") " pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:40:17.734696 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:17.733954    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/140115cd-5270-4373-b7eb-3db479036e81-original-pull-secret\") pod \"global-pull-secret-syncer-v289k\" (UID: \"140115cd-5270-4373-b7eb-3db479036e81\") " pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:17.734696 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:17.734090    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:17.734696 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:17.734151    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/140115cd-5270-4373-b7eb-3db479036e81-original-pull-secret podName:140115cd-5270-4373-b7eb-3db479036e81 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:21.734133516 +0000 UTC m=+12.081347122 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/140115cd-5270-4373-b7eb-3db479036e81-original-pull-secret") pod "global-pull-secret-syncer-v289k" (UID: "140115cd-5270-4373-b7eb-3db479036e81") : object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:17.734696 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:17.734572    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 11:40:17.734696 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:17.734590    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 11:40:17.734696 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:17.734604    2581 projected.go:196] Error preparing data for projected volume kube-api-access-c9dzb for pod openshift-network-diagnostics/network-check-target-86xkz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:17.734696 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:17.734649    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f060945-568c-484f-affd-2bbaa59faa07-kube-api-access-c9dzb podName:9f060945-568c-484f-affd-2bbaa59faa07 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:21.734633802 +0000 UTC m=+12.081847410 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-c9dzb" (UniqueName: "kubernetes.io/projected/9f060945-568c-484f-affd-2bbaa59faa07-kube-api-access-c9dzb") pod "network-check-target-86xkz" (UID: "9f060945-568c-484f-affd-2bbaa59faa07") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:18.169771 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:18.169001    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:40:18.169771 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:18.169157    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86xkz" podUID="9f060945-568c-484f-affd-2bbaa59faa07"
Jul 07 11:40:18.169771 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:18.169555    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:18.169771 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:18.169651    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v289k" podUID="140115cd-5270-4373-b7eb-3db479036e81"
Jul 07 11:40:19.166226 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:19.166170    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:19.166424 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:19.166319    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt9fg" podUID="fd7d46ac-c068-44b2-adca-77ddc1da152e"
Jul 07 11:40:20.167416 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:20.167382    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:20.167933 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:20.167525    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v289k" podUID="140115cd-5270-4373-b7eb-3db479036e81"
Jul 07 11:40:20.168075 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:20.168052    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:40:20.168189 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:20.168172    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86xkz" podUID="9f060945-568c-484f-affd-2bbaa59faa07"
Jul 07 11:40:21.166974 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:21.166372    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:21.166974 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:21.166524    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt9fg" podUID="fd7d46ac-c068-44b2-adca-77ddc1da152e"
Jul 07 11:40:21.669552 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:21.669509    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd7d46ac-c068-44b2-adca-77ddc1da152e-metrics-certs\") pod \"network-metrics-daemon-mt9fg\" (UID: \"fd7d46ac-c068-44b2-adca-77ddc1da152e\") " pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:21.670075 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:21.669693    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:21.670075 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:21.669759    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd7d46ac-c068-44b2-adca-77ddc1da152e-metrics-certs podName:fd7d46ac-c068-44b2-adca-77ddc1da152e nodeName:}" failed. No retries permitted until 2026-07-07 11:40:29.669740198 +0000 UTC m=+20.016953807 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd7d46ac-c068-44b2-adca-77ddc1da152e-metrics-certs") pod "network-metrics-daemon-mt9fg" (UID: "fd7d46ac-c068-44b2-adca-77ddc1da152e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:21.770396 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:21.770351    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9dzb\" (UniqueName: \"kubernetes.io/projected/9f060945-568c-484f-affd-2bbaa59faa07-kube-api-access-c9dzb\") pod \"network-check-target-86xkz\" (UID: \"9f060945-568c-484f-affd-2bbaa59faa07\") " pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:40:21.770571 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:21.770408    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/140115cd-5270-4373-b7eb-3db479036e81-original-pull-secret\") pod \"global-pull-secret-syncer-v289k\" (UID: \"140115cd-5270-4373-b7eb-3db479036e81\") " pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:21.770571 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:21.770537    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:21.770709 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:21.770603    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/140115cd-5270-4373-b7eb-3db479036e81-original-pull-secret podName:140115cd-5270-4373-b7eb-3db479036e81 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:29.770585416 +0000 UTC m=+20.117799027 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/140115cd-5270-4373-b7eb-3db479036e81-original-pull-secret") pod "global-pull-secret-syncer-v289k" (UID: "140115cd-5270-4373-b7eb-3db479036e81") : object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:21.770709 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:21.770615    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 11:40:21.770709 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:21.770641    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 11:40:21.770709 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:21.770655    2581 projected.go:196] Error preparing data for projected volume kube-api-access-c9dzb for pod openshift-network-diagnostics/network-check-target-86xkz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:21.770921 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:21.770713    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f060945-568c-484f-affd-2bbaa59faa07-kube-api-access-c9dzb podName:9f060945-568c-484f-affd-2bbaa59faa07 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:29.770695544 +0000 UTC m=+20.117909152 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-c9dzb" (UniqueName: "kubernetes.io/projected/9f060945-568c-484f-affd-2bbaa59faa07-kube-api-access-c9dzb") pod "network-check-target-86xkz" (UID: "9f060945-568c-484f-affd-2bbaa59faa07") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:22.168561 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:22.167848    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:40:22.168561 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:22.167973    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86xkz" podUID="9f060945-568c-484f-affd-2bbaa59faa07"
Jul 07 11:40:22.168561 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:22.168371    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:22.168561 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:22.168472    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v289k" podUID="140115cd-5270-4373-b7eb-3db479036e81"
Jul 07 11:40:23.166592 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:23.166417    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:23.166592 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:23.166554    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt9fg" podUID="fd7d46ac-c068-44b2-adca-77ddc1da152e"
Jul 07 11:40:24.166342 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:24.166257    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:40:24.166563 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:24.166380    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86xkz" podUID="9f060945-568c-484f-affd-2bbaa59faa07"
Jul 07 11:40:24.167218 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:24.167063    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:24.167218 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:24.167175    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v289k" podUID="140115cd-5270-4373-b7eb-3db479036e81"
Jul 07 11:40:25.167072 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:25.167044    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:25.167232 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:25.167171    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt9fg" podUID="fd7d46ac-c068-44b2-adca-77ddc1da152e"
Jul 07 11:40:26.166902 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:26.166867    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:40:26.166902 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:26.166894    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:26.167138 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:26.166984    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86xkz" podUID="9f060945-568c-484f-affd-2bbaa59faa07"
Jul 07 11:40:26.167138 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:26.167102    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v289k" podUID="140115cd-5270-4373-b7eb-3db479036e81"
Jul 07 11:40:27.166722 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:27.166690    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:27.167197 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:27.166804    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt9fg" podUID="fd7d46ac-c068-44b2-adca-77ddc1da152e"
Jul 07 11:40:28.166668 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:28.166631    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:40:28.166824 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:28.166791    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86xkz" podUID="9f060945-568c-484f-affd-2bbaa59faa07"
Jul 07 11:40:28.167165 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:28.166844    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:28.167165 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:28.166959    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v289k" podUID="140115cd-5270-4373-b7eb-3db479036e81"
Jul 07 11:40:29.166109 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:29.166080    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:29.166339 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:29.166211    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt9fg" podUID="fd7d46ac-c068-44b2-adca-77ddc1da152e"
Jul 07 11:40:29.738017 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:29.737987    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd7d46ac-c068-44b2-adca-77ddc1da152e-metrics-certs\") pod \"network-metrics-daemon-mt9fg\" (UID: \"fd7d46ac-c068-44b2-adca-77ddc1da152e\") " pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:29.738525 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:29.738121    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:29.738525 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:29.738185    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd7d46ac-c068-44b2-adca-77ddc1da152e-metrics-certs podName:fd7d46ac-c068-44b2-adca-77ddc1da152e nodeName:}" failed. No retries permitted until 2026-07-07 11:40:45.738165882 +0000 UTC m=+36.085379484 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd7d46ac-c068-44b2-adca-77ddc1da152e-metrics-certs") pod "network-metrics-daemon-mt9fg" (UID: "fd7d46ac-c068-44b2-adca-77ddc1da152e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:29.839075 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:29.839047    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9dzb\" (UniqueName: \"kubernetes.io/projected/9f060945-568c-484f-affd-2bbaa59faa07-kube-api-access-c9dzb\") pod \"network-check-target-86xkz\" (UID: \"9f060945-568c-484f-affd-2bbaa59faa07\") " pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:40:29.839245 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:29.839092    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/140115cd-5270-4373-b7eb-3db479036e81-original-pull-secret\") pod \"global-pull-secret-syncer-v289k\" (UID: \"140115cd-5270-4373-b7eb-3db479036e81\") " pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:29.839299 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:29.839260    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:29.839345 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:29.839322    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/140115cd-5270-4373-b7eb-3db479036e81-original-pull-secret podName:140115cd-5270-4373-b7eb-3db479036e81 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:45.839304805 +0000 UTC m=+36.186518422 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/140115cd-5270-4373-b7eb-3db479036e81-original-pull-secret") pod "global-pull-secret-syncer-v289k" (UID: "140115cd-5270-4373-b7eb-3db479036e81") : object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:29.839441 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:29.839261    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 11:40:29.839441 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:29.839374    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 11:40:29.839441 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:29.839388    2581 projected.go:196] Error preparing data for projected volume kube-api-access-c9dzb for pod openshift-network-diagnostics/network-check-target-86xkz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:29.839560 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:29.839450    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f060945-568c-484f-affd-2bbaa59faa07-kube-api-access-c9dzb podName:9f060945-568c-484f-affd-2bbaa59faa07 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:45.839435197 +0000 UTC m=+36.186648798 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-c9dzb" (UniqueName: "kubernetes.io/projected/9f060945-568c-484f-affd-2bbaa59faa07-kube-api-access-c9dzb") pod "network-check-target-86xkz" (UID: "9f060945-568c-484f-affd-2bbaa59faa07") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:30.167056 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:30.166953    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:40:30.167212 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:30.166961    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:30.167287 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:30.167103    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86xkz" podUID="9f060945-568c-484f-affd-2bbaa59faa07"
Jul 07 11:40:30.167287 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:30.167227    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v289k" podUID="140115cd-5270-4373-b7eb-3db479036e81"
Jul 07 11:40:31.166295 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:31.166223    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:31.166668 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:31.166331    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt9fg" podUID="fd7d46ac-c068-44b2-adca-77ddc1da152e"
Jul 07 11:40:32.167386 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:32.167208    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:32.167386 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:32.167316    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v289k" podUID="140115cd-5270-4373-b7eb-3db479036e81"
Jul 07 11:40:32.167386 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:32.167373    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:40:32.167948 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:32.167482    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86xkz" podUID="9f060945-568c-484f-affd-2bbaa59faa07"
Jul 07 11:40:33.167024 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:33.166977    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:33.167616 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:33.167122    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt9fg" podUID="fd7d46ac-c068-44b2-adca-77ddc1da152e"
Jul 07 11:40:34.167029 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:34.166977    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:40:34.167201 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:34.166977    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:34.167201 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:34.167105    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86xkz" podUID="9f060945-568c-484f-affd-2bbaa59faa07"
Jul 07 11:40:34.167301 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:34.167205    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v289k" podUID="140115cd-5270-4373-b7eb-3db479036e81"
Jul 07 11:40:35.166447 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:35.166187    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:35.166954 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:35.166528    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt9fg" podUID="fd7d46ac-c068-44b2-adca-77ddc1da152e"
Jul 07 11:40:35.380095 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:35.380027    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl9lc" event={"ID":"245c8810-46ea-4bd3-97d8-34e8b93e0312","Type":"ContainerStarted","Data":"54967f30bc09d34072205a00bb2786553bdb18bd94a7721a001ccc9886b3b1f9"}
Jul 07 11:40:35.383404 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:35.383385    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kf48j_4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801/ovn-acl-logging/0.log"
Jul 07 11:40:35.383848 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:35.383791    2581 generic.go:356] "Generic (PLEG): container finished" podID="4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801" containerID="8c95c56236f6d54698bae3dceecbef68eb2cdf6a135bab1477656ccf2d2faf87" exitCode=1
Jul 07 11:40:35.383995 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:35.383865    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kf48j" event={"ID":"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801","Type":"ContainerStarted","Data":"d3b9c4ddddfbc2594897b93c74a420537fff9ed48f4bed750299f7bd33c44d9b"}
Jul 07 11:40:35.383995 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:35.383953    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kf48j" event={"ID":"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801","Type":"ContainerStarted","Data":"c74a78f8df6d3f136de902d2fd7585e40691bb685dc816fe12dd2345f371792b"}
Jul 07 11:40:35.384185 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:35.384002    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kf48j" event={"ID":"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801","Type":"ContainerStarted","Data":"6f1e4405aebfbdf5a0fd1b63f9cbc66cec7d4823776d9067ad8dfc77b6e6f661"}
Jul 07 11:40:35.384185 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:35.384072    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kf48j" event={"ID":"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801","Type":"ContainerStarted","Data":"872fb42e9a23fe4c09ddbb9bc62ec53feaa3d216748f04f0a911899fdb074de5"}
Jul 07 11:40:35.384185 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:35.384122    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kf48j" event={"ID":"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801","Type":"ContainerDied","Data":"8c95c56236f6d54698bae3dceecbef68eb2cdf6a135bab1477656ccf2d2faf87"}
Jul 07 11:40:35.384185 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:35.384171    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kf48j" event={"ID":"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801","Type":"ContainerStarted","Data":"538dea2fd64afc0e1e7fc13fe2e245071fcf0256b4a46291e9c484ffe7b95a1e"}
Jul 07 11:40:35.385437 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:35.385417    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gpk8v" event={"ID":"7b7d6bea-562a-423c-bd1c-4b12eb5103ef","Type":"ContainerStarted","Data":"a504aa8bc677ec26c853597bbaa258ed39a1b29546da22e9c34ae5d63b6d17fa"}
Jul 07 11:40:35.387624 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:35.387595    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r47kl" event={"ID":"04b0713d-434f-4a0a-af55-d93a2a9e9804","Type":"ContainerStarted","Data":"230a9b0ececc79110604a4e23e161c89d13903215d6e2478c2ba193db5c644df"}
Jul 07 11:40:35.389313 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:35.389274    2581 generic.go:356] "Generic (PLEG): container finished" podID="5d7ce38f-f030-42d2-a486-84f6a8b6ebc4" containerID="a3719d357b0421fa9bbb788ad64da4c66d7755e8f378104fdce42a3994cf4b2a" exitCode=0
Jul 07 11:40:35.389712 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:35.389569    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p5l2w" event={"ID":"5d7ce38f-f030-42d2-a486-84f6a8b6ebc4","Type":"ContainerDied","Data":"a3719d357b0421fa9bbb788ad64da4c66d7755e8f378104fdce42a3994cf4b2a"}
Jul 07 11:40:35.394955 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:35.394903    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n49lw" event={"ID":"17e097fb-7012-4f55-a0c1-8cc356fc093d","Type":"ContainerStarted","Data":"7af2f1f2afefc1b981eda06f0abf91bab32052703019a56a390a5cc2c707a771"}
Jul 07 11:40:35.397433 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:35.397220    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9j6l9" event={"ID":"70d0c362-29de-4978-b557-ce4181aabb9a","Type":"ContainerStarted","Data":"441f28708d4c5031fa009bb2e513131112bbbcf7982f2be8494d6773d949c6ff"}
Jul 07 11:40:35.399031 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:35.398983    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8c7wn" event={"ID":"f1745e00-65cf-4d21-9fd1-4ff342a9bf55","Type":"ContainerStarted","Data":"815c2691975c7418757bd6a84dea4a252d9ed3de54cd0d2db11c4f775f6eea16"}
Jul 07 11:40:35.404905 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:35.404859    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gpk8v" podStartSLOduration=3.080251733 podStartE2EDuration="22.404843693s" totalImagesPullingTime="19.32459196s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:40:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:40:35.404063462 +0000 UTC m=+25.751277086" watchObservedRunningTime="2026-07-07 11:40:35.404843693 +0000 UTC m=+25.752057317"
Jul 07 11:40:35.427301 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:35.427263    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9j6l9" podStartSLOduration=4.059520169 podStartE2EDuration="23.427252028s" totalImagesPullingTime="19.367731859s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:40:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:40:35.426611068 +0000 UTC m=+25.773824729" watchObservedRunningTime="2026-07-07 11:40:35.427252028 +0000 UTC m=+25.774465650"
Jul 07 11:40:35.445196 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:35.445148    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8c7wn" podStartSLOduration=6.929051062 podStartE2EDuration="22.445131373s" totalImagesPullingTime="15.516080311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:40:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:40:35.444589717 +0000 UTC m=+25.791803339" watchObservedRunningTime="2026-07-07 11:40:35.445131373 +0000 UTC m=+25.792345000"
Jul 07 11:40:35.470297 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:35.470257    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-n49lw" podStartSLOduration=3.154244763 podStartE2EDuration="22.470240754s" totalImagesPullingTime="19.315995991s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:40:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:40:35.469825054 +0000 UTC m=+25.817038677" watchObservedRunningTime="2026-07-07 11:40:35.470240754 +0000 UTC m=+25.817454380"
Jul 07 11:40:35.490442 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:35.490409    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-r47kl" podStartSLOduration=3.1778525220000002 podStartE2EDuration="22.49039844s" totalImagesPullingTime="19.312545918s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:40:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:40:35.489648221 +0000 UTC m=+25.836861845" watchObservedRunningTime="2026-07-07 11:40:35.49039844 +0000 UTC m=+25.837612062"
Jul 07 11:40:36.166657 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:36.166500    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:40:36.166657 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:36.166615    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86xkz" podUID="9f060945-568c-484f-affd-2bbaa59faa07"
Jul 07 11:40:36.167237 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:36.166705    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:36.167237 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:36.166812    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v289k" podUID="140115cd-5270-4373-b7eb-3db479036e81"
Jul 07 11:40:36.167392 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:36.167292    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 07 11:40:36.402886 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:36.402771    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rmjgp" event={"ID":"f1937f0c-df84-4f64-b4be-678b7e871e8c","Type":"ContainerStarted","Data":"238b28687c8c9f9b4ab5fdf3fabc2f523c324c195a6a4c2dfad84db4785a8d0f"}
Jul 07 11:40:36.405024 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:36.404980    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl9lc" event={"ID":"245c8810-46ea-4bd3-97d8-34e8b93e0312","Type":"ContainerStarted","Data":"c67179d8e498404e35f8044f7f288e4fffdd8feddb12aa3a3d9c7e4506c6043f"}
Jul 07 11:40:37.063152 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:37.063050    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T11:40:36.167303757Z","UUID":"e12b87fd-161e-4b3b-bbbe-d72b7c013df4","Handler":null,"Name":"","Endpoint":""}
Jul 07 11:40:37.064624 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:37.064603    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 07 11:40:37.064760 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:37.064646    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 07 11:40:37.166594 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:37.166567    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:37.166812 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:37.166667    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt9fg" podUID="fd7d46ac-c068-44b2-adca-77ddc1da152e"
Jul 07 11:40:37.704722 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:37.704643    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8c7wn"
Jul 07 11:40:37.705268 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:37.705247    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8c7wn"
Jul 07 11:40:37.726728 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:37.726684    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rmjgp" podStartSLOduration=5.415573746 podStartE2EDuration="24.726669378s" totalImagesPullingTime="19.311095632s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:40:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:40:36.440168567 +0000 UTC m=+26.787382191" watchObservedRunningTime="2026-07-07 11:40:37.726669378 +0000 UTC m=+28.073883000"
Jul 07 11:40:38.166859 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:38.166584    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:40:38.167250 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:38.166651    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:38.167250 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:38.166941    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86xkz" podUID="9f060945-568c-484f-affd-2bbaa59faa07"
Jul 07 11:40:38.167250 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:38.166974    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v289k" podUID="140115cd-5270-4373-b7eb-3db479036e81"
Jul 07 11:40:38.411382 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:38.411351    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl9lc" event={"ID":"245c8810-46ea-4bd3-97d8-34e8b93e0312","Type":"ContainerStarted","Data":"af42f5880273ff36ff0bcb0e2a72644a184d5040c97a52fe2469dbfe8b060528"}
Jul 07 11:40:38.414787 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:38.414759    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kf48j_4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801/ovn-acl-logging/0.log"
Jul 07 11:40:38.415260 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:38.415232    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kf48j" event={"ID":"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801","Type":"ContainerStarted","Data":"405e2ffe44ad7da1905342a27f50f43ac39d8dae1e49e2168c55f19891647211"}
Jul 07 11:40:38.415377 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:38.415326    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8c7wn"
Jul 07 11:40:38.415999 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:38.415978    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8c7wn"
Jul 07 11:40:38.450742 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:38.450667    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl9lc" podStartSLOduration=3.843534868 podStartE2EDuration="25.45065408s" totalImagesPullingTime="21.607119212s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:40:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:40:38.43375424 +0000 UTC m=+28.780967864" watchObservedRunningTime="2026-07-07 11:40:38.45065408 +0000 UTC m=+28.797867703"
Jul 07 11:40:39.166689 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:39.166662    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:39.166882 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:39.166801    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt9fg" podUID="fd7d46ac-c068-44b2-adca-77ddc1da152e"
Jul 07 11:40:40.166575 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:40.166338    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:40.166724 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:40.166339    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:40:40.166724 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:40.166597    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v289k" podUID="140115cd-5270-4373-b7eb-3db479036e81"
Jul 07 11:40:40.166724 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:40.166674    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86xkz" podUID="9f060945-568c-484f-affd-2bbaa59faa07"
Jul 07 11:40:40.419911 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:40.419839    2581 generic.go:356] "Generic (PLEG): container finished" podID="5d7ce38f-f030-42d2-a486-84f6a8b6ebc4" containerID="a4456b6c02026d7e71f4ad7e511c1b4f94fbea5a9a7721e23a79cce31eba10ac" exitCode=0
Jul 07 11:40:40.420666 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:40.419932    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p5l2w" event={"ID":"5d7ce38f-f030-42d2-a486-84f6a8b6ebc4","Type":"ContainerDied","Data":"a4456b6c02026d7e71f4ad7e511c1b4f94fbea5a9a7721e23a79cce31eba10ac"}
Jul 07 11:40:40.423395 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:40.423378    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kf48j_4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801/ovn-acl-logging/0.log"
Jul 07 11:40:40.423727 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:40.423707    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kf48j" event={"ID":"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801","Type":"ContainerStarted","Data":"6d32cde94caad4b48b5a722571430ebd1851db3fa5a48db1e39acb94b44d2d45"}
Jul 07 11:40:40.424128 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:40.424113    2581 scope.go:117] "RemoveContainer" containerID="8c95c56236f6d54698bae3dceecbef68eb2cdf6a135bab1477656ccf2d2faf87"
Jul 07 11:40:41.166172 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:41.166139    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:41.166378 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:41.166252    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt9fg" podUID="fd7d46ac-c068-44b2-adca-77ddc1da152e"
Jul 07 11:40:41.429894 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:41.429829    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kf48j_4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801/ovn-acl-logging/0.log"
Jul 07 11:40:41.430367 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:41.430228    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kf48j" event={"ID":"4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801","Type":"ContainerStarted","Data":"3baa9d9e86f0a97230719ebc8dfc7bc63b8b279f309a6218a4842ee0a115ab1c"}
Jul 07 11:40:41.430592 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:41.430562    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:41.430673 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:41.430633    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:41.430713 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:41.430674    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:41.445555 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:41.445529    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:41.445882 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:41.445870    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:40:41.466388 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:41.466336    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kf48j" podStartSLOduration=9.073155753 podStartE2EDuration="28.466317842s" totalImagesPullingTime="19.393162089s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:40:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:40:41.46602809 +0000 UTC m=+31.813241709" watchObservedRunningTime="2026-07-07 11:40:41.466317842 +0000 UTC m=+31.813531467"
Jul 07 11:40:41.728828 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:41.728762    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v289k"]
Jul 07 11:40:41.728931 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:41.728903    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:41.729001 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:41.728982    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v289k" podUID="140115cd-5270-4373-b7eb-3db479036e81"
Jul 07 11:40:41.729559 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:41.729536    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mt9fg"]
Jul 07 11:40:41.729645 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:41.729631    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:41.729738 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:41.729722    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt9fg" podUID="fd7d46ac-c068-44b2-adca-77ddc1da152e"
Jul 07 11:40:41.730174 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:41.730158    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-86xkz"]
Jul 07 11:40:41.730251 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:41.730240    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:40:41.730378 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:41.730319    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86xkz" podUID="9f060945-568c-484f-affd-2bbaa59faa07"
Jul 07 11:40:42.434662 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:42.434628    2581 generic.go:356] "Generic (PLEG): container finished" podID="5d7ce38f-f030-42d2-a486-84f6a8b6ebc4" containerID="6642aa3e40b5f6dc5091983240c6b8f61894267f47e5e80620aa01eb9787cf38" exitCode=0
Jul 07 11:40:42.435215 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:42.434707    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p5l2w" event={"ID":"5d7ce38f-f030-42d2-a486-84f6a8b6ebc4","Type":"ContainerDied","Data":"6642aa3e40b5f6dc5091983240c6b8f61894267f47e5e80620aa01eb9787cf38"}
Jul 07 11:40:43.166354 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:43.166327    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:43.166526 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:43.166369    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:43.166526 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:43.166366    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:40:43.166526 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:43.166458    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt9fg" podUID="fd7d46ac-c068-44b2-adca-77ddc1da152e"
Jul 07 11:40:43.166647 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:43.166572    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v289k" podUID="140115cd-5270-4373-b7eb-3db479036e81"
Jul 07 11:40:43.166647 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:43.166631    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86xkz" podUID="9f060945-568c-484f-affd-2bbaa59faa07"
Jul 07 11:40:43.813074 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:43.813052    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r47kl_04b0713d-434f-4a0a-af55-d93a2a9e9804/dns-node-resolver/0.log"
Jul 07 11:40:44.443093 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:44.443052    2581 generic.go:356] "Generic (PLEG): container finished" podID="5d7ce38f-f030-42d2-a486-84f6a8b6ebc4" containerID="7aada172cf5ba74795ea9aa73ddbf935a3a673e06b46431184a6dfeaf6a53850" exitCode=0
Jul 07 11:40:44.443202 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:44.443104    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p5l2w" event={"ID":"5d7ce38f-f030-42d2-a486-84f6a8b6ebc4","Type":"ContainerDied","Data":"7aada172cf5ba74795ea9aa73ddbf935a3a673e06b46431184a6dfeaf6a53850"}
Jul 07 11:40:44.753869 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:44.753804    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gpk8v_7b7d6bea-562a-423c-bd1c-4b12eb5103ef/node-ca/0.log"
Jul 07 11:40:45.166975 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:45.166943    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:45.166975 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:45.166977    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:45.167702 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:45.167082    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:40:45.167702 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:45.167075    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v289k" podUID="140115cd-5270-4373-b7eb-3db479036e81"
Jul 07 11:40:45.167702 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:45.167210    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt9fg" podUID="fd7d46ac-c068-44b2-adca-77ddc1da152e"
Jul 07 11:40:45.167702 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:45.167276    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86xkz" podUID="9f060945-568c-484f-affd-2bbaa59faa07"
Jul 07 11:40:45.760260 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:45.760228    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd7d46ac-c068-44b2-adca-77ddc1da152e-metrics-certs\") pod \"network-metrics-daemon-mt9fg\" (UID: \"fd7d46ac-c068-44b2-adca-77ddc1da152e\") " pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:45.760441 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:45.760337    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:45.760441 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:45.760384    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd7d46ac-c068-44b2-adca-77ddc1da152e-metrics-certs podName:fd7d46ac-c068-44b2-adca-77ddc1da152e nodeName:}" failed. No retries permitted until 2026-07-07 11:41:17.760372717 +0000 UTC m=+68.107586319 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd7d46ac-c068-44b2-adca-77ddc1da152e-metrics-certs") pod "network-metrics-daemon-mt9fg" (UID: "fd7d46ac-c068-44b2-adca-77ddc1da152e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:45.861462 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:45.861435    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9dzb\" (UniqueName: \"kubernetes.io/projected/9f060945-568c-484f-affd-2bbaa59faa07-kube-api-access-c9dzb\") pod \"network-check-target-86xkz\" (UID: \"9f060945-568c-484f-affd-2bbaa59faa07\") " pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:40:45.861616 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:45.861471    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/140115cd-5270-4373-b7eb-3db479036e81-original-pull-secret\") pod \"global-pull-secret-syncer-v289k\" (UID: \"140115cd-5270-4373-b7eb-3db479036e81\") " pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:45.861616 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:45.861576    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:45.861616 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:45.861591    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 11:40:45.861616 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:45.861607    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 11:40:45.861616 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:45.861619    2581 projected.go:196] Error preparing data for projected volume kube-api-access-c9dzb for pod openshift-network-diagnostics/network-check-target-86xkz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:45.862031 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:45.861636    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/140115cd-5270-4373-b7eb-3db479036e81-original-pull-secret podName:140115cd-5270-4373-b7eb-3db479036e81 nodeName:}" failed. No retries permitted until 2026-07-07 11:41:17.86161953 +0000 UTC m=+68.208833133 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/140115cd-5270-4373-b7eb-3db479036e81-original-pull-secret") pod "global-pull-secret-syncer-v289k" (UID: "140115cd-5270-4373-b7eb-3db479036e81") : object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:45.862031 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:45.861654    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f060945-568c-484f-affd-2bbaa59faa07-kube-api-access-c9dzb podName:9f060945-568c-484f-affd-2bbaa59faa07 nodeName:}" failed. No retries permitted until 2026-07-07 11:41:17.861643534 +0000 UTC m=+68.208857141 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-c9dzb" (UniqueName: "kubernetes.io/projected/9f060945-568c-484f-affd-2bbaa59faa07-kube-api-access-c9dzb") pod "network-check-target-86xkz" (UID: "9f060945-568c-484f-affd-2bbaa59faa07") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:47.166245 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:47.166206    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:47.166245 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:47.166230    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:40:47.166742 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:47.166230    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:47.166742 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:47.166349    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt9fg" podUID="fd7d46ac-c068-44b2-adca-77ddc1da152e"
Jul 07 11:40:47.166742 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:47.166412    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86xkz" podUID="9f060945-568c-484f-affd-2bbaa59faa07"
Jul 07 11:40:47.166742 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:47.166485    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v289k" podUID="140115cd-5270-4373-b7eb-3db479036e81"
Jul 07 11:40:49.166343 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:49.166307    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:49.166343 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:49.166326    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:40:49.166847 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:49.166399    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:49.166847 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:49.166407    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v289k" podUID="140115cd-5270-4373-b7eb-3db479036e81"
Jul 07 11:40:49.166847 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:49.166474    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86xkz" podUID="9f060945-568c-484f-affd-2bbaa59faa07"
Jul 07 11:40:49.166847 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:49.166555    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt9fg" podUID="fd7d46ac-c068-44b2-adca-77ddc1da152e"
Jul 07 11:40:49.456838 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:49.456757    2581 generic.go:356] "Generic (PLEG): container finished" podID="5d7ce38f-f030-42d2-a486-84f6a8b6ebc4" containerID="a2466ceee92323d4914075198d4520959c4f4dbc19df6cb909fe933842d0f634" exitCode=0
Jul 07 11:40:49.456838 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:49.456797    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p5l2w" event={"ID":"5d7ce38f-f030-42d2-a486-84f6a8b6ebc4","Type":"ContainerDied","Data":"a2466ceee92323d4914075198d4520959c4f4dbc19df6cb909fe933842d0f634"}
Jul 07 11:40:50.461657 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:50.461623    2581 generic.go:356] "Generic (PLEG): container finished" podID="5d7ce38f-f030-42d2-a486-84f6a8b6ebc4" containerID="74e581535e097434cbf1ac8c847225a2a735eff5a215e5b730fcb0eb52968bef" exitCode=0
Jul 07 11:40:50.461994 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:50.461671    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p5l2w" event={"ID":"5d7ce38f-f030-42d2-a486-84f6a8b6ebc4","Type":"ContainerDied","Data":"74e581535e097434cbf1ac8c847225a2a735eff5a215e5b730fcb0eb52968bef"}
Jul 07 11:40:51.166535 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:51.166356    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:40:51.166714 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:51.166357    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:51.166714 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:51.166602    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86xkz" podUID="9f060945-568c-484f-affd-2bbaa59faa07"
Jul 07 11:40:51.166714 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:51.166361    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:51.166714 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:51.166673    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v289k" podUID="140115cd-5270-4373-b7eb-3db479036e81"
Jul 07 11:40:51.166845 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:51.166741    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt9fg" podUID="fd7d46ac-c068-44b2-adca-77ddc1da152e"
Jul 07 11:40:51.466568 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:51.466487    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p5l2w" event={"ID":"5d7ce38f-f030-42d2-a486-84f6a8b6ebc4","Type":"ContainerStarted","Data":"f96f822ae434bf709e89070967479be0b29b10cd4604036162915d41cfda622e"}
Jul 07 11:40:53.166100 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:53.166067    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:53.166100 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:53.166095    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:40:53.166598 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:53.166093    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:53.166598 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:53.166170    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt9fg" podUID="fd7d46ac-c068-44b2-adca-77ddc1da152e"
Jul 07 11:40:53.166598 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:53.166266    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86xkz" podUID="9f060945-568c-484f-affd-2bbaa59faa07"
Jul 07 11:40:53.166598 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:53.166357    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v289k" podUID="140115cd-5270-4373-b7eb-3db479036e81"
Jul 07 11:40:55.167117 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:55.167076    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:40:55.167117 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:55.167107    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:55.167518 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:55.167076    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:55.167518 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:55.167194    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86xkz" podUID="9f060945-568c-484f-affd-2bbaa59faa07"
Jul 07 11:40:55.167518 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:55.167264    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt9fg" podUID="fd7d46ac-c068-44b2-adca-77ddc1da152e"
Jul 07 11:40:55.167518 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:55.167312    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v289k" podUID="140115cd-5270-4373-b7eb-3db479036e81"
Jul 07 11:40:57.166777 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:57.166749    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:57.167195 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:57.166750    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:57.167195 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:57.166756    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:40:57.167195 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:57.166929    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt9fg" podUID="fd7d46ac-c068-44b2-adca-77ddc1da152e"
Jul 07 11:40:57.167195 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:57.166985    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86xkz" podUID="9f060945-568c-484f-affd-2bbaa59faa07"
Jul 07 11:40:57.167195 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:57.166843    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v289k" podUID="140115cd-5270-4373-b7eb-3db479036e81"
Jul 07 11:40:59.166922 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:59.166892    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:40:59.167431 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:59.166891    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:40:59.167431 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:59.166991    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v289k" podUID="140115cd-5270-4373-b7eb-3db479036e81"
Jul 07 11:40:59.167431 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:40:59.166896    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:40:59.167431 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:59.167062    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86xkz" podUID="9f060945-568c-484f-affd-2bbaa59faa07"
Jul 07 11:40:59.167431 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:40:59.167161    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt9fg" podUID="fd7d46ac-c068-44b2-adca-77ddc1da152e"
Jul 07 11:41:01.167096 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:01.167062    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:41:01.167096 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:01.167082    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:41:01.167598 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:01.167062    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:41:01.167598 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:41:01.167180    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v289k" podUID="140115cd-5270-4373-b7eb-3db479036e81"
Jul 07 11:41:01.167598 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:41:01.167236    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt9fg" podUID="fd7d46ac-c068-44b2-adca-77ddc1da152e"
Jul 07 11:41:01.167598 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:41:01.167304    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86xkz" podUID="9f060945-568c-484f-affd-2bbaa59faa07"
Jul 07 11:41:03.166174 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:03.166131    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:41:03.166547 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:41:03.166233    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86xkz" podUID="9f060945-568c-484f-affd-2bbaa59faa07"
Jul 07 11:41:03.166547 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:03.166147    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:41:03.166547 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:03.166274    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:41:03.166547 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:41:03.166340    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt9fg" podUID="fd7d46ac-c068-44b2-adca-77ddc1da152e"
Jul 07 11:41:03.166547 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:41:03.166401    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v289k" podUID="140115cd-5270-4373-b7eb-3db479036e81"
Jul 07 11:41:05.166522 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:05.166493    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:41:05.166931 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:05.166594    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:41:05.166931 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:41:05.166604    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt9fg" podUID="fd7d46ac-c068-44b2-adca-77ddc1da152e"
Jul 07 11:41:05.166931 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:05.166614    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:41:05.166931 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:41:05.166672    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86xkz" podUID="9f060945-568c-484f-affd-2bbaa59faa07"
Jul 07 11:41:05.166931 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:41:05.166727    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v289k" podUID="140115cd-5270-4373-b7eb-3db479036e81"
Jul 07 11:41:07.166871 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:07.166847    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:41:07.167254 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:07.166883    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:41:07.167254 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:41:07.166944    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt9fg" podUID="fd7d46ac-c068-44b2-adca-77ddc1da152e"
Jul 07 11:41:07.167254 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:07.166978    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:41:07.167254 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:41:07.167065    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v289k" podUID="140115cd-5270-4373-b7eb-3db479036e81"
Jul 07 11:41:07.167254 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:41:07.167113    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86xkz" podUID="9f060945-568c-484f-affd-2bbaa59faa07"
Jul 07 11:41:09.167080 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:09.167044    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:41:09.167454 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:09.167052    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:41:09.167454 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:41:09.167147    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v289k" podUID="140115cd-5270-4373-b7eb-3db479036e81"
Jul 07 11:41:09.167454 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:09.167044    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:41:09.167454 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:41:09.167221    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86xkz" podUID="9f060945-568c-484f-affd-2bbaa59faa07"
Jul 07 11:41:09.167454 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:41:09.167308    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt9fg" podUID="fd7d46ac-c068-44b2-adca-77ddc1da152e"
Jul 07 11:41:11.166152 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:11.166121    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:41:11.166608 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:11.166121    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:41:11.166608 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:41:11.166236    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86xkz" podUID="9f060945-568c-484f-affd-2bbaa59faa07"
Jul 07 11:41:11.166608 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:11.166123    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:41:11.166608 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:41:11.166282    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v289k" podUID="140115cd-5270-4373-b7eb-3db479036e81"
Jul 07 11:41:11.166608 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:41:11.166352    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mt9fg" podUID="fd7d46ac-c068-44b2-adca-77ddc1da152e"
Jul 07 11:41:12.412968 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.412943    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-183.ec2.internal" event="NodeReady"
Jul 07 11:41:12.413338 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.413065    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 07 11:41:12.475198 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.475147    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-p5l2w" podStartSLOduration=29.36894847 podStartE2EDuration="59.475134387s" totalImagesPullingTime="30.106185917s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:40:13 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:40:51.528701984 +0000 UTC m=+41.875915607" watchObservedRunningTime="2026-07-07 11:41:12.475134387 +0000 UTC m=+62.822348010"
Jul 07 11:41:12.475653 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.475633    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9q5bf"]
Jul 07 11:41:12.477281 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.477268    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9q5bf"
Jul 07 11:41:12.481285 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.481268    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 07 11:41:12.481868 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.481843    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tpsgs\""
Jul 07 11:41:12.481969 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.481905    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 07 11:41:12.485844 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.485824    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hb9w7"]
Jul 07 11:41:12.487272 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.487259    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hb9w7"
Jul 07 11:41:12.490256 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.490238    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 07 11:41:12.490341 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.490311    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-z8gkc\""
Jul 07 11:41:12.490405 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.490333    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 07 11:41:12.490505 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.490487    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 07 11:41:12.497283 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.497263    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9q5bf"]
Jul 07 11:41:12.515828 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.515809    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hb9w7"]
Jul 07 11:41:12.523693 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.523676    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rf5j2"]
Jul 07 11:41:12.525410 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.525397    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rf5j2"
Jul 07 11:41:12.532950 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.532934    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 07 11:41:12.534232 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.534214    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 07 11:41:12.534319 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.534236    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 07 11:41:12.534319 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.534220    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 07 11:41:12.534319 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.534221    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vpqj4\""
Jul 07 11:41:12.547509 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.547492    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rf5j2"]
Jul 07 11:41:12.558232 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.558213    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23cf426f-2bff-437d-93a2-c004d20e6eb6-cert\") pod \"ingress-canary-hb9w7\" (UID: \"23cf426f-2bff-437d-93a2-c004d20e6eb6\") " pod="openshift-ingress-canary/ingress-canary-hb9w7"
Jul 07 11:41:12.558310 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.558237    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x464j\" (UniqueName: \"kubernetes.io/projected/23cf426f-2bff-437d-93a2-c004d20e6eb6-kube-api-access-x464j\") pod \"ingress-canary-hb9w7\" (UID: \"23cf426f-2bff-437d-93a2-c004d20e6eb6\") " pod="openshift-ingress-canary/ingress-canary-hb9w7"
Jul 07 11:41:12.558310 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.558261    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3f5839e5-63b3-422a-a575-ec979c30e759-crio-socket\") pod \"insights-runtime-extractor-rf5j2\" (UID: \"3f5839e5-63b3-422a-a575-ec979c30e759\") " pod="openshift-insights/insights-runtime-extractor-rf5j2"
Jul 07 11:41:12.558310 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.558285    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpdsp\" (UniqueName: \"kubernetes.io/projected/3f5839e5-63b3-422a-a575-ec979c30e759-kube-api-access-dpdsp\") pod \"insights-runtime-extractor-rf5j2\" (UID: \"3f5839e5-63b3-422a-a575-ec979c30e759\") " pod="openshift-insights/insights-runtime-extractor-rf5j2"
Jul 07 11:41:12.558310 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.558306    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqv7n\" (UniqueName: \"kubernetes.io/projected/8bcb4b6a-f2fd-4e77-8c2a-d5e2d7e39e3e-kube-api-access-tqv7n\") pod \"dns-default-9q5bf\" (UID: \"8bcb4b6a-f2fd-4e77-8c2a-d5e2d7e39e3e\") " pod="openshift-dns/dns-default-9q5bf"
Jul 07 11:41:12.558433 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.558341    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3f5839e5-63b3-422a-a575-ec979c30e759-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rf5j2\" (UID: \"3f5839e5-63b3-422a-a575-ec979c30e759\") " pod="openshift-insights/insights-runtime-extractor-rf5j2"
Jul 07 11:41:12.558433 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.558371    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f5839e5-63b3-422a-a575-ec979c30e759-data-volume\") pod \"insights-runtime-extractor-rf5j2\" (UID: \"3f5839e5-63b3-422a-a575-ec979c30e759\") " pod="openshift-insights/insights-runtime-extractor-rf5j2"
Jul 07 11:41:12.558433 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.558387    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3f5839e5-63b3-422a-a575-ec979c30e759-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rf5j2\" (UID: \"3f5839e5-63b3-422a-a575-ec979c30e759\") " pod="openshift-insights/insights-runtime-extractor-rf5j2"
Jul 07 11:41:12.558433 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.558406    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bcb4b6a-f2fd-4e77-8c2a-d5e2d7e39e3e-metrics-tls\") pod \"dns-default-9q5bf\" (UID: \"8bcb4b6a-f2fd-4e77-8c2a-d5e2d7e39e3e\") " pod="openshift-dns/dns-default-9q5bf"
Jul 07 11:41:12.558552 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.558459    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8bcb4b6a-f2fd-4e77-8c2a-d5e2d7e39e3e-config-volume\") pod \"dns-default-9q5bf\" (UID: \"8bcb4b6a-f2fd-4e77-8c2a-d5e2d7e39e3e\") " pod="openshift-dns/dns-default-9q5bf"
Jul 07 11:41:12.558552 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.558497    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bcb4b6a-f2fd-4e77-8c2a-d5e2d7e39e3e-tmp-dir\") pod \"dns-default-9q5bf\" (UID: \"8bcb4b6a-f2fd-4e77-8c2a-d5e2d7e39e3e\") " pod="openshift-dns/dns-default-9q5bf"
Jul 07 11:41:12.659670 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.659636    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bcb4b6a-f2fd-4e77-8c2a-d5e2d7e39e3e-metrics-tls\") pod \"dns-default-9q5bf\" (UID: \"8bcb4b6a-f2fd-4e77-8c2a-d5e2d7e39e3e\") " pod="openshift-dns/dns-default-9q5bf"
Jul 07 11:41:12.659792 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.659693    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8bcb4b6a-f2fd-4e77-8c2a-d5e2d7e39e3e-config-volume\") pod \"dns-default-9q5bf\" (UID: \"8bcb4b6a-f2fd-4e77-8c2a-d5e2d7e39e3e\") " pod="openshift-dns/dns-default-9q5bf"
Jul 07 11:41:12.659792 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.659718    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bcb4b6a-f2fd-4e77-8c2a-d5e2d7e39e3e-tmp-dir\") pod \"dns-default-9q5bf\" (UID: \"8bcb4b6a-f2fd-4e77-8c2a-d5e2d7e39e3e\") " pod="openshift-dns/dns-default-9q5bf"
Jul 07 11:41:12.659792 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.659735    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23cf426f-2bff-437d-93a2-c004d20e6eb6-cert\") pod \"ingress-canary-hb9w7\" (UID: \"23cf426f-2bff-437d-93a2-c004d20e6eb6\") " pod="openshift-ingress-canary/ingress-canary-hb9w7"
Jul 07 11:41:12.659792 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.659749    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x464j\" (UniqueName: \"kubernetes.io/projected/23cf426f-2bff-437d-93a2-c004d20e6eb6-kube-api-access-x464j\") pod \"ingress-canary-hb9w7\" (UID: \"23cf426f-2bff-437d-93a2-c004d20e6eb6\") " pod="openshift-ingress-canary/ingress-canary-hb9w7"
Jul 07 11:41:12.659792 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.659770    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3f5839e5-63b3-422a-a575-ec979c30e759-crio-socket\") pod \"insights-runtime-extractor-rf5j2\" (UID: \"3f5839e5-63b3-422a-a575-ec979c30e759\") " pod="openshift-insights/insights-runtime-extractor-rf5j2"
Jul 07 11:41:12.660048 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.659796    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpdsp\" (UniqueName: \"kubernetes.io/projected/3f5839e5-63b3-422a-a575-ec979c30e759-kube-api-access-dpdsp\") pod \"insights-runtime-extractor-rf5j2\" (UID: \"3f5839e5-63b3-422a-a575-ec979c30e759\") " pod="openshift-insights/insights-runtime-extractor-rf5j2"
Jul 07 11:41:12.660048 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.659819    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqv7n\" (UniqueName: \"kubernetes.io/projected/8bcb4b6a-f2fd-4e77-8c2a-d5e2d7e39e3e-kube-api-access-tqv7n\") pod \"dns-default-9q5bf\" (UID: \"8bcb4b6a-f2fd-4e77-8c2a-d5e2d7e39e3e\") " pod="openshift-dns/dns-default-9q5bf"
Jul 07 11:41:12.660048 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.659846    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3f5839e5-63b3-422a-a575-ec979c30e759-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rf5j2\" (UID: \"3f5839e5-63b3-422a-a575-ec979c30e759\") " pod="openshift-insights/insights-runtime-extractor-rf5j2"
Jul 07 11:41:12.660048 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.659878    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f5839e5-63b3-422a-a575-ec979c30e759-data-volume\") pod \"insights-runtime-extractor-rf5j2\" (UID: \"3f5839e5-63b3-422a-a575-ec979c30e759\") " pod="openshift-insights/insights-runtime-extractor-rf5j2"
Jul 07 11:41:12.660048 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.659900    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3f5839e5-63b3-422a-a575-ec979c30e759-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rf5j2\" (UID: \"3f5839e5-63b3-422a-a575-ec979c30e759\") " pod="openshift-insights/insights-runtime-extractor-rf5j2"
Jul 07 11:41:12.660048 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.659904    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3f5839e5-63b3-422a-a575-ec979c30e759-crio-socket\") pod \"insights-runtime-extractor-rf5j2\" (UID: \"3f5839e5-63b3-422a-a575-ec979c30e759\") " pod="openshift-insights/insights-runtime-extractor-rf5j2"
Jul 07 11:41:12.660349 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.660321    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f5839e5-63b3-422a-a575-ec979c30e759-data-volume\") pod \"insights-runtime-extractor-rf5j2\" (UID: \"3f5839e5-63b3-422a-a575-ec979c30e759\") " pod="openshift-insights/insights-runtime-extractor-rf5j2"
Jul 07 11:41:12.660405 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.660336    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bcb4b6a-f2fd-4e77-8c2a-d5e2d7e39e3e-tmp-dir\") pod \"dns-default-9q5bf\" (UID: \"8bcb4b6a-f2fd-4e77-8c2a-d5e2d7e39e3e\") " pod="openshift-dns/dns-default-9q5bf"
Jul 07 11:41:12.660648 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.660596    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3f5839e5-63b3-422a-a575-ec979c30e759-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rf5j2\" (UID: \"3f5839e5-63b3-422a-a575-ec979c30e759\") " pod="openshift-insights/insights-runtime-extractor-rf5j2"
Jul 07 11:41:12.660648 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.660608    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8bcb4b6a-f2fd-4e77-8c2a-d5e2d7e39e3e-config-volume\") pod \"dns-default-9q5bf\" (UID: \"8bcb4b6a-f2fd-4e77-8c2a-d5e2d7e39e3e\") " pod="openshift-dns/dns-default-9q5bf"
Jul 07 11:41:12.663473 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.663417    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3f5839e5-63b3-422a-a575-ec979c30e759-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rf5j2\" (UID: \"3f5839e5-63b3-422a-a575-ec979c30e759\") " pod="openshift-insights/insights-runtime-extractor-rf5j2"
Jul 07 11:41:12.663567 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.663482    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bcb4b6a-f2fd-4e77-8c2a-d5e2d7e39e3e-metrics-tls\") pod \"dns-default-9q5bf\" (UID: \"8bcb4b6a-f2fd-4e77-8c2a-d5e2d7e39e3e\") " pod="openshift-dns/dns-default-9q5bf"
Jul 07 11:41:12.663647 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.663631    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23cf426f-2bff-437d-93a2-c004d20e6eb6-cert\") pod \"ingress-canary-hb9w7\" (UID: \"23cf426f-2bff-437d-93a2-c004d20e6eb6\") " pod="openshift-ingress-canary/ingress-canary-hb9w7"
Jul 07 11:41:12.676742 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.676723    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpdsp\" (UniqueName: \"kubernetes.io/projected/3f5839e5-63b3-422a-a575-ec979c30e759-kube-api-access-dpdsp\") pod \"insights-runtime-extractor-rf5j2\" (UID: \"3f5839e5-63b3-422a-a575-ec979c30e759\") " pod="openshift-insights/insights-runtime-extractor-rf5j2"
Jul 07 11:41:12.678920 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.678900    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x464j\" (UniqueName: \"kubernetes.io/projected/23cf426f-2bff-437d-93a2-c004d20e6eb6-kube-api-access-x464j\") pod \"ingress-canary-hb9w7\" (UID: \"23cf426f-2bff-437d-93a2-c004d20e6eb6\") " pod="openshift-ingress-canary/ingress-canary-hb9w7"
Jul 07 11:41:12.684544 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.684522    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqv7n\" (UniqueName: \"kubernetes.io/projected/8bcb4b6a-f2fd-4e77-8c2a-d5e2d7e39e3e-kube-api-access-tqv7n\") pod \"dns-default-9q5bf\" (UID: \"8bcb4b6a-f2fd-4e77-8c2a-d5e2d7e39e3e\") " pod="openshift-dns/dns-default-9q5bf"
Jul 07 11:41:12.785815 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.785788    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9q5bf"
Jul 07 11:41:12.794695 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.794680    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hb9w7"
Jul 07 11:41:12.833163 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.833126    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rf5j2"
Jul 07 11:41:12.958347 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.958317    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hb9w7"]
Jul 07 11:41:12.961584 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:41:12.961548    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23cf426f_2bff_437d_93a2_c004d20e6eb6.slice/crio-1fbae1155f982722598829a7b9dd13e870b98047474fdec82978e6cb9547e2b3 WatchSource:0}: Error finding container 1fbae1155f982722598829a7b9dd13e870b98047474fdec82978e6cb9547e2b3: Status 404 returned error can't find the container with id 1fbae1155f982722598829a7b9dd13e870b98047474fdec82978e6cb9547e2b3
Jul 07 11:41:12.977567 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:12.977547    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9q5bf"]
Jul 07 11:41:12.980796 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:41:12.980767    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bcb4b6a_f2fd_4e77_8c2a_d5e2d7e39e3e.slice/crio-86ed1c1dadeed3600c865db5abda2870ce093b3edb8d87ca8d113a63ff5a86c1 WatchSource:0}: Error finding container 86ed1c1dadeed3600c865db5abda2870ce093b3edb8d87ca8d113a63ff5a86c1: Status 404 returned error can't find the container with id 86ed1c1dadeed3600c865db5abda2870ce093b3edb8d87ca8d113a63ff5a86c1
Jul 07 11:41:13.001949 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:13.001930    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rf5j2"]
Jul 07 11:41:13.004270 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:41:13.004238    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5839e5_63b3_422a_a575_ec979c30e759.slice/crio-504bf2ca594b819dc8006ac4fadd9322e0940c206b10773d0767c25829de91af WatchSource:0}: Error finding container 504bf2ca594b819dc8006ac4fadd9322e0940c206b10773d0767c25829de91af: Status 404 returned error can't find the container with id 504bf2ca594b819dc8006ac4fadd9322e0940c206b10773d0767c25829de91af
Jul 07 11:41:13.166589 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:13.166560    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:41:13.166728 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:13.166682    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:41:13.166783 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:13.166763    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:41:13.173207 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:13.172888    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 11:41:13.173207 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:13.172891    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 11:41:13.174217 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:13.174199    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-h9gmt\""
Jul 07 11:41:13.174217 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:13.174206    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dbs59\""
Jul 07 11:41:13.174367 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:13.174200    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 11:41:13.174367 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:13.174206    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 11:41:13.449718 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:13.449651    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kf48j"
Jul 07 11:41:13.512241 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:13.512204    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rf5j2" event={"ID":"3f5839e5-63b3-422a-a575-ec979c30e759","Type":"ContainerStarted","Data":"504bf2ca594b819dc8006ac4fadd9322e0940c206b10773d0767c25829de91af"}
Jul 07 11:41:13.514879 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:13.514824    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hb9w7" event={"ID":"23cf426f-2bff-437d-93a2-c004d20e6eb6","Type":"ContainerStarted","Data":"1fbae1155f982722598829a7b9dd13e870b98047474fdec82978e6cb9547e2b3"}
Jul 07 11:41:13.516325 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:13.516267    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9q5bf" event={"ID":"8bcb4b6a-f2fd-4e77-8c2a-d5e2d7e39e3e","Type":"ContainerStarted","Data":"86ed1c1dadeed3600c865db5abda2870ce093b3edb8d87ca8d113a63ff5a86c1"}
Jul 07 11:41:14.976039 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:14.975991    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-82xzg"]
Jul 07 11:41:14.978049 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:14.978003    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-82xzg"
Jul 07 11:41:14.981318 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:14.981296    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 07 11:41:14.981436 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:14.981331    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 07 11:41:14.981436 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:14.981352    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-5jlrh\""
Jul 07 11:41:14.982740 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:14.982712    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 07 11:41:14.982914 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:14.982841    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 07 11:41:14.982914 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:14.982904    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 07 11:41:14.989310 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:14.989287    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-82xzg"]
Jul 07 11:41:15.078771 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:15.078733    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2da87046-353b-4951-aa42-8d095745d164-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-82xzg\" (UID: \"2da87046-353b-4951-aa42-8d095745d164\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-82xzg"
Jul 07 11:41:15.078929 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:15.078827    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2da87046-353b-4951-aa42-8d095745d164-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-82xzg\" (UID: \"2da87046-353b-4951-aa42-8d095745d164\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-82xzg"
Jul 07 11:41:15.078929 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:15.078858    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbj2m\" (UniqueName: \"kubernetes.io/projected/2da87046-353b-4951-aa42-8d095745d164-kube-api-access-zbj2m\") pod \"prometheus-operator-567f7b8b7c-82xzg\" (UID: \"2da87046-353b-4951-aa42-8d095745d164\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-82xzg"
Jul 07 11:41:15.079057 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:15.078959    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2da87046-353b-4951-aa42-8d095745d164-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-82xzg\" (UID: \"2da87046-353b-4951-aa42-8d095745d164\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-82xzg"
Jul 07 11:41:15.180025 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:15.179969    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2da87046-353b-4951-aa42-8d095745d164-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-82xzg\" (UID: \"2da87046-353b-4951-aa42-8d095745d164\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-82xzg"
Jul 07 11:41:15.180208 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:15.180061    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbj2m\" (UniqueName: \"kubernetes.io/projected/2da87046-353b-4951-aa42-8d095745d164-kube-api-access-zbj2m\") pod \"prometheus-operator-567f7b8b7c-82xzg\" (UID: \"2da87046-353b-4951-aa42-8d095745d164\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-82xzg"
Jul 07 11:41:15.180208 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:15.180113    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2da87046-353b-4951-aa42-8d095745d164-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-82xzg\" (UID: \"2da87046-353b-4951-aa42-8d095745d164\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-82xzg"
Jul 07 11:41:15.180208 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:15.180193    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2da87046-353b-4951-aa42-8d095745d164-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-82xzg\" (UID: \"2da87046-353b-4951-aa42-8d095745d164\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-82xzg"
Jul 07 11:41:15.180926 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:15.180900    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2da87046-353b-4951-aa42-8d095745d164-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-82xzg\" (UID: \"2da87046-353b-4951-aa42-8d095745d164\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-82xzg"
Jul 07 11:41:15.182587 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:15.182562    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2da87046-353b-4951-aa42-8d095745d164-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-82xzg\" (UID: \"2da87046-353b-4951-aa42-8d095745d164\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-82xzg"
Jul 07 11:41:15.182757 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:15.182736    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2da87046-353b-4951-aa42-8d095745d164-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-82xzg\" (UID: \"2da87046-353b-4951-aa42-8d095745d164\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-82xzg"
Jul 07 11:41:15.190294 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:15.190275    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbj2m\" (UniqueName: \"kubernetes.io/projected/2da87046-353b-4951-aa42-8d095745d164-kube-api-access-zbj2m\") pod \"prometheus-operator-567f7b8b7c-82xzg\" (UID: \"2da87046-353b-4951-aa42-8d095745d164\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-82xzg"
Jul 07 11:41:15.288756 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:15.288680    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-82xzg"
Jul 07 11:41:17.799638 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:17.799602    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd7d46ac-c068-44b2-adca-77ddc1da152e-metrics-certs\") pod \"network-metrics-daemon-mt9fg\" (UID: \"fd7d46ac-c068-44b2-adca-77ddc1da152e\") " pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:41:17.802390 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:17.802356    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd7d46ac-c068-44b2-adca-77ddc1da152e-metrics-certs\") pod \"network-metrics-daemon-mt9fg\" (UID: \"fd7d46ac-c068-44b2-adca-77ddc1da152e\") " pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:41:17.901234 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:17.900386    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/140115cd-5270-4373-b7eb-3db479036e81-original-pull-secret\") pod \"global-pull-secret-syncer-v289k\" (UID: \"140115cd-5270-4373-b7eb-3db479036e81\") " pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:41:17.901234 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:17.900483    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9dzb\" (UniqueName: \"kubernetes.io/projected/9f060945-568c-484f-affd-2bbaa59faa07-kube-api-access-c9dzb\") pod \"network-check-target-86xkz\" (UID: \"9f060945-568c-484f-affd-2bbaa59faa07\") " pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:41:17.903434 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:17.903383    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9dzb\" (UniqueName: \"kubernetes.io/projected/9f060945-568c-484f-affd-2bbaa59faa07-kube-api-access-c9dzb\") pod \"network-check-target-86xkz\" (UID: \"9f060945-568c-484f-affd-2bbaa59faa07\") " pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:41:17.903837 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:17.903796    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/140115cd-5270-4373-b7eb-3db479036e81-original-pull-secret\") pod \"global-pull-secret-syncer-v289k\" (UID: \"140115cd-5270-4373-b7eb-3db479036e81\") " pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:41:17.975935 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:17.975910    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:41:17.984396 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:17.984317    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-82xzg"]
Jul 07 11:41:17.986888 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:17.986862    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v289k"
Jul 07 11:41:17.987164 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:17.987142    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mt9fg"
Jul 07 11:41:18.158376 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:18.158314    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-86xkz"]
Jul 07 11:41:18.173029 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:18.172984    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v289k"]
Jul 07 11:41:18.174141 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:41:18.174116    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod140115cd_5270_4373_b7eb_3db479036e81.slice/crio-50d0351bf31c63eb55d050710fbf75c0cfdff7cf3d56eaa144b03d41c9ab401a WatchSource:0}: Error finding container 50d0351bf31c63eb55d050710fbf75c0cfdff7cf3d56eaa144b03d41c9ab401a: Status 404 returned error can't find the container with id 50d0351bf31c63eb55d050710fbf75c0cfdff7cf3d56eaa144b03d41c9ab401a
Jul 07 11:41:18.190784 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:18.190719    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mt9fg"]
Jul 07 11:41:18.194087 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:41:18.194065    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd7d46ac_c068_44b2_adca_77ddc1da152e.slice/crio-a98ed88ae9caa06e539f8e32b2d225a7979b84f499353a6c42db6dddcd918dc1 WatchSource:0}: Error finding container a98ed88ae9caa06e539f8e32b2d225a7979b84f499353a6c42db6dddcd918dc1: Status 404 returned error can't find the container with id a98ed88ae9caa06e539f8e32b2d225a7979b84f499353a6c42db6dddcd918dc1
Jul 07 11:41:18.533785 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:18.533746    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rf5j2" event={"ID":"3f5839e5-63b3-422a-a575-ec979c30e759","Type":"ContainerStarted","Data":"4f7ec2aa52433bd5cf5786781483c379167bb406ee63b29cd9e0ed24af20e1fd"}
Jul 07 11:41:18.535227 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:18.535156    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mt9fg" event={"ID":"fd7d46ac-c068-44b2-adca-77ddc1da152e","Type":"ContainerStarted","Data":"a98ed88ae9caa06e539f8e32b2d225a7979b84f499353a6c42db6dddcd918dc1"}
Jul 07 11:41:18.537258 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:18.537234    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-82xzg" event={"ID":"2da87046-353b-4951-aa42-8d095745d164","Type":"ContainerStarted","Data":"ba053a3cc5a30c0532e663357b3e353715d0f54987fedd75ad8b499e21fdb3d7"}
Jul 07 11:41:18.541941 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:18.540940    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9q5bf" event={"ID":"8bcb4b6a-f2fd-4e77-8c2a-d5e2d7e39e3e","Type":"ContainerStarted","Data":"c5da23862d0eaeb5a3ce3b066735348df9aadc1c9a2356aa63acef34c43e985e"}
Jul 07 11:41:18.541941 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:18.540987    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9q5bf" event={"ID":"8bcb4b6a-f2fd-4e77-8c2a-d5e2d7e39e3e","Type":"ContainerStarted","Data":"03ec65714954e211286d5df68ef1140bfa316d81538797f8cb59d6d7cc92a72e"}
Jul 07 11:41:18.541941 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:18.541885    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9q5bf"
Jul 07 11:41:18.544521 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:18.544438    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v289k" event={"ID":"140115cd-5270-4373-b7eb-3db479036e81","Type":"ContainerStarted","Data":"50d0351bf31c63eb55d050710fbf75c0cfdff7cf3d56eaa144b03d41c9ab401a"}
Jul 07 11:41:18.549368 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:18.549286    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-86xkz" event={"ID":"9f060945-568c-484f-affd-2bbaa59faa07","Type":"ContainerStarted","Data":"8f964eaec4b3eeb9457335e67eeb7bb1d1e0e6f19db2c3d8352b04c600d6d468"}
Jul 07 11:41:18.553279 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:18.553254    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hb9w7" event={"ID":"23cf426f-2bff-437d-93a2-c004d20e6eb6","Type":"ContainerStarted","Data":"1ef5ec0ca0dbb95c36094588c49adb82a02bf3d41fb2b4cb0874dd54fe391534"}
Jul 07 11:41:18.564083 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:18.563129    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9q5bf" podStartSLOduration=1.7006204440000001 podStartE2EDuration="6.563111372s" totalImagesPullingTime="4.862490928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:41:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:41:18.561860201 +0000 UTC m=+68.909073825" watchObservedRunningTime="2026-07-07 11:41:18.563111372 +0000 UTC m=+68.910324996"
Jul 07 11:41:18.580602 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:18.580548    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hb9w7" podStartSLOduration=1.698188131 podStartE2EDuration="6.58053009s" totalImagesPullingTime="4.882341959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:41:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:41:18.579900491 +0000 UTC m=+68.927114116" watchObservedRunningTime="2026-07-07 11:41:18.58053009 +0000 UTC m=+68.927743711"
Jul 07 11:41:20.392004 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:20.391697    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-nhq5h"]
Jul 07 11:41:20.394031 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:20.393990    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-nhq5h"
Jul 07 11:41:20.397161 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:20.397137    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 07 11:41:20.397451 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:20.397259    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-fwffp\""
Jul 07 11:41:20.397451 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:20.397405    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 07 11:41:20.408547 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:20.408528    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-nhq5h"]
Jul 07 11:41:20.518151 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:20.518115    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr5w7\" (UniqueName: \"kubernetes.io/projected/33fb0109-522c-4a2d-ab29-985199d918ef-kube-api-access-hr5w7\") pod \"downloads-685bfb5f64-nhq5h\" (UID: \"33fb0109-522c-4a2d-ab29-985199d918ef\") " pod="openshift-console/downloads-685bfb5f64-nhq5h"
Jul 07 11:41:20.564963 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:20.564923    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rf5j2" event={"ID":"3f5839e5-63b3-422a-a575-ec979c30e759","Type":"ContainerStarted","Data":"de596797adad4b9c718ed9e782b545322bf39fd19eda60daa723afe7dd63c900"}
Jul 07 11:41:20.567125 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:20.567094    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mt9fg" event={"ID":"fd7d46ac-c068-44b2-adca-77ddc1da152e","Type":"ContainerStarted","Data":"2af2381f9e623df90b48eee40feb0921affddf09f439001393759b96b6dd99fa"}
Jul 07 11:41:20.567236 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:20.567152    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mt9fg" event={"ID":"fd7d46ac-c068-44b2-adca-77ddc1da152e","Type":"ContainerStarted","Data":"1adde46af9f1b3df622ee0126c38f438f7cad9b62b21c1b92ac165a69d8f13ca"}
Jul 07 11:41:20.571165 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:20.571069    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-82xzg" event={"ID":"2da87046-353b-4951-aa42-8d095745d164","Type":"ContainerStarted","Data":"c86cfc8e673bb8ab2dcbd1aa51b01c06387a5b3c39f0ad95c475c57d568990b4"}
Jul 07 11:41:20.571165 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:20.571122    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-82xzg" event={"ID":"2da87046-353b-4951-aa42-8d095745d164","Type":"ContainerStarted","Data":"1dc4cfb2bee106f753abe0b3ee8bdf25ad724dae56120fed79874248e879a5fa"}
Jul 07 11:41:20.595602 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:20.595555    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mt9fg" podStartSLOduration=67.021592475 podStartE2EDuration="1m8.595537602s" totalImagesPullingTime="1.573945127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:40:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:41:20.594746845 +0000 UTC m=+70.941960470" watchObservedRunningTime="2026-07-07 11:41:20.595537602 +0000 UTC m=+70.942751228"
Jul 07 11:41:20.617875 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:20.617823    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-82xzg" podStartSLOduration=4.845594783 podStartE2EDuration="6.617805633s" totalImagesPullingTime="1.77221085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:41:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:41:20.616433511 +0000 UTC m=+70.963647135" watchObservedRunningTime="2026-07-07 11:41:20.617805633 +0000 UTC m=+70.965019257"
Jul 07 11:41:20.619903 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:20.618557    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr5w7\" (UniqueName: \"kubernetes.io/projected/33fb0109-522c-4a2d-ab29-985199d918ef-kube-api-access-hr5w7\") pod \"downloads-685bfb5f64-nhq5h\" (UID: \"33fb0109-522c-4a2d-ab29-985199d918ef\") " pod="openshift-console/downloads-685bfb5f64-nhq5h"
Jul 07 11:41:20.636141 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:20.636098    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr5w7\" (UniqueName: \"kubernetes.io/projected/33fb0109-522c-4a2d-ab29-985199d918ef-kube-api-access-hr5w7\") pod \"downloads-685bfb5f64-nhq5h\" (UID: \"33fb0109-522c-4a2d-ab29-985199d918ef\") " pod="openshift-console/downloads-685bfb5f64-nhq5h"
Jul 07 11:41:20.707168 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:20.706986    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-nhq5h"
Jul 07 11:41:22.449344 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:22.449308    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fl86s"]
Jul 07 11:41:22.471804 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:22.471775    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fl86s"
Jul 07 11:41:22.474759 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:22.474734    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 07 11:41:22.475073 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:22.475046    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2p72q\""
Jul 07 11:41:22.475186 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:22.475136    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 07 11:41:22.475526 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:22.475507    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 07 11:41:22.534404 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:22.534369    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/148c135d-0b0f-45d0-af99-c4de21b03097-sys\") pod \"node-exporter-fl86s\" (UID: \"148c135d-0b0f-45d0-af99-c4de21b03097\") " pod="openshift-monitoring/node-exporter-fl86s"
Jul 07 11:41:22.534554 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:22.534409    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/148c135d-0b0f-45d0-af99-c4de21b03097-root\") pod \"node-exporter-fl86s\" (UID: \"148c135d-0b0f-45d0-af99-c4de21b03097\") " pod="openshift-monitoring/node-exporter-fl86s"
Jul 07 11:41:22.534554 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:22.534440    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8xvv\" (UniqueName: \"kubernetes.io/projected/148c135d-0b0f-45d0-af99-c4de21b03097-kube-api-access-c8xvv\") pod \"node-exporter-fl86s\" (UID: \"148c135d-0b0f-45d0-af99-c4de21b03097\") " pod="openshift-monitoring/node-exporter-fl86s"
Jul 07 11:41:22.534554 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:22.534518    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/148c135d-0b0f-45d0-af99-c4de21b03097-node-exporter-tls\") pod \"node-exporter-fl86s\" (UID: \"148c135d-0b0f-45d0-af99-c4de21b03097\") " pod="openshift-monitoring/node-exporter-fl86s"
Jul 07 11:41:22.534713 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:22.534571    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/148c135d-0b0f-45d0-af99-c4de21b03097-metrics-client-ca\") pod \"node-exporter-fl86s\" (UID: \"148c135d-0b0f-45d0-af99-c4de21b03097\") " pod="openshift-monitoring/node-exporter-fl86s"
Jul 07 11:41:22.534713 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:22.534616    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/148c135d-0b0f-45d0-af99-c4de21b03097-node-exporter-wtmp\") pod \"node-exporter-fl86s\" (UID: \"148c135d-0b0f-45d0-af99-c4de21b03097\") " pod="openshift-monitoring/node-exporter-fl86s"
Jul 07 11:41:22.534713 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:22.534646    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/148c135d-0b0f-45d0-af99-c4de21b03097-node-exporter-textfile\") pod \"node-exporter-fl86s\" (UID: \"148c135d-0b0f-45d0-af99-c4de21b03097\") " pod="openshift-monitoring/node-exporter-fl86s"
Jul 07 11:41:22.534713 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:22.534672    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/148c135d-0b0f-45d0-af99-c4de21b03097-node-exporter-accelerators-collector-config\") pod \"node-exporter-fl86s\" (UID: \"148c135d-0b0f-45d0-af99-c4de21b03097\") " pod="openshift-monitoring/node-exporter-fl86s"
Jul 07 11:41:22.534884 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:22.534715    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/148c135d-0b0f-45d0-af99-c4de21b03097-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fl86s\" (UID: \"148c135d-0b0f-45d0-af99-c4de21b03097\") " pod="openshift-monitoring/node-exporter-fl86s"
Jul 07 11:41:22.635998 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:22.635951    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/148c135d-0b0f-45d0-af99-c4de21b03097-node-exporter-tls\") pod \"node-exporter-fl86s\" (UID: \"148c135d-0b0f-45d0-af99-c4de21b03097\") " pod="openshift-monitoring/node-exporter-fl86s"
Jul 07 11:41:22.635998 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:22.635996    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/148c135d-0b0f-45d0-af99-c4de21b03097-metrics-client-ca\") pod \"node-exporter-fl86s\" (UID: \"148c135d-0b0f-45d0-af99-c4de21b03097\") " pod="openshift-monitoring/node-exporter-fl86s"
Jul 07 11:41:22.636248 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:22.636037    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/148c135d-0b0f-45d0-af99-c4de21b03097-node-exporter-wtmp\") pod \"node-exporter-fl86s\" (UID: \"148c135d-0b0f-45d0-af99-c4de21b03097\") " pod="openshift-monitoring/node-exporter-fl86s"
Jul 07 11:41:22.636248 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:22.636064    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/148c135d-0b0f-45d0-af99-c4de21b03097-node-exporter-textfile\") pod \"node-exporter-fl86s\" (UID: \"148c135d-0b0f-45d0-af99-c4de21b03097\") " pod="openshift-monitoring/node-exporter-fl86s"
Jul 07 11:41:22.636248 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:22.636089    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/148c135d-0b0f-45d0-af99-c4de21b03097-node-exporter-accelerators-collector-config\") pod \"node-exporter-fl86s\" (UID: \"148c135d-0b0f-45d0-af99-c4de21b03097\") " pod="openshift-monitoring/node-exporter-fl86s"
Jul 07 11:41:22.636248 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:22.636117    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/148c135d-0b0f-45d0-af99-c4de21b03097-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fl86s\" (UID: \"148c135d-0b0f-45d0-af99-c4de21b03097\") " pod="openshift-monitoring/node-exporter-fl86s"
Jul 07 11:41:22.636248 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:22.636152    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/148c135d-0b0f-45d0-af99-c4de21b03097-sys\") pod \"node-exporter-fl86s\" (UID: \"148c135d-0b0f-45d0-af99-c4de21b03097\") " pod="openshift-monitoring/node-exporter-fl86s"
Jul 07 11:41:22.636248 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:22.636179    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/148c135d-0b0f-45d0-af99-c4de21b03097-root\") pod \"node-exporter-fl86s\" (UID: \"148c135d-0b0f-45d0-af99-c4de21b03097\") " pod="openshift-monitoring/node-exporter-fl86s"
Jul 07 11:41:22.636248 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:22.636205    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8xvv\" (UniqueName: \"kubernetes.io/projected/148c135d-0b0f-45d0-af99-c4de21b03097-kube-api-access-c8xvv\") pod \"node-exporter-fl86s\" (UID: \"148c135d-0b0f-45d0-af99-c4de21b03097\") " pod="openshift-monitoring/node-exporter-fl86s"
Jul 07 11:41:22.636582 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:22.636304    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/148c135d-0b0f-45d0-af99-c4de21b03097-sys\") pod \"node-exporter-fl86s\" (UID: \"148c135d-0b0f-45d0-af99-c4de21b03097\") " pod="openshift-monitoring/node-exporter-fl86s"
Jul 07 11:41:22.636582 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:22.636365    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/148c135d-0b0f-45d0-af99-c4de21b03097-node-exporter-wtmp\") pod \"node-exporter-fl86s\" (UID: \"148c135d-0b0f-45d0-af99-c4de21b03097\") " pod="openshift-monitoring/node-exporter-fl86s"
Jul 07 11:41:22.636680 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:22.636664    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/148c135d-0b0f-45d0-af99-c4de21b03097-node-exporter-textfile\") pod \"node-exporter-fl86s\" (UID: \"148c135d-0b0f-45d0-af99-c4de21b03097\") " pod="openshift-monitoring/node-exporter-fl86s"
Jul 07 11:41:22.636736 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:22.636712    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/148c135d-0b0f-45d0-af99-c4de21b03097-metrics-client-ca\") pod \"node-exporter-fl86s\" (UID: \"148c135d-0b0f-45d0-af99-c4de21b03097\") " pod="openshift-monitoring/node-exporter-fl86s"
Jul 07 11:41:22.636736 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:22.636723    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/148c135d-0b0f-45d0-af99-c4de21b03097-root\") pod \"node-exporter-fl86s\" (UID: \"148c135d-0b0f-45d0-af99-c4de21b03097\") " pod="openshift-monitoring/node-exporter-fl86s"
Jul 07 11:41:22.636868 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:22.636841    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/148c135d-0b0f-45d0-af99-c4de21b03097-node-exporter-accelerators-collector-config\") pod \"node-exporter-fl86s\" (UID: \"148c135d-0b0f-45d0-af99-c4de21b03097\") " pod="openshift-monitoring/node-exporter-fl86s"
Jul 07 11:41:22.638612 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:22.638588    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/148c135d-0b0f-45d0-af99-c4de21b03097-node-exporter-tls\") pod \"node-exporter-fl86s\" (UID: \"148c135d-0b0f-45d0-af99-c4de21b03097\") " pod="openshift-monitoring/node-exporter-fl86s"
Jul 07 11:41:22.638740 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:22.638699    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/148c135d-0b0f-45d0-af99-c4de21b03097-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fl86s\" (UID: \"148c135d-0b0f-45d0-af99-c4de21b03097\") " pod="openshift-monitoring/node-exporter-fl86s"
Jul 07 11:41:22.647173 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:22.647151    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8xvv\" (UniqueName: \"kubernetes.io/projected/148c135d-0b0f-45d0-af99-c4de21b03097-kube-api-access-c8xvv\") pod \"node-exporter-fl86s\" (UID: \"148c135d-0b0f-45d0-af99-c4de21b03097\") " pod="openshift-monitoring/node-exporter-fl86s"
Jul 07 11:41:22.782625 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:22.782545    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fl86s"
Jul 07 11:41:23.626973 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.626945    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 11:41:23.631628 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.631605    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.636263 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.636237    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 07 11:41:23.636478 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.636457    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 07 11:41:23.636634 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.636600    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 07 11:41:23.639663 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.637228    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 07 11:41:23.639663 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.637297    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wxkvr\""
Jul 07 11:41:23.639663 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.637297    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 07 11:41:23.639663 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.637373    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 07 11:41:23.639663 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.637374    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 07 11:41:23.639663 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.637445    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 07 11:41:23.639663 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.637630    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 07 11:41:23.656456 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.656435    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 11:41:23.742602 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.742573    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b269716-9397-4dfd-98c0-9f02068c115d-config-out\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.742602 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.742607    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b269716-9397-4dfd-98c0-9f02068c115d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.742885 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.742624    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.742885 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.742653    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.742885 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.742717    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8b269716-9397-4dfd-98c0-9f02068c115d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.742885 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.742751    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b269716-9397-4dfd-98c0-9f02068c115d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.742885 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.742847    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-config-volume\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.742885 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.742876    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.743186 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.742909    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.743186 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.742991    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-web-config\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.743186 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.743055    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.743186 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.743104    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b269716-9397-4dfd-98c0-9f02068c115d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.743186 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.743136    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmptz\" (UniqueName: \"kubernetes.io/projected/8b269716-9397-4dfd-98c0-9f02068c115d-kube-api-access-nmptz\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.844479 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.844395    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.844479 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.844450    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b269716-9397-4dfd-98c0-9f02068c115d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.844479 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.844483    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmptz\" (UniqueName: \"kubernetes.io/projected/8b269716-9397-4dfd-98c0-9f02068c115d-kube-api-access-nmptz\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.844757 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.844516    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b269716-9397-4dfd-98c0-9f02068c115d-config-out\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.844757 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.844537    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b269716-9397-4dfd-98c0-9f02068c115d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.844757 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.844561    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.844757 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.844596    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.844757 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.844623    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8b269716-9397-4dfd-98c0-9f02068c115d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.844757 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.844645    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b269716-9397-4dfd-98c0-9f02068c115d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.844757 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.844676    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-config-volume\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.844757 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.844714    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.844757 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.844743    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.845219 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.844774    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-web-config\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.845837 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.845530    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b269716-9397-4dfd-98c0-9f02068c115d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.845837 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.845554    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b269716-9397-4dfd-98c0-9f02068c115d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.845837 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:41:23.845637    2581 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 07 11:41:23.845837 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:41:23.845705    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-secret-alertmanager-main-tls podName:8b269716-9397-4dfd-98c0-9f02068c115d nodeName:}" failed. No retries permitted until 2026-07-07 11:41:24.345686787 +0000 UTC m=+74.692900403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "8b269716-9397-4dfd-98c0-9f02068c115d") : secret "alertmanager-main-tls" not found
Jul 07 11:41:23.846324 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.845923    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8b269716-9397-4dfd-98c0-9f02068c115d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.848254 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.848223    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b269716-9397-4dfd-98c0-9f02068c115d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.848347 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.848313    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.848441 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.848422    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-web-config\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.848495 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.848448    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.848582 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.848553    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b269716-9397-4dfd-98c0-9f02068c115d-config-out\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.849456 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.849401    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-config-volume\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.849757 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.849710    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.850478 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.850443    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:23.865439 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:23.865420    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmptz\" (UniqueName: \"kubernetes.io/projected/8b269716-9397-4dfd-98c0-9f02068c115d-kube-api-access-nmptz\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:24.349918 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:24.349888    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:24.352198 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:24.352174    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:24.366971 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:41:24.366873    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod148c135d_0b0f_45d0_af99_c4de21b03097.slice/crio-dffce4a5f991d4fdace6c85f102e7a96e4128c7ce4a934fe6dc0ad26f035e533 WatchSource:0}: Error finding container dffce4a5f991d4fdace6c85f102e7a96e4128c7ce4a934fe6dc0ad26f035e533: Status 404 returned error can't find the container with id dffce4a5f991d4fdace6c85f102e7a96e4128c7ce4a934fe6dc0ad26f035e533
Jul 07 11:41:24.507576 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:24.507547    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-nhq5h"]
Jul 07 11:41:24.510472 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:41:24.510447    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33fb0109_522c_4a2d_ab29_985199d918ef.slice/crio-3ccef289928968d992d0616ec5d4bda5fb392c821313874d47ef4cbd68ad7b32 WatchSource:0}: Error finding container 3ccef289928968d992d0616ec5d4bda5fb392c821313874d47ef4cbd68ad7b32: Status 404 returned error can't find the container with id 3ccef289928968d992d0616ec5d4bda5fb392c821313874d47ef4cbd68ad7b32
Jul 07 11:41:24.545679 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:24.545658    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:41:24.588583 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:24.588547    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fl86s" event={"ID":"148c135d-0b0f-45d0-af99-c4de21b03097","Type":"ContainerStarted","Data":"dffce4a5f991d4fdace6c85f102e7a96e4128c7ce4a934fe6dc0ad26f035e533"}
Jul 07 11:41:24.591636 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:24.591103    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rf5j2" event={"ID":"3f5839e5-63b3-422a-a575-ec979c30e759","Type":"ContainerStarted","Data":"3578bdcbd77a9dda2139701c3f310be6b5fb492396a9b9c8386a1940fec865d8"}
Jul 07 11:41:24.593134 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:24.593108    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-nhq5h" event={"ID":"33fb0109-522c-4a2d-ab29-985199d918ef","Type":"ContainerStarted","Data":"3ccef289928968d992d0616ec5d4bda5fb392c821313874d47ef4cbd68ad7b32"}
Jul 07 11:41:24.594566 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:24.594543    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v289k" event={"ID":"140115cd-5270-4373-b7eb-3db479036e81","Type":"ContainerStarted","Data":"2977b320e92e0dfb35abf650a99c552ed550018fc7d6d3e4dc87494b06ed8a48"}
Jul 07 11:41:24.596772 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:24.596400    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-86xkz" event={"ID":"9f060945-568c-484f-affd-2bbaa59faa07","Type":"ContainerStarted","Data":"cc4b9a9353de98354a912a76112b3f10f2c805e5bf04da5d3aa3c4a0008cc231"}
Jul 07 11:41:24.596880 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:24.596841    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:41:24.616622 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:24.616564    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rf5j2" podStartSLOduration=1.442941144 podStartE2EDuration="12.616543679s" totalImagesPullingTime="11.173602535s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:41:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:41:24.61618303 +0000 UTC m=+74.963396666" watchObservedRunningTime="2026-07-07 11:41:24.616543679 +0000 UTC m=+74.963757302"
Jul 07 11:41:24.637088 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:24.636996    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-86xkz" podStartSLOduration=65.435314597 podStartE2EDuration="1m11.636977182s" totalImagesPullingTime="6.201662585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:40:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:41:24.635202406 +0000 UTC m=+74.982416029" watchObservedRunningTime="2026-07-07 11:41:24.636977182 +0000 UTC m=+74.984190806"
Jul 07 11:41:24.656522 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:24.656482    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-v289k" podStartSLOduration=65.462710117 podStartE2EDuration="1m11.656468677s" totalImagesPullingTime="6.19375856s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:40:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:41:24.656085072 +0000 UTC m=+75.003298695" watchObservedRunningTime="2026-07-07 11:41:24.656468677 +0000 UTC m=+75.003682278"
Jul 07 11:41:24.689569 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:41:24.689545    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b269716_9397_4dfd_98c0_9f02068c115d.slice/crio-1b4a6a6ed90dc5460f63a029d5188fa4c65b2cc13122bbf943f8fed80751c2cd WatchSource:0}: Error finding container 1b4a6a6ed90dc5460f63a029d5188fa4c65b2cc13122bbf943f8fed80751c2cd: Status 404 returned error can't find the container with id 1b4a6a6ed90dc5460f63a029d5188fa4c65b2cc13122bbf943f8fed80751c2cd
Jul 07 11:41:24.691278 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:24.691260    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 11:41:25.478438 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.478404    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-67b754fb45-58684"]
Jul 07 11:41:25.482615 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.482595    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-67b754fb45-58684"
Jul 07 11:41:25.488177 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.488153    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 07 11:41:25.488283 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.488179    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 07 11:41:25.488283 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.488259    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-37vac5u1r341m\""
Jul 07 11:41:25.488283 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.488267    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 07 11:41:25.488283 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.488179    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 07 11:41:25.488597 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.488579    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 07 11:41:25.488825 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.488653    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-8f5wf\""
Jul 07 11:41:25.497096 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.497034    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-67b754fb45-58684"]
Jul 07 11:41:25.558039 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.557976    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5273bb01-e902-4ae5-a8ea-9f2cedfcf29c-metrics-client-ca\") pod \"thanos-querier-67b754fb45-58684\" (UID: \"5273bb01-e902-4ae5-a8ea-9f2cedfcf29c\") " pod="openshift-monitoring/thanos-querier-67b754fb45-58684"
Jul 07 11:41:25.558192 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.558041    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5273bb01-e902-4ae5-a8ea-9f2cedfcf29c-secret-thanos-querier-tls\") pod \"thanos-querier-67b754fb45-58684\" (UID: \"5273bb01-e902-4ae5-a8ea-9f2cedfcf29c\") " pod="openshift-monitoring/thanos-querier-67b754fb45-58684"
Jul 07 11:41:25.558192 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.558095    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5273bb01-e902-4ae5-a8ea-9f2cedfcf29c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67b754fb45-58684\" (UID: \"5273bb01-e902-4ae5-a8ea-9f2cedfcf29c\") " pod="openshift-monitoring/thanos-querier-67b754fb45-58684"
Jul 07 11:41:25.558192 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.558167    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5273bb01-e902-4ae5-a8ea-9f2cedfcf29c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67b754fb45-58684\" (UID: \"5273bb01-e902-4ae5-a8ea-9f2cedfcf29c\") " pod="openshift-monitoring/thanos-querier-67b754fb45-58684"
Jul 07 11:41:25.558351 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.558191    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5273bb01-e902-4ae5-a8ea-9f2cedfcf29c-secret-grpc-tls\") pod \"thanos-querier-67b754fb45-58684\" (UID: \"5273bb01-e902-4ae5-a8ea-9f2cedfcf29c\") " pod="openshift-monitoring/thanos-querier-67b754fb45-58684"
Jul 07 11:41:25.558351 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.558220    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fc95\" (UniqueName: \"kubernetes.io/projected/5273bb01-e902-4ae5-a8ea-9f2cedfcf29c-kube-api-access-2fc95\") pod \"thanos-querier-67b754fb45-58684\" (UID: \"5273bb01-e902-4ae5-a8ea-9f2cedfcf29c\") " pod="openshift-monitoring/thanos-querier-67b754fb45-58684"
Jul 07 11:41:25.558351 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.558338    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5273bb01-e902-4ae5-a8ea-9f2cedfcf29c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67b754fb45-58684\" (UID: \"5273bb01-e902-4ae5-a8ea-9f2cedfcf29c\") " pod="openshift-monitoring/thanos-querier-67b754fb45-58684"
Jul 07 11:41:25.558480 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.558381    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5273bb01-e902-4ae5-a8ea-9f2cedfcf29c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67b754fb45-58684\" (UID: \"5273bb01-e902-4ae5-a8ea-9f2cedfcf29c\") " pod="openshift-monitoring/thanos-querier-67b754fb45-58684"
Jul 07 11:41:25.602350 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.602311    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b269716-9397-4dfd-98c0-9f02068c115d","Type":"ContainerStarted","Data":"1b4a6a6ed90dc5460f63a029d5188fa4c65b2cc13122bbf943f8fed80751c2cd"}
Jul 07 11:41:25.659362 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.659307    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5273bb01-e902-4ae5-a8ea-9f2cedfcf29c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67b754fb45-58684\" (UID: \"5273bb01-e902-4ae5-a8ea-9f2cedfcf29c\") " pod="openshift-monitoring/thanos-querier-67b754fb45-58684"
Jul 07 11:41:25.659362 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.659365    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5273bb01-e902-4ae5-a8ea-9f2cedfcf29c-secret-grpc-tls\") pod \"thanos-querier-67b754fb45-58684\" (UID: \"5273bb01-e902-4ae5-a8ea-9f2cedfcf29c\") " pod="openshift-monitoring/thanos-querier-67b754fb45-58684"
Jul 07 11:41:25.659945 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.659395    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fc95\" (UniqueName: \"kubernetes.io/projected/5273bb01-e902-4ae5-a8ea-9f2cedfcf29c-kube-api-access-2fc95\") pod \"thanos-querier-67b754fb45-58684\" (UID: \"5273bb01-e902-4ae5-a8ea-9f2cedfcf29c\") " pod="openshift-monitoring/thanos-querier-67b754fb45-58684"
Jul 07 11:41:25.659945 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.659431    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5273bb01-e902-4ae5-a8ea-9f2cedfcf29c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67b754fb45-58684\" (UID: \"5273bb01-e902-4ae5-a8ea-9f2cedfcf29c\") " pod="openshift-monitoring/thanos-querier-67b754fb45-58684"
Jul 07 11:41:25.659945 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.659549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5273bb01-e902-4ae5-a8ea-9f2cedfcf29c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67b754fb45-58684\" (UID: \"5273bb01-e902-4ae5-a8ea-9f2cedfcf29c\") " pod="openshift-monitoring/thanos-querier-67b754fb45-58684"
Jul 07 11:41:25.659945 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.659676    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5273bb01-e902-4ae5-a8ea-9f2cedfcf29c-metrics-client-ca\") pod \"thanos-querier-67b754fb45-58684\" (UID: \"5273bb01-e902-4ae5-a8ea-9f2cedfcf29c\") " pod="openshift-monitoring/thanos-querier-67b754fb45-58684"
Jul 07 11:41:25.659945 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.659704    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5273bb01-e902-4ae5-a8ea-9f2cedfcf29c-secret-thanos-querier-tls\") pod \"thanos-querier-67b754fb45-58684\" (UID: \"5273bb01-e902-4ae5-a8ea-9f2cedfcf29c\") " pod="openshift-monitoring/thanos-querier-67b754fb45-58684"
Jul 07 11:41:25.659945 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.659753    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5273bb01-e902-4ae5-a8ea-9f2cedfcf29c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67b754fb45-58684\" (UID: \"5273bb01-e902-4ae5-a8ea-9f2cedfcf29c\") " pod="openshift-monitoring/thanos-querier-67b754fb45-58684"
Jul 07 11:41:25.661096 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.661000    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5273bb01-e902-4ae5-a8ea-9f2cedfcf29c-metrics-client-ca\") pod \"thanos-querier-67b754fb45-58684\" (UID: \"5273bb01-e902-4ae5-a8ea-9f2cedfcf29c\") " pod="openshift-monitoring/thanos-querier-67b754fb45-58684"
Jul 07 11:41:25.663564 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.663489    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5273bb01-e902-4ae5-a8ea-9f2cedfcf29c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67b754fb45-58684\" (UID: \"5273bb01-e902-4ae5-a8ea-9f2cedfcf29c\") " pod="openshift-monitoring/thanos-querier-67b754fb45-58684"
Jul 07 11:41:25.664295 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.664249    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5273bb01-e902-4ae5-a8ea-9f2cedfcf29c-secret-thanos-querier-tls\") pod \"thanos-querier-67b754fb45-58684\" (UID: \"5273bb01-e902-4ae5-a8ea-9f2cedfcf29c\") " pod="openshift-monitoring/thanos-querier-67b754fb45-58684"
Jul 07 11:41:25.664414 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.664336    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5273bb01-e902-4ae5-a8ea-9f2cedfcf29c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67b754fb45-58684\" (UID: \"5273bb01-e902-4ae5-a8ea-9f2cedfcf29c\") " pod="openshift-monitoring/thanos-querier-67b754fb45-58684"
Jul 07 11:41:25.664895 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.664865    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5273bb01-e902-4ae5-a8ea-9f2cedfcf29c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67b754fb45-58684\" (UID: \"5273bb01-e902-4ae5-a8ea-9f2cedfcf29c\") " pod="openshift-monitoring/thanos-querier-67b754fb45-58684"
Jul 07 11:41:25.665279 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.665240    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5273bb01-e902-4ae5-a8ea-9f2cedfcf29c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67b754fb45-58684\" (UID: \"5273bb01-e902-4ae5-a8ea-9f2cedfcf29c\") " pod="openshift-monitoring/thanos-querier-67b754fb45-58684"
Jul 07 11:41:25.666692 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.666670    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5273bb01-e902-4ae5-a8ea-9f2cedfcf29c-secret-grpc-tls\") pod \"thanos-querier-67b754fb45-58684\" (UID: \"5273bb01-e902-4ae5-a8ea-9f2cedfcf29c\") " pod="openshift-monitoring/thanos-querier-67b754fb45-58684"
Jul 07 11:41:25.670825 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.670782    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fc95\" (UniqueName: \"kubernetes.io/projected/5273bb01-e902-4ae5-a8ea-9f2cedfcf29c-kube-api-access-2fc95\") pod \"thanos-querier-67b754fb45-58684\" (UID: \"5273bb01-e902-4ae5-a8ea-9f2cedfcf29c\") " pod="openshift-monitoring/thanos-querier-67b754fb45-58684"
Jul 07 11:41:25.794691 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.794665    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-67b754fb45-58684"
Jul 07 11:41:25.960929 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:25.960905    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-67b754fb45-58684"]
Jul 07 11:41:25.968150 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:41:25.968118    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5273bb01_e902_4ae5_a8ea_9f2cedfcf29c.slice/crio-69d993952a41dff0b82d6bf0be0d4e0d243492044ea5749c625811ab250b4362 WatchSource:0}: Error finding container 69d993952a41dff0b82d6bf0be0d4e0d243492044ea5749c625811ab250b4362: Status 404 returned error can't find the container with id 69d993952a41dff0b82d6bf0be0d4e0d243492044ea5749c625811ab250b4362
Jul 07 11:41:26.615875 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:26.615837    2581 generic.go:356] "Generic (PLEG): container finished" podID="148c135d-0b0f-45d0-af99-c4de21b03097" containerID="f7929ebd44a6cc937a7184d0255bf3c6cb6e2c446e57307e9ddee2936c132324" exitCode=0
Jul 07 11:41:26.616119 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:26.615927    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fl86s" event={"ID":"148c135d-0b0f-45d0-af99-c4de21b03097","Type":"ContainerDied","Data":"f7929ebd44a6cc937a7184d0255bf3c6cb6e2c446e57307e9ddee2936c132324"}
Jul 07 11:41:26.618346 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:26.618186    2581 generic.go:356] "Generic (PLEG): container finished" podID="8b269716-9397-4dfd-98c0-9f02068c115d" containerID="49d51b2ea927026197b3cca0cce9f0dc9138354ed8312876bb3d318b1e8535d6" exitCode=0
Jul 07 11:41:26.618346 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:26.618291    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b269716-9397-4dfd-98c0-9f02068c115d","Type":"ContainerDied","Data":"49d51b2ea927026197b3cca0cce9f0dc9138354ed8312876bb3d318b1e8535d6"}
Jul 07 11:41:26.619988 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:26.619696    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67b754fb45-58684" event={"ID":"5273bb01-e902-4ae5-a8ea-9f2cedfcf29c","Type":"ContainerStarted","Data":"69d993952a41dff0b82d6bf0be0d4e0d243492044ea5749c625811ab250b4362"}
Jul 07 11:41:26.831191 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:26.831153    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6877c5899b-4jnwg"]
Jul 07 11:41:26.834540 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:26.834491    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6877c5899b-4jnwg"
Jul 07 11:41:26.839128 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:26.838981    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-6q7t7\""
Jul 07 11:41:26.839128 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:26.839060    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 07 11:41:26.839128 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:26.839074    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 07 11:41:26.839355 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:26.839151    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-46nrsk3cifcqq\""
Jul 07 11:41:26.839355 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:26.839253    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 07 11:41:26.839355 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:26.839330    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 07 11:41:26.852347 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:26.852291    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6877c5899b-4jnwg"]
Jul 07 11:41:26.972494 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:26.972411    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/62e155d9-5bf1-441f-a9a6-720f883b176f-client-ca-bundle\") pod \"metrics-server-6877c5899b-4jnwg\" (UID: \"62e155d9-5bf1-441f-a9a6-720f883b176f\") " pod="openshift-monitoring/metrics-server-6877c5899b-4jnwg"
Jul 07 11:41:26.972494 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:26.972446    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbbr8\" (UniqueName: \"kubernetes.io/projected/62e155d9-5bf1-441f-a9a6-720f883b176f-kube-api-access-kbbr8\") pod \"metrics-server-6877c5899b-4jnwg\" (UID: \"62e155d9-5bf1-441f-a9a6-720f883b176f\") " pod="openshift-monitoring/metrics-server-6877c5899b-4jnwg"
Jul 07 11:41:26.972494 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:26.972476    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/62e155d9-5bf1-441f-a9a6-720f883b176f-metrics-server-audit-profiles\") pod \"metrics-server-6877c5899b-4jnwg\" (UID: \"62e155d9-5bf1-441f-a9a6-720f883b176f\") " pod="openshift-monitoring/metrics-server-6877c5899b-4jnwg"
Jul 07 11:41:26.972725 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:26.972509    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/62e155d9-5bf1-441f-a9a6-720f883b176f-audit-log\") pod \"metrics-server-6877c5899b-4jnwg\" (UID: \"62e155d9-5bf1-441f-a9a6-720f883b176f\") " pod="openshift-monitoring/metrics-server-6877c5899b-4jnwg"
Jul 07 11:41:26.972725 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:26.972538    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/62e155d9-5bf1-441f-a9a6-720f883b176f-secret-metrics-server-client-certs\") pod \"metrics-server-6877c5899b-4jnwg\" (UID: \"62e155d9-5bf1-441f-a9a6-720f883b176f\") " pod="openshift-monitoring/metrics-server-6877c5899b-4jnwg"
Jul 07 11:41:26.972725 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:26.972562    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/62e155d9-5bf1-441f-a9a6-720f883b176f-secret-metrics-server-tls\") pod \"metrics-server-6877c5899b-4jnwg\" (UID: \"62e155d9-5bf1-441f-a9a6-720f883b176f\") " pod="openshift-monitoring/metrics-server-6877c5899b-4jnwg"
Jul 07 11:41:26.972725 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:26.972656    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62e155d9-5bf1-441f-a9a6-720f883b176f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6877c5899b-4jnwg\" (UID: \"62e155d9-5bf1-441f-a9a6-720f883b176f\") " pod="openshift-monitoring/metrics-server-6877c5899b-4jnwg"
Jul 07 11:41:27.073701 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:27.073663    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/62e155d9-5bf1-441f-a9a6-720f883b176f-client-ca-bundle\") pod \"metrics-server-6877c5899b-4jnwg\" (UID: \"62e155d9-5bf1-441f-a9a6-720f883b176f\") " pod="openshift-monitoring/metrics-server-6877c5899b-4jnwg"
Jul 07 11:41:27.073861 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:27.073708    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbbr8\" (UniqueName: \"kubernetes.io/projected/62e155d9-5bf1-441f-a9a6-720f883b176f-kube-api-access-kbbr8\") pod \"metrics-server-6877c5899b-4jnwg\" (UID: \"62e155d9-5bf1-441f-a9a6-720f883b176f\") " pod="openshift-monitoring/metrics-server-6877c5899b-4jnwg"
Jul 07 11:41:27.073861 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:27.073736    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/62e155d9-5bf1-441f-a9a6-720f883b176f-metrics-server-audit-profiles\") pod \"metrics-server-6877c5899b-4jnwg\" (UID: \"62e155d9-5bf1-441f-a9a6-720f883b176f\") " pod="openshift-monitoring/metrics-server-6877c5899b-4jnwg"
Jul 07 11:41:27.073861 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:27.073766    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/62e155d9-5bf1-441f-a9a6-720f883b176f-audit-log\") pod \"metrics-server-6877c5899b-4jnwg\" (UID: \"62e155d9-5bf1-441f-a9a6-720f883b176f\") " pod="openshift-monitoring/metrics-server-6877c5899b-4jnwg"
Jul 07 11:41:27.073861 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:27.073795    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/62e155d9-5bf1-441f-a9a6-720f883b176f-secret-metrics-server-client-certs\") pod \"metrics-server-6877c5899b-4jnwg\" (UID: \"62e155d9-5bf1-441f-a9a6-720f883b176f\") " pod="openshift-monitoring/metrics-server-6877c5899b-4jnwg"
Jul 07 11:41:27.073861 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:27.073822    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/62e155d9-5bf1-441f-a9a6-720f883b176f-secret-metrics-server-tls\") pod \"metrics-server-6877c5899b-4jnwg\" (UID: \"62e155d9-5bf1-441f-a9a6-720f883b176f\") " pod="openshift-monitoring/metrics-server-6877c5899b-4jnwg"
Jul 07 11:41:27.074139 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:27.073871    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62e155d9-5bf1-441f-a9a6-720f883b176f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6877c5899b-4jnwg\" (UID: \"62e155d9-5bf1-441f-a9a6-720f883b176f\") " pod="openshift-monitoring/metrics-server-6877c5899b-4jnwg"
Jul 07 11:41:27.074382 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:27.074356    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/62e155d9-5bf1-441f-a9a6-720f883b176f-audit-log\") pod \"metrics-server-6877c5899b-4jnwg\" (UID: \"62e155d9-5bf1-441f-a9a6-720f883b176f\") " pod="openshift-monitoring/metrics-server-6877c5899b-4jnwg"
Jul 07 11:41:27.074698 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:27.074646    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62e155d9-5bf1-441f-a9a6-720f883b176f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6877c5899b-4jnwg\" (UID: \"62e155d9-5bf1-441f-a9a6-720f883b176f\") " pod="openshift-monitoring/metrics-server-6877c5899b-4jnwg"
Jul 07 11:41:27.075287 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:27.075262    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/62e155d9-5bf1-441f-a9a6-720f883b176f-metrics-server-audit-profiles\") pod \"metrics-server-6877c5899b-4jnwg\" (UID: \"62e155d9-5bf1-441f-a9a6-720f883b176f\") " pod="openshift-monitoring/metrics-server-6877c5899b-4jnwg"
Jul 07 11:41:27.076963 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:27.076922    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/62e155d9-5bf1-441f-a9a6-720f883b176f-secret-metrics-server-client-certs\") pod \"metrics-server-6877c5899b-4jnwg\" (UID: \"62e155d9-5bf1-441f-a9a6-720f883b176f\") " pod="openshift-monitoring/metrics-server-6877c5899b-4jnwg"
Jul 07 11:41:27.077833 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:27.077796    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/62e155d9-5bf1-441f-a9a6-720f883b176f-client-ca-bundle\") pod \"metrics-server-6877c5899b-4jnwg\" (UID: \"62e155d9-5bf1-441f-a9a6-720f883b176f\") " pod="openshift-monitoring/metrics-server-6877c5899b-4jnwg"
Jul 07 11:41:27.078401 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:27.078357    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/62e155d9-5bf1-441f-a9a6-720f883b176f-secret-metrics-server-tls\") pod \"metrics-server-6877c5899b-4jnwg\" (UID: \"62e155d9-5bf1-441f-a9a6-720f883b176f\") " pod="openshift-monitoring/metrics-server-6877c5899b-4jnwg"
Jul 07 11:41:27.103072 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:27.103028    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbbr8\" (UniqueName: \"kubernetes.io/projected/62e155d9-5bf1-441f-a9a6-720f883b176f-kube-api-access-kbbr8\") pod \"metrics-server-6877c5899b-4jnwg\" (UID: \"62e155d9-5bf1-441f-a9a6-720f883b176f\") " pod="openshift-monitoring/metrics-server-6877c5899b-4jnwg"
Jul 07 11:41:27.146508 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:27.146241    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6877c5899b-4jnwg"
Jul 07 11:41:27.328004 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:27.327981    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6877c5899b-4jnwg"]
Jul 07 11:41:27.330325 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:41:27.330290    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e155d9_5bf1_441f_a9a6_720f883b176f.slice/crio-e5e3baf5c549fea0cb23f8bd92a0fb2fe8a98c6832bdd83df36e4e8a5b10323d WatchSource:0}: Error finding container e5e3baf5c549fea0cb23f8bd92a0fb2fe8a98c6832bdd83df36e4e8a5b10323d: Status 404 returned error can't find the container with id e5e3baf5c549fea0cb23f8bd92a0fb2fe8a98c6832bdd83df36e4e8a5b10323d
Jul 07 11:41:27.627416 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:27.627380    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fl86s" event={"ID":"148c135d-0b0f-45d0-af99-c4de21b03097","Type":"ContainerStarted","Data":"9317e65b8657ea5e3fd19bb9661e8bf258f56625cba4c3eb12d91ff526a963a2"}
Jul 07 11:41:27.627601 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:27.627448    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fl86s" event={"ID":"148c135d-0b0f-45d0-af99-c4de21b03097","Type":"ContainerStarted","Data":"e77ee98013c74773bc41f795a896c36e7da577ef83004c2aee01bce5b7bce64d"}
Jul 07 11:41:27.629748 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:27.629710    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6877c5899b-4jnwg" event={"ID":"62e155d9-5bf1-441f-a9a6-720f883b176f","Type":"ContainerStarted","Data":"e5e3baf5c549fea0cb23f8bd92a0fb2fe8a98c6832bdd83df36e4e8a5b10323d"}
Jul 07 11:41:27.653693 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:27.653641    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fl86s" podStartSLOduration=4.218349306 podStartE2EDuration="5.653621166s" totalImagesPullingTime="1.350792424s" totalInitContainerRuntime="84.479436ms" isStatefulPod=true podCreationTimestamp="2026-07-07 11:41:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:41:27.652256793 +0000 UTC m=+77.999470417" watchObservedRunningTime="2026-07-07 11:41:27.653621166 +0000 UTC m=+78.000834795"
Jul 07 11:41:28.838700 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.838673    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 11:41:28.850511 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.850479    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:28.855329 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.854883    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 07 11:41:28.855329 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.855170    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 07 11:41:28.856409 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.856227    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 07 11:41:28.859971 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.858618    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 07 11:41:28.859971 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.858901    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 07 11:41:28.859971 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.859171    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cm6d30d9a76h0\""
Jul 07 11:41:28.859971 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.859396    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-pv2bn\""
Jul 07 11:41:28.859971 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.859570    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 07 11:41:28.859971 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.859758    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 07 11:41:28.860345 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.859991    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 07 11:41:28.872534 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.869337    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 07 11:41:28.872534 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.871628    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 11:41:28.872534 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.871932    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 07 11:41:28.872534 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.872156    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 07 11:41:28.872534 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.872319    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 07 11:41:28.873353 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.873237    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 07 11:41:28.877740 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.877293    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 07 11:41:28.991410 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.991178    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:28.991410 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.991219    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:28.991410 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.991252    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-web-config\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:28.991410 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.991280    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-config\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:28.991410 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.991305    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:28.991410 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.991336    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-config-out\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:28.991410 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.991362    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:28.991410 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.991385    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:28.991974 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.991438    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:28.991974 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.991493    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:28.991974 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.991522    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:28.991974 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.991547    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:28.991974 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.991569    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:28.991974 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.991601    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:28.991974 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.991625    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:28.991974 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.991655    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:28.991974 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.991680    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:28.991974 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.991707    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:28.992604 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.992030    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:28.992604 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:28.992059    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-454v5\" (UniqueName: \"kubernetes.io/projected/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-kube-api-access-454v5\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.095486 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.092704    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.095486 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.092746    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.095486 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.092775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.095486 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.092802    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.095486 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.092843    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.095486 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.092867    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.095486 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.092899    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.095486 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.092930    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.095486 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.092958    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.095486 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.092993    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.095486 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.093036    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-454v5\" (UniqueName: \"kubernetes.io/projected/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-kube-api-access-454v5\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.095486 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.093084    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.095486 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.093108    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.095486 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.093138    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-web-config\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.095486 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.093171    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-config\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.095486 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.093198    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.095486 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.093255    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-config-out\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.096408 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.093279    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.096408 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.093302    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.096408 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.093336    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.096989 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.096960    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.097604 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.097409    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.101851 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.098236    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.101851 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.100439    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.101851 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.100941    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.101851 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.101487    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.101851 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.101643    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.101851 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.101651    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-config\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.102381 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.101859    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.102536 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.102453    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.102984 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.102941    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.104484 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.104441    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-config-out\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.104585 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.104539    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.104680 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.104657    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.105135 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.105082    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.106675 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.105908    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.106675 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.106354    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-web-config\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.107308 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.107261    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.108617 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.108589    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.110991 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.110924    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-454v5\" (UniqueName: \"kubernetes.io/projected/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-kube-api-access-454v5\") pod \"prometheus-k8s-0\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.195169 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.194931    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:29.383362 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.383254    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 11:41:29.387410 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:41:29.387381    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa9bb81_dc87_4b74_be29_587ab84bf1e9.slice/crio-083ead2cab0a8e8a90a996103ac4e5cb5e2c20eb56e9ec8c280a91e107232a33 WatchSource:0}: Error finding container 083ead2cab0a8e8a90a996103ac4e5cb5e2c20eb56e9ec8c280a91e107232a33: Status 404 returned error can't find the container with id 083ead2cab0a8e8a90a996103ac4e5cb5e2c20eb56e9ec8c280a91e107232a33
Jul 07 11:41:29.573291 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.573256    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9q5bf"
Jul 07 11:41:29.646526 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.646386    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67b754fb45-58684" event={"ID":"5273bb01-e902-4ae5-a8ea-9f2cedfcf29c","Type":"ContainerStarted","Data":"6e8f271b6cd1d48f817f37f9a3099a180147758c9fa2ed95347806949904bca8"}
Jul 07 11:41:29.646526 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.646470    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67b754fb45-58684" event={"ID":"5273bb01-e902-4ae5-a8ea-9f2cedfcf29c","Type":"ContainerStarted","Data":"f7fd0e6d0effb73daaab0eb11c187a0fbbc5cf9c9740bc886a4d847f4be0a895"}
Jul 07 11:41:29.646526 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.646511    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67b754fb45-58684" event={"ID":"5273bb01-e902-4ae5-a8ea-9f2cedfcf29c","Type":"ContainerStarted","Data":"409195709c8f05f9e5375fbe766ff8759cdc8a475816f46253b5a42b81e0cfc3"}
Jul 07 11:41:29.648501 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.648469    2581 generic.go:356] "Generic (PLEG): container finished" podID="aaa9bb81-dc87-4b74-be29-587ab84bf1e9" containerID="fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce" exitCode=0
Jul 07 11:41:29.648634 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.648555    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aaa9bb81-dc87-4b74-be29-587ab84bf1e9","Type":"ContainerDied","Data":"fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce"}
Jul 07 11:41:29.648706 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.648635    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aaa9bb81-dc87-4b74-be29-587ab84bf1e9","Type":"ContainerStarted","Data":"083ead2cab0a8e8a90a996103ac4e5cb5e2c20eb56e9ec8c280a91e107232a33"}
Jul 07 11:41:29.652886 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.652862    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b269716-9397-4dfd-98c0-9f02068c115d","Type":"ContainerStarted","Data":"c93578c664db89c19be0817d871e3c4812a1e4658ab84fe08c412d3326f2a04a"}
Jul 07 11:41:29.652972 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.652928    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b269716-9397-4dfd-98c0-9f02068c115d","Type":"ContainerStarted","Data":"1cceb6e37766a703dcc9d934f4606f654e8ac2874a6819910eb156f4fb2289a1"}
Jul 07 11:41:29.653036 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.652967    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b269716-9397-4dfd-98c0-9f02068c115d","Type":"ContainerStarted","Data":"2a54923fc25a7a6dd1249cf2f375cdfe64c1eb5d1e09432464e04ff064ba0e38"}
Jul 07 11:41:29.653036 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.653031    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b269716-9397-4dfd-98c0-9f02068c115d","Type":"ContainerStarted","Data":"d37d15f4831299c673931a1cfc4b0c4616ee481152376def63392692b2db1ff9"}
Jul 07 11:41:29.653118 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:29.653070    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b269716-9397-4dfd-98c0-9f02068c115d","Type":"ContainerStarted","Data":"cf13b380fdc40fe2bf8f3c9b615f7f906295b9b9ccf2e71fae776a21067b6b2c"}
Jul 07 11:41:30.660580 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:30.659820    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6877c5899b-4jnwg" event={"ID":"62e155d9-5bf1-441f-a9a6-720f883b176f","Type":"ContainerStarted","Data":"3dae44bdd6bb91363676d59e33f22830116c192d4635bd0bc3d87fee5629bb84"}
Jul 07 11:41:30.664546 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:30.664479    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67b754fb45-58684" event={"ID":"5273bb01-e902-4ae5-a8ea-9f2cedfcf29c","Type":"ContainerStarted","Data":"0cbfaf3260220f78ebdc83787a21655f0157863a2e0cbbe91e850155d653b214"}
Jul 07 11:41:30.664664 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:30.664547    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67b754fb45-58684" event={"ID":"5273bb01-e902-4ae5-a8ea-9f2cedfcf29c","Type":"ContainerStarted","Data":"63af357712144cfc34feb2f26deeb41076355d67adb0d64a46920398d59d789b"}
Jul 07 11:41:30.664664 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:30.664590    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67b754fb45-58684" event={"ID":"5273bb01-e902-4ae5-a8ea-9f2cedfcf29c","Type":"ContainerStarted","Data":"f0c1ffe029749210d4f6d6101e9ecbef63b3b2be2706660e0212bdaea439a5ef"}
Jul 07 11:41:30.664774 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:30.664730    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-67b754fb45-58684"
Jul 07 11:41:30.668898 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:30.668875    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b269716-9397-4dfd-98c0-9f02068c115d","Type":"ContainerStarted","Data":"4073217113df8717613e00992b1d5ca869409b10d546d16fc1eafef902c0a910"}
Jul 07 11:41:30.689488 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:30.689209    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6877c5899b-4jnwg" podStartSLOduration=1.842023512 podStartE2EDuration="4.689196541s" totalImagesPullingTime="2.847173029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:41:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:41:30.688832108 +0000 UTC m=+81.036045731" watchObservedRunningTime="2026-07-07 11:41:30.689196541 +0000 UTC m=+81.036410164"
Jul 07 11:41:30.792236 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:30.791843    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-67b754fb45-58684" podStartSLOduration=1.8944824040000001 podStartE2EDuration="5.791820218s" totalImagesPullingTime="3.897337814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:41:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:41:30.790087243 +0000 UTC m=+81.137300868" watchObservedRunningTime="2026-07-07 11:41:30.791820218 +0000 UTC m=+81.139033819"
Jul 07 11:41:30.792534 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:30.792500    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.635755423 podStartE2EDuration="7.792487485s" totalImagesPullingTime="4.143458977s" totalInitContainerRuntime="13.273085ms" isStatefulPod=false podCreationTimestamp="2026-07-07 11:41:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:41:30.739921643 +0000 UTC m=+81.087135267" watchObservedRunningTime="2026-07-07 11:41:30.792487485 +0000 UTC m=+81.139701108"
Jul 07 11:41:33.686506 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:33.686475    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aaa9bb81-dc87-4b74-be29-587ab84bf1e9","Type":"ContainerStarted","Data":"c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6"}
Jul 07 11:41:33.686816 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:33.686563    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aaa9bb81-dc87-4b74-be29-587ab84bf1e9","Type":"ContainerStarted","Data":"f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb"}
Jul 07 11:41:33.686816 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:33.686607    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aaa9bb81-dc87-4b74-be29-587ab84bf1e9","Type":"ContainerStarted","Data":"f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5"}
Jul 07 11:41:33.686816 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:33.686647    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aaa9bb81-dc87-4b74-be29-587ab84bf1e9","Type":"ContainerStarted","Data":"f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1"}
Jul 07 11:41:33.686816 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:33.686687    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aaa9bb81-dc87-4b74-be29-587ab84bf1e9","Type":"ContainerStarted","Data":"9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4"}
Jul 07 11:41:34.694879 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:34.694840    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aaa9bb81-dc87-4b74-be29-587ab84bf1e9","Type":"ContainerStarted","Data":"16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7"}
Jul 07 11:41:34.746274 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:34.746214    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.125137961 podStartE2EDuration="6.7461976s" totalImagesPullingTime="3.602641202s" totalInitContainerRuntime="18.418437ms" isStatefulPod=false podCreationTimestamp="2026-07-07 11:41:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:41:34.743435708 +0000 UTC m=+85.090649331" watchObservedRunningTime="2026-07-07 11:41:34.7461976 +0000 UTC m=+85.093411222"
Jul 07 11:41:36.681036 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:36.680992    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-67b754fb45-58684"
Jul 07 11:41:39.196192 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:39.196154    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:41:43.734966 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:43.734922    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-nhq5h" event={"ID":"33fb0109-522c-4a2d-ab29-985199d918ef","Type":"ContainerStarted","Data":"7fcfac64109c6b950750361fceb9895d5fcc25b1bb72e23558d8a1ccd8e1bf45"}
Jul 07 11:41:43.735482 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:43.735183    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-nhq5h"
Jul 07 11:41:43.750810 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:43.750780    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-nhq5h"
Jul 07 11:41:43.764124 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:43.764066    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-nhq5h" podStartSLOduration=5.337902161 podStartE2EDuration="23.764046503s" totalImagesPullingTime="18.426144342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:41:43.761455747 +0000 UTC m=+94.108669370" watchObservedRunningTime="2026-07-07 11:41:43.764046503 +0000 UTC m=+94.111260130"
Jul 07 11:41:47.147559 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:47.147528    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6877c5899b-4jnwg"
Jul 07 11:41:47.147995 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:47.147597    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6877c5899b-4jnwg"
Jul 07 11:41:56.622058 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:41:56.622027    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-86xkz"
Jul 07 11:42:07.152602 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:07.152573    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6877c5899b-4jnwg"
Jul 07 11:42:07.157520 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:07.157491    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6877c5899b-4jnwg"
Jul 07 11:42:29.195566 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:29.195527    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:29.268328 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:29.268303    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:29.929065 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:29.928975    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:42.878316 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:42.878283    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 11:42:42.878811 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:42.878568    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8b269716-9397-4dfd-98c0-9f02068c115d" containerName="alertmanager" containerID="cri-o://cf13b380fdc40fe2bf8f3c9b615f7f906295b9b9ccf2e71fae776a21067b6b2c" gracePeriod=120
Jul 07 11:42:42.878811 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:42.878640    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8b269716-9397-4dfd-98c0-9f02068c115d" containerName="kube-rbac-proxy-metric" containerID="cri-o://c93578c664db89c19be0817d871e3c4812a1e4658ab84fe08c412d3326f2a04a" gracePeriod=120
Jul 07 11:42:42.878811 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:42.878686    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8b269716-9397-4dfd-98c0-9f02068c115d" containerName="kube-rbac-proxy-web" containerID="cri-o://2a54923fc25a7a6dd1249cf2f375cdfe64c1eb5d1e09432464e04ff064ba0e38" gracePeriod=120
Jul 07 11:42:42.878811 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:42.878714    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8b269716-9397-4dfd-98c0-9f02068c115d" containerName="prom-label-proxy" containerID="cri-o://4073217113df8717613e00992b1d5ca869409b10d546d16fc1eafef902c0a910" gracePeriod=120
Jul 07 11:42:42.878811 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:42.878748    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8b269716-9397-4dfd-98c0-9f02068c115d" containerName="kube-rbac-proxy" containerID="cri-o://1cceb6e37766a703dcc9d934f4606f654e8ac2874a6819910eb156f4fb2289a1" gracePeriod=120
Jul 07 11:42:42.878811 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:42.878715    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8b269716-9397-4dfd-98c0-9f02068c115d" containerName="config-reloader" containerID="cri-o://d37d15f4831299c673931a1cfc4b0c4616ee481152376def63392692b2db1ff9" gracePeriod=120
Jul 07 11:42:43.971245 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:43.971205    2581 generic.go:356] "Generic (PLEG): container finished" podID="8b269716-9397-4dfd-98c0-9f02068c115d" containerID="4073217113df8717613e00992b1d5ca869409b10d546d16fc1eafef902c0a910" exitCode=0
Jul 07 11:42:43.971245 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:43.971238    2581 generic.go:356] "Generic (PLEG): container finished" podID="8b269716-9397-4dfd-98c0-9f02068c115d" containerID="1cceb6e37766a703dcc9d934f4606f654e8ac2874a6819910eb156f4fb2289a1" exitCode=0
Jul 07 11:42:43.971623 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:43.971248    2581 generic.go:356] "Generic (PLEG): container finished" podID="8b269716-9397-4dfd-98c0-9f02068c115d" containerID="d37d15f4831299c673931a1cfc4b0c4616ee481152376def63392692b2db1ff9" exitCode=0
Jul 07 11:42:43.971623 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:43.971257    2581 generic.go:356] "Generic (PLEG): container finished" podID="8b269716-9397-4dfd-98c0-9f02068c115d" containerID="cf13b380fdc40fe2bf8f3c9b615f7f906295b9b9ccf2e71fae776a21067b6b2c" exitCode=0
Jul 07 11:42:43.971623 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:43.971255    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b269716-9397-4dfd-98c0-9f02068c115d","Type":"ContainerDied","Data":"4073217113df8717613e00992b1d5ca869409b10d546d16fc1eafef902c0a910"}
Jul 07 11:42:43.971623 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:43.971387    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b269716-9397-4dfd-98c0-9f02068c115d","Type":"ContainerDied","Data":"1cceb6e37766a703dcc9d934f4606f654e8ac2874a6819910eb156f4fb2289a1"}
Jul 07 11:42:43.971623 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:43.971436    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b269716-9397-4dfd-98c0-9f02068c115d","Type":"ContainerDied","Data":"d37d15f4831299c673931a1cfc4b0c4616ee481152376def63392692b2db1ff9"}
Jul 07 11:42:43.971623 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:43.971477    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b269716-9397-4dfd-98c0-9f02068c115d","Type":"ContainerDied","Data":"cf13b380fdc40fe2bf8f3c9b615f7f906295b9b9ccf2e71fae776a21067b6b2c"}
Jul 07 11:42:44.118825 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.118806    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:44.188620 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.188550    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-secret-alertmanager-kube-rbac-proxy-web\") pod \"8b269716-9397-4dfd-98c0-9f02068c115d\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") "
Jul 07 11:42:44.188620 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.188584    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b269716-9397-4dfd-98c0-9f02068c115d-tls-assets\") pod \"8b269716-9397-4dfd-98c0-9f02068c115d\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") "
Jul 07 11:42:44.188620 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.188613    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b269716-9397-4dfd-98c0-9f02068c115d-metrics-client-ca\") pod \"8b269716-9397-4dfd-98c0-9f02068c115d\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") "
Jul 07 11:42:44.188877 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.188652    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b269716-9397-4dfd-98c0-9f02068c115d-config-out\") pod \"8b269716-9397-4dfd-98c0-9f02068c115d\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") "
Jul 07 11:42:44.188877 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.188683    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmptz\" (UniqueName: \"kubernetes.io/projected/8b269716-9397-4dfd-98c0-9f02068c115d-kube-api-access-nmptz\") pod \"8b269716-9397-4dfd-98c0-9f02068c115d\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") "
Jul 07 11:42:44.188877 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.188700    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-web-config\") pod \"8b269716-9397-4dfd-98c0-9f02068c115d\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") "
Jul 07 11:42:44.188877 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.188731    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b269716-9397-4dfd-98c0-9f02068c115d-alertmanager-trusted-ca-bundle\") pod \"8b269716-9397-4dfd-98c0-9f02068c115d\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") "
Jul 07 11:42:44.188877 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.188789    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8b269716-9397-4dfd-98c0-9f02068c115d-alertmanager-main-db\") pod \"8b269716-9397-4dfd-98c0-9f02068c115d\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") "
Jul 07 11:42:44.188877 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.188824    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-secret-alertmanager-kube-rbac-proxy\") pod \"8b269716-9397-4dfd-98c0-9f02068c115d\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") "
Jul 07 11:42:44.188877 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.188878    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-config-volume\") pod \"8b269716-9397-4dfd-98c0-9f02068c115d\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") "
Jul 07 11:42:44.189951 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.189255    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b269716-9397-4dfd-98c0-9f02068c115d-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "8b269716-9397-4dfd-98c0-9f02068c115d" (UID: "8b269716-9397-4dfd-98c0-9f02068c115d"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:42:44.189951 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.189333    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"8b269716-9397-4dfd-98c0-9f02068c115d\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") "
Jul 07 11:42:44.189951 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.189377    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-cluster-tls-config\") pod \"8b269716-9397-4dfd-98c0-9f02068c115d\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") "
Jul 07 11:42:44.189951 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.189502    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-secret-alertmanager-main-tls\") pod \"8b269716-9397-4dfd-98c0-9f02068c115d\" (UID: \"8b269716-9397-4dfd-98c0-9f02068c115d\") "
Jul 07 11:42:44.189951 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.189523    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b269716-9397-4dfd-98c0-9f02068c115d-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "8b269716-9397-4dfd-98c0-9f02068c115d" (UID: "8b269716-9397-4dfd-98c0-9f02068c115d"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 11:42:44.189951 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.189655    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b269716-9397-4dfd-98c0-9f02068c115d-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "8b269716-9397-4dfd-98c0-9f02068c115d" (UID: "8b269716-9397-4dfd-98c0-9f02068c115d"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:42:44.189951 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.189802    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b269716-9397-4dfd-98c0-9f02068c115d-metrics-client-ca\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:42:44.189951 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.189823    2581 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b269716-9397-4dfd-98c0-9f02068c115d-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:42:44.189951 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.189840    2581 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8b269716-9397-4dfd-98c0-9f02068c115d-alertmanager-main-db\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:42:44.192964 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.192937    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b269716-9397-4dfd-98c0-9f02068c115d-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8b269716-9397-4dfd-98c0-9f02068c115d" (UID: "8b269716-9397-4dfd-98c0-9f02068c115d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 11:42:44.193109 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.192980    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-config-volume" (OuterVolumeSpecName: "config-volume") pod "8b269716-9397-4dfd-98c0-9f02068c115d" (UID: "8b269716-9397-4dfd-98c0-9f02068c115d"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:42:44.193178 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.193118    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b269716-9397-4dfd-98c0-9f02068c115d-kube-api-access-nmptz" (OuterVolumeSpecName: "kube-api-access-nmptz") pod "8b269716-9397-4dfd-98c0-9f02068c115d" (UID: "8b269716-9397-4dfd-98c0-9f02068c115d"). InnerVolumeSpecName "kube-api-access-nmptz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 11:42:44.193178 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.193140    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b269716-9397-4dfd-98c0-9f02068c115d-config-out" (OuterVolumeSpecName: "config-out") pod "8b269716-9397-4dfd-98c0-9f02068c115d" (UID: "8b269716-9397-4dfd-98c0-9f02068c115d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 11:42:44.193849 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.193797    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "8b269716-9397-4dfd-98c0-9f02068c115d" (UID: "8b269716-9397-4dfd-98c0-9f02068c115d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:42:44.193945 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.193904    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "8b269716-9397-4dfd-98c0-9f02068c115d" (UID: "8b269716-9397-4dfd-98c0-9f02068c115d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:42:44.194041 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.194002    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "8b269716-9397-4dfd-98c0-9f02068c115d" (UID: "8b269716-9397-4dfd-98c0-9f02068c115d"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:42:44.194141 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.194115    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "8b269716-9397-4dfd-98c0-9f02068c115d" (UID: "8b269716-9397-4dfd-98c0-9f02068c115d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:42:44.198274 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.198183    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "8b269716-9397-4dfd-98c0-9f02068c115d" (UID: "8b269716-9397-4dfd-98c0-9f02068c115d"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:42:44.203948 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.203922    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-web-config" (OuterVolumeSpecName: "web-config") pod "8b269716-9397-4dfd-98c0-9f02068c115d" (UID: "8b269716-9397-4dfd-98c0-9f02068c115d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:42:44.290568 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.290544    2581 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b269716-9397-4dfd-98c0-9f02068c115d-config-out\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:42:44.290568 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.290566    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmptz\" (UniqueName: \"kubernetes.io/projected/8b269716-9397-4dfd-98c0-9f02068c115d-kube-api-access-nmptz\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:42:44.290709 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.290575    2581 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-web-config\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:42:44.290709 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.290585    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:42:44.290709 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.290594    2581 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-config-volume\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:42:44.290709 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.290603    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:42:44.290709 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.290612    2581 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-cluster-tls-config\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:42:44.290709 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.290621    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-secret-alertmanager-main-tls\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:42:44.290709 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.290629    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b269716-9397-4dfd-98c0-9f02068c115d-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:42:44.290709 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.290637    2581 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b269716-9397-4dfd-98c0-9f02068c115d-tls-assets\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:42:44.978538 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.978507    2581 generic.go:356] "Generic (PLEG): container finished" podID="8b269716-9397-4dfd-98c0-9f02068c115d" containerID="c93578c664db89c19be0817d871e3c4812a1e4658ab84fe08c412d3326f2a04a" exitCode=0
Jul 07 11:42:44.978538 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.978528    2581 generic.go:356] "Generic (PLEG): container finished" podID="8b269716-9397-4dfd-98c0-9f02068c115d" containerID="2a54923fc25a7a6dd1249cf2f375cdfe64c1eb5d1e09432464e04ff064ba0e38" exitCode=0
Jul 07 11:42:44.978903 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.978562    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b269716-9397-4dfd-98c0-9f02068c115d","Type":"ContainerDied","Data":"c93578c664db89c19be0817d871e3c4812a1e4658ab84fe08c412d3326f2a04a"}
Jul 07 11:42:44.978903 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.978583    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:44.978903 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.978634    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b269716-9397-4dfd-98c0-9f02068c115d","Type":"ContainerDied","Data":"2a54923fc25a7a6dd1249cf2f375cdfe64c1eb5d1e09432464e04ff064ba0e38"}
Jul 07 11:42:44.978903 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.978661    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b269716-9397-4dfd-98c0-9f02068c115d","Type":"ContainerDied","Data":"1b4a6a6ed90dc5460f63a029d5188fa4c65b2cc13122bbf943f8fed80751c2cd"}
Jul 07 11:42:44.978903 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.978704    2581 scope.go:117] "RemoveContainer" containerID="4073217113df8717613e00992b1d5ca869409b10d546d16fc1eafef902c0a910"
Jul 07 11:42:44.987076 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.987058    2581 scope.go:117] "RemoveContainer" containerID="c93578c664db89c19be0817d871e3c4812a1e4658ab84fe08c412d3326f2a04a"
Jul 07 11:42:44.993857 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:44.993686    2581 scope.go:117] "RemoveContainer" containerID="1cceb6e37766a703dcc9d934f4606f654e8ac2874a6819910eb156f4fb2289a1"
Jul 07 11:42:45.000665 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.000650    2581 scope.go:117] "RemoveContainer" containerID="2a54923fc25a7a6dd1249cf2f375cdfe64c1eb5d1e09432464e04ff064ba0e38"
Jul 07 11:42:45.006754 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.006737    2581 scope.go:117] "RemoveContainer" containerID="d37d15f4831299c673931a1cfc4b0c4616ee481152376def63392692b2db1ff9"
Jul 07 11:42:45.011345 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.011316    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 11:42:45.014711 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.014694    2581 scope.go:117] "RemoveContainer" containerID="cf13b380fdc40fe2bf8f3c9b615f7f906295b9b9ccf2e71fae776a21067b6b2c"
Jul 07 11:42:45.015270 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.015248    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 11:42:45.021162 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.021145    2581 scope.go:117] "RemoveContainer" containerID="49d51b2ea927026197b3cca0cce9f0dc9138354ed8312876bb3d318b1e8535d6"
Jul 07 11:42:45.027540 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.027523    2581 scope.go:117] "RemoveContainer" containerID="4073217113df8717613e00992b1d5ca869409b10d546d16fc1eafef902c0a910"
Jul 07 11:42:45.027795 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:42:45.027777    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4073217113df8717613e00992b1d5ca869409b10d546d16fc1eafef902c0a910\": container with ID starting with 4073217113df8717613e00992b1d5ca869409b10d546d16fc1eafef902c0a910 not found: ID does not exist" containerID="4073217113df8717613e00992b1d5ca869409b10d546d16fc1eafef902c0a910"
Jul 07 11:42:45.027917 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.027807    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4073217113df8717613e00992b1d5ca869409b10d546d16fc1eafef902c0a910"} err="failed to get container status \"4073217113df8717613e00992b1d5ca869409b10d546d16fc1eafef902c0a910\": rpc error: code = NotFound desc = could not find container \"4073217113df8717613e00992b1d5ca869409b10d546d16fc1eafef902c0a910\": container with ID starting with 4073217113df8717613e00992b1d5ca869409b10d546d16fc1eafef902c0a910 not found: ID does not exist"
Jul 07 11:42:45.027917 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.027846    2581 scope.go:117] "RemoveContainer" containerID="c93578c664db89c19be0817d871e3c4812a1e4658ab84fe08c412d3326f2a04a"
Jul 07 11:42:45.028084 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:42:45.028068    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c93578c664db89c19be0817d871e3c4812a1e4658ab84fe08c412d3326f2a04a\": container with ID starting with c93578c664db89c19be0817d871e3c4812a1e4658ab84fe08c412d3326f2a04a not found: ID does not exist" containerID="c93578c664db89c19be0817d871e3c4812a1e4658ab84fe08c412d3326f2a04a"
Jul 07 11:42:45.028135 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.028092    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c93578c664db89c19be0817d871e3c4812a1e4658ab84fe08c412d3326f2a04a"} err="failed to get container status \"c93578c664db89c19be0817d871e3c4812a1e4658ab84fe08c412d3326f2a04a\": rpc error: code = NotFound desc = could not find container \"c93578c664db89c19be0817d871e3c4812a1e4658ab84fe08c412d3326f2a04a\": container with ID starting with c93578c664db89c19be0817d871e3c4812a1e4658ab84fe08c412d3326f2a04a not found: ID does not exist"
Jul 07 11:42:45.028135 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.028114    2581 scope.go:117] "RemoveContainer" containerID="1cceb6e37766a703dcc9d934f4606f654e8ac2874a6819910eb156f4fb2289a1"
Jul 07 11:42:45.028317 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:42:45.028297    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cceb6e37766a703dcc9d934f4606f654e8ac2874a6819910eb156f4fb2289a1\": container with ID starting with 1cceb6e37766a703dcc9d934f4606f654e8ac2874a6819910eb156f4fb2289a1 not found: ID does not exist" containerID="1cceb6e37766a703dcc9d934f4606f654e8ac2874a6819910eb156f4fb2289a1"
Jul 07 11:42:45.028386 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.028323    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cceb6e37766a703dcc9d934f4606f654e8ac2874a6819910eb156f4fb2289a1"} err="failed to get container status \"1cceb6e37766a703dcc9d934f4606f654e8ac2874a6819910eb156f4fb2289a1\": rpc error: code = NotFound desc = could not find container \"1cceb6e37766a703dcc9d934f4606f654e8ac2874a6819910eb156f4fb2289a1\": container with ID starting with 1cceb6e37766a703dcc9d934f4606f654e8ac2874a6819910eb156f4fb2289a1 not found: ID does not exist"
Jul 07 11:42:45.028386 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.028343    2581 scope.go:117] "RemoveContainer" containerID="2a54923fc25a7a6dd1249cf2f375cdfe64c1eb5d1e09432464e04ff064ba0e38"
Jul 07 11:42:45.028595 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:42:45.028575    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a54923fc25a7a6dd1249cf2f375cdfe64c1eb5d1e09432464e04ff064ba0e38\": container with ID starting with 2a54923fc25a7a6dd1249cf2f375cdfe64c1eb5d1e09432464e04ff064ba0e38 not found: ID does not exist" containerID="2a54923fc25a7a6dd1249cf2f375cdfe64c1eb5d1e09432464e04ff064ba0e38"
Jul 07 11:42:45.028644 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.028601    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a54923fc25a7a6dd1249cf2f375cdfe64c1eb5d1e09432464e04ff064ba0e38"} err="failed to get container status \"2a54923fc25a7a6dd1249cf2f375cdfe64c1eb5d1e09432464e04ff064ba0e38\": rpc error: code = NotFound desc = could not find container \"2a54923fc25a7a6dd1249cf2f375cdfe64c1eb5d1e09432464e04ff064ba0e38\": container with ID starting with 2a54923fc25a7a6dd1249cf2f375cdfe64c1eb5d1e09432464e04ff064ba0e38 not found: ID does not exist"
Jul 07 11:42:45.028644 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.028615    2581 scope.go:117] "RemoveContainer" containerID="d37d15f4831299c673931a1cfc4b0c4616ee481152376def63392692b2db1ff9"
Jul 07 11:42:45.028848 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:42:45.028830    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d37d15f4831299c673931a1cfc4b0c4616ee481152376def63392692b2db1ff9\": container with ID starting with d37d15f4831299c673931a1cfc4b0c4616ee481152376def63392692b2db1ff9 not found: ID does not exist" containerID="d37d15f4831299c673931a1cfc4b0c4616ee481152376def63392692b2db1ff9"
Jul 07 11:42:45.028896 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.028853    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d37d15f4831299c673931a1cfc4b0c4616ee481152376def63392692b2db1ff9"} err="failed to get container status \"d37d15f4831299c673931a1cfc4b0c4616ee481152376def63392692b2db1ff9\": rpc error: code = NotFound desc = could not find container \"d37d15f4831299c673931a1cfc4b0c4616ee481152376def63392692b2db1ff9\": container with ID starting with d37d15f4831299c673931a1cfc4b0c4616ee481152376def63392692b2db1ff9 not found: ID does not exist"
Jul 07 11:42:45.028896 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.028868    2581 scope.go:117] "RemoveContainer" containerID="cf13b380fdc40fe2bf8f3c9b615f7f906295b9b9ccf2e71fae776a21067b6b2c"
Jul 07 11:42:45.029086 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:42:45.029069    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf13b380fdc40fe2bf8f3c9b615f7f906295b9b9ccf2e71fae776a21067b6b2c\": container with ID starting with cf13b380fdc40fe2bf8f3c9b615f7f906295b9b9ccf2e71fae776a21067b6b2c not found: ID does not exist" containerID="cf13b380fdc40fe2bf8f3c9b615f7f906295b9b9ccf2e71fae776a21067b6b2c"
Jul 07 11:42:45.029143 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.029090    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf13b380fdc40fe2bf8f3c9b615f7f906295b9b9ccf2e71fae776a21067b6b2c"} err="failed to get container status \"cf13b380fdc40fe2bf8f3c9b615f7f906295b9b9ccf2e71fae776a21067b6b2c\": rpc error: code = NotFound desc = could not find container \"cf13b380fdc40fe2bf8f3c9b615f7f906295b9b9ccf2e71fae776a21067b6b2c\": container with ID starting with cf13b380fdc40fe2bf8f3c9b615f7f906295b9b9ccf2e71fae776a21067b6b2c not found: ID does not exist"
Jul 07 11:42:45.029143 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.029106    2581 scope.go:117] "RemoveContainer" containerID="49d51b2ea927026197b3cca0cce9f0dc9138354ed8312876bb3d318b1e8535d6"
Jul 07 11:42:45.029341 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:42:45.029325    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49d51b2ea927026197b3cca0cce9f0dc9138354ed8312876bb3d318b1e8535d6\": container with ID starting with 49d51b2ea927026197b3cca0cce9f0dc9138354ed8312876bb3d318b1e8535d6 not found: ID does not exist" containerID="49d51b2ea927026197b3cca0cce9f0dc9138354ed8312876bb3d318b1e8535d6"
Jul 07 11:42:45.029382 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.029346    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49d51b2ea927026197b3cca0cce9f0dc9138354ed8312876bb3d318b1e8535d6"} err="failed to get container status \"49d51b2ea927026197b3cca0cce9f0dc9138354ed8312876bb3d318b1e8535d6\": rpc error: code = NotFound desc = could not find container \"49d51b2ea927026197b3cca0cce9f0dc9138354ed8312876bb3d318b1e8535d6\": container with ID starting with 49d51b2ea927026197b3cca0cce9f0dc9138354ed8312876bb3d318b1e8535d6 not found: ID does not exist"
Jul 07 11:42:45.029382 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.029361    2581 scope.go:117] "RemoveContainer" containerID="4073217113df8717613e00992b1d5ca869409b10d546d16fc1eafef902c0a910"
Jul 07 11:42:45.029579 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.029562    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4073217113df8717613e00992b1d5ca869409b10d546d16fc1eafef902c0a910"} err="failed to get container status \"4073217113df8717613e00992b1d5ca869409b10d546d16fc1eafef902c0a910\": rpc error: code = NotFound desc = could not find container \"4073217113df8717613e00992b1d5ca869409b10d546d16fc1eafef902c0a910\": container with ID starting with 4073217113df8717613e00992b1d5ca869409b10d546d16fc1eafef902c0a910 not found: ID does not exist"
Jul 07 11:42:45.029621 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.029587    2581 scope.go:117] "RemoveContainer" containerID="c93578c664db89c19be0817d871e3c4812a1e4658ab84fe08c412d3326f2a04a"
Jul 07 11:42:45.029799 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.029785    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c93578c664db89c19be0817d871e3c4812a1e4658ab84fe08c412d3326f2a04a"} err="failed to get container status \"c93578c664db89c19be0817d871e3c4812a1e4658ab84fe08c412d3326f2a04a\": rpc error: code = NotFound desc = could not find container \"c93578c664db89c19be0817d871e3c4812a1e4658ab84fe08c412d3326f2a04a\": container with ID starting with c93578c664db89c19be0817d871e3c4812a1e4658ab84fe08c412d3326f2a04a not found: ID does not exist"
Jul 07 11:42:45.029835 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.029800    2581 scope.go:117] "RemoveContainer" containerID="1cceb6e37766a703dcc9d934f4606f654e8ac2874a6819910eb156f4fb2289a1"
Jul 07 11:42:45.030024 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.029987    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cceb6e37766a703dcc9d934f4606f654e8ac2874a6819910eb156f4fb2289a1"} err="failed to get container status \"1cceb6e37766a703dcc9d934f4606f654e8ac2874a6819910eb156f4fb2289a1\": rpc error: code = NotFound desc = could not find container \"1cceb6e37766a703dcc9d934f4606f654e8ac2874a6819910eb156f4fb2289a1\": container with ID starting with 1cceb6e37766a703dcc9d934f4606f654e8ac2874a6819910eb156f4fb2289a1 not found: ID does not exist"
Jul 07 11:42:45.030076 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.030022    2581 scope.go:117] "RemoveContainer" containerID="2a54923fc25a7a6dd1249cf2f375cdfe64c1eb5d1e09432464e04ff064ba0e38"
Jul 07 11:42:45.030230 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.030214    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a54923fc25a7a6dd1249cf2f375cdfe64c1eb5d1e09432464e04ff064ba0e38"} err="failed to get container status \"2a54923fc25a7a6dd1249cf2f375cdfe64c1eb5d1e09432464e04ff064ba0e38\": rpc error: code = NotFound desc = could not find container \"2a54923fc25a7a6dd1249cf2f375cdfe64c1eb5d1e09432464e04ff064ba0e38\": container with ID starting with 2a54923fc25a7a6dd1249cf2f375cdfe64c1eb5d1e09432464e04ff064ba0e38 not found: ID does not exist"
Jul 07 11:42:45.030285 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.030230    2581 scope.go:117] "RemoveContainer" containerID="d37d15f4831299c673931a1cfc4b0c4616ee481152376def63392692b2db1ff9"
Jul 07 11:42:45.030442 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.030426    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d37d15f4831299c673931a1cfc4b0c4616ee481152376def63392692b2db1ff9"} err="failed to get container status \"d37d15f4831299c673931a1cfc4b0c4616ee481152376def63392692b2db1ff9\": rpc error: code = NotFound desc = could not find container \"d37d15f4831299c673931a1cfc4b0c4616ee481152376def63392692b2db1ff9\": container with ID starting with d37d15f4831299c673931a1cfc4b0c4616ee481152376def63392692b2db1ff9 not found: ID does not exist"
Jul 07 11:42:45.030493 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.030443    2581 scope.go:117] "RemoveContainer" containerID="cf13b380fdc40fe2bf8f3c9b615f7f906295b9b9ccf2e71fae776a21067b6b2c"
Jul 07 11:42:45.030616 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.030602    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf13b380fdc40fe2bf8f3c9b615f7f906295b9b9ccf2e71fae776a21067b6b2c"} err="failed to get container status \"cf13b380fdc40fe2bf8f3c9b615f7f906295b9b9ccf2e71fae776a21067b6b2c\": rpc error: code = NotFound desc = could not find container \"cf13b380fdc40fe2bf8f3c9b615f7f906295b9b9ccf2e71fae776a21067b6b2c\": container with ID starting with cf13b380fdc40fe2bf8f3c9b615f7f906295b9b9ccf2e71fae776a21067b6b2c not found: ID does not exist"
Jul 07 11:42:45.030674 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.030616    2581 scope.go:117] "RemoveContainer" containerID="49d51b2ea927026197b3cca0cce9f0dc9138354ed8312876bb3d318b1e8535d6"
Jul 07 11:42:45.030830 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.030811    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49d51b2ea927026197b3cca0cce9f0dc9138354ed8312876bb3d318b1e8535d6"} err="failed to get container status \"49d51b2ea927026197b3cca0cce9f0dc9138354ed8312876bb3d318b1e8535d6\": rpc error: code = NotFound desc = could not find container \"49d51b2ea927026197b3cca0cce9f0dc9138354ed8312876bb3d318b1e8535d6\": container with ID starting with 49d51b2ea927026197b3cca0cce9f0dc9138354ed8312876bb3d318b1e8535d6 not found: ID does not exist"
Jul 07 11:42:45.058326 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.058299    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 11:42:45.058562 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.058551    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b269716-9397-4dfd-98c0-9f02068c115d" containerName="alertmanager"
Jul 07 11:42:45.058602 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.058564    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b269716-9397-4dfd-98c0-9f02068c115d" containerName="alertmanager"
Jul 07 11:42:45.058602 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.058577    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b269716-9397-4dfd-98c0-9f02068c115d" containerName="kube-rbac-proxy-metric"
Jul 07 11:42:45.058602 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.058584    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b269716-9397-4dfd-98c0-9f02068c115d" containerName="kube-rbac-proxy-metric"
Jul 07 11:42:45.058602 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.058592    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b269716-9397-4dfd-98c0-9f02068c115d" containerName="prom-label-proxy"
Jul 07 11:42:45.058602 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.058598    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b269716-9397-4dfd-98c0-9f02068c115d" containerName="prom-label-proxy"
Jul 07 11:42:45.058746 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.058607    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b269716-9397-4dfd-98c0-9f02068c115d" containerName="init-config-reloader"
Jul 07 11:42:45.058746 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.058612    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b269716-9397-4dfd-98c0-9f02068c115d" containerName="init-config-reloader"
Jul 07 11:42:45.058746 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.058617    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b269716-9397-4dfd-98c0-9f02068c115d" containerName="kube-rbac-proxy-web"
Jul 07 11:42:45.058746 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.058623    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b269716-9397-4dfd-98c0-9f02068c115d" containerName="kube-rbac-proxy-web"
Jul 07 11:42:45.058746 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.058629    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b269716-9397-4dfd-98c0-9f02068c115d" containerName="config-reloader"
Jul 07 11:42:45.058746 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.058633    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b269716-9397-4dfd-98c0-9f02068c115d" containerName="config-reloader"
Jul 07 11:42:45.058746 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.058639    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b269716-9397-4dfd-98c0-9f02068c115d" containerName="kube-rbac-proxy"
Jul 07 11:42:45.058746 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.058643    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b269716-9397-4dfd-98c0-9f02068c115d" containerName="kube-rbac-proxy"
Jul 07 11:42:45.058746 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.058684    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b269716-9397-4dfd-98c0-9f02068c115d" containerName="prom-label-proxy"
Jul 07 11:42:45.058746 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.058691    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b269716-9397-4dfd-98c0-9f02068c115d" containerName="config-reloader"
Jul 07 11:42:45.058746 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.058696    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b269716-9397-4dfd-98c0-9f02068c115d" containerName="kube-rbac-proxy"
Jul 07 11:42:45.058746 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.058702    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b269716-9397-4dfd-98c0-9f02068c115d" containerName="kube-rbac-proxy-web"
Jul 07 11:42:45.058746 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.058707    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b269716-9397-4dfd-98c0-9f02068c115d" containerName="kube-rbac-proxy-metric"
Jul 07 11:42:45.058746 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.058712    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b269716-9397-4dfd-98c0-9f02068c115d" containerName="alertmanager"
Jul 07 11:42:45.061884 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.061868    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.071148 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.071123    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 07 11:42:45.071423 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.071405    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 07 11:42:45.071553 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.071408    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 07 11:42:45.071553 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.071493    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 07 11:42:45.071716 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.071702    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wxkvr\""
Jul 07 11:42:45.071833 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.071724    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 07 11:42:45.071833 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.071759    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 07 11:42:45.071935 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.071902    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 07 11:42:45.072188 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.072172    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 07 11:42:45.078117 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.078087    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 07 11:42:45.085529 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.085507    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 11:42:45.195308 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.195232    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff9decad-737f-4260-bc1a-a55386e2e26d-web-config\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.195308 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.195269    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff9decad-737f-4260-bc1a-a55386e2e26d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.195308 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.195296    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff9decad-737f-4260-bc1a-a55386e2e26d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.195550 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.195318    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff9decad-737f-4260-bc1a-a55386e2e26d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.195550 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.195337    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff9decad-737f-4260-bc1a-a55386e2e26d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.195550 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.195352    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ff9decad-737f-4260-bc1a-a55386e2e26d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.195550 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.195376    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ff9decad-737f-4260-bc1a-a55386e2e26d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.195550 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.195470    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff9decad-737f-4260-bc1a-a55386e2e26d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.195550 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.195507    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7cjn\" (UniqueName: \"kubernetes.io/projected/ff9decad-737f-4260-bc1a-a55386e2e26d-kube-api-access-z7cjn\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.195550 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.195547    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff9decad-737f-4260-bc1a-a55386e2e26d-config-out\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.195827 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.195588    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ff9decad-737f-4260-bc1a-a55386e2e26d-config-volume\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.195827 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.195611    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff9decad-737f-4260-bc1a-a55386e2e26d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.195827 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.195639    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ff9decad-737f-4260-bc1a-a55386e2e26d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.296573 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.296549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff9decad-737f-4260-bc1a-a55386e2e26d-web-config\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.296676 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.296603    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff9decad-737f-4260-bc1a-a55386e2e26d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.296676 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.296623    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff9decad-737f-4260-bc1a-a55386e2e26d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.296676 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.296640    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff9decad-737f-4260-bc1a-a55386e2e26d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.296676 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.296665    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff9decad-737f-4260-bc1a-a55386e2e26d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.296870 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.296827    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ff9decad-737f-4260-bc1a-a55386e2e26d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.296927 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.296867    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ff9decad-737f-4260-bc1a-a55386e2e26d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.296927 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.296904    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff9decad-737f-4260-bc1a-a55386e2e26d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.297054 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.296929    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7cjn\" (UniqueName: \"kubernetes.io/projected/ff9decad-737f-4260-bc1a-a55386e2e26d-kube-api-access-z7cjn\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.297054 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.296955    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff9decad-737f-4260-bc1a-a55386e2e26d-config-out\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.297054 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.297003    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ff9decad-737f-4260-bc1a-a55386e2e26d-config-volume\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.297054 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.297051    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff9decad-737f-4260-bc1a-a55386e2e26d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.297268 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.297079    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ff9decad-737f-4260-bc1a-a55386e2e26d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.297268 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.297202    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ff9decad-737f-4260-bc1a-a55386e2e26d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.297837 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.297807    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff9decad-737f-4260-bc1a-a55386e2e26d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.298676 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.298644    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff9decad-737f-4260-bc1a-a55386e2e26d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.300366 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.299946    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ff9decad-737f-4260-bc1a-a55386e2e26d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.300366 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.299973    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff9decad-737f-4260-bc1a-a55386e2e26d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.300366 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.300175    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ff9decad-737f-4260-bc1a-a55386e2e26d-config-volume\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.300366 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.300322    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff9decad-737f-4260-bc1a-a55386e2e26d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.300366 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.300358    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff9decad-737f-4260-bc1a-a55386e2e26d-web-config\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.300634 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.300376    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff9decad-737f-4260-bc1a-a55386e2e26d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.300634 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.300454    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff9decad-737f-4260-bc1a-a55386e2e26d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.300712 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.300693    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff9decad-737f-4260-bc1a-a55386e2e26d-config-out\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.301443 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.301419    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ff9decad-737f-4260-bc1a-a55386e2e26d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.306528 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.306509    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7cjn\" (UniqueName: \"kubernetes.io/projected/ff9decad-737f-4260-bc1a-a55386e2e26d-kube-api-access-z7cjn\") pod \"alertmanager-main-0\" (UID: \"ff9decad-737f-4260-bc1a-a55386e2e26d\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.370014 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.369995    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 07 11:42:45.499382 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.499359    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 11:42:45.500376 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:42:45.500350    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff9decad_737f_4260_bc1a_a55386e2e26d.slice/crio-e1ac1ce6e4e80551f3685ad4dd783451abf10ec2e848750e4a15fb9e9739e2a2 WatchSource:0}: Error finding container e1ac1ce6e4e80551f3685ad4dd783451abf10ec2e848750e4a15fb9e9739e2a2: Status 404 returned error can't find the container with id e1ac1ce6e4e80551f3685ad4dd783451abf10ec2e848750e4a15fb9e9739e2a2
Jul 07 11:42:45.984395 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.984361    2581 generic.go:356] "Generic (PLEG): container finished" podID="ff9decad-737f-4260-bc1a-a55386e2e26d" containerID="bbcfb86ffbf190fb5ca60a8c0047eb6d38d8e39c61cebc01bdff8e29e6e844e0" exitCode=0
Jul 07 11:42:45.984768 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.984451    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff9decad-737f-4260-bc1a-a55386e2e26d","Type":"ContainerDied","Data":"bbcfb86ffbf190fb5ca60a8c0047eb6d38d8e39c61cebc01bdff8e29e6e844e0"}
Jul 07 11:42:45.984768 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:45.984529    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff9decad-737f-4260-bc1a-a55386e2e26d","Type":"ContainerStarted","Data":"e1ac1ce6e4e80551f3685ad4dd783451abf10ec2e848750e4a15fb9e9739e2a2"}
Jul 07 11:42:46.172565 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:46.172532    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b269716-9397-4dfd-98c0-9f02068c115d" path="/var/lib/kubelet/pods/8b269716-9397-4dfd-98c0-9f02068c115d/volumes"
Jul 07 11:42:46.921861 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:46.921832    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-666b56dbc-kp4zb"]
Jul 07 11:42:46.924216 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:46.924190    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-666b56dbc-kp4zb"
Jul 07 11:42:46.929194 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:46.929170    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 07 11:42:46.929313 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:46.929299    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 07 11:42:46.929394 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:46.929373    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 07 11:42:46.930240 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:46.930221    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 07 11:42:46.930368 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:46.930222    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-mxfwn\""
Jul 07 11:42:46.930368 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:46.930222    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 07 11:42:46.936059 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:46.936041    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 07 11:42:46.947107 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:46.947087    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-666b56dbc-kp4zb"]
Jul 07 11:42:46.994071 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:46.994041    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff9decad-737f-4260-bc1a-a55386e2e26d","Type":"ContainerStarted","Data":"2eedbd80c4067806528a9248561665fd38551e0cc66be1934eabc6b0e83a7f90"}
Jul 07 11:42:46.994415 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:46.994107    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff9decad-737f-4260-bc1a-a55386e2e26d","Type":"ContainerStarted","Data":"ef2be80780c1adb17a6a504c4c05f720777d079ee9e74279ef6b2edf29bbf992"}
Jul 07 11:42:46.994415 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:46.994136    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff9decad-737f-4260-bc1a-a55386e2e26d","Type":"ContainerStarted","Data":"542da17fae8bbf58133df8ba1e49d5b5e6f3d0f29f959ff391eab7dba241e795"}
Jul 07 11:42:46.994415 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:46.994162    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff9decad-737f-4260-bc1a-a55386e2e26d","Type":"ContainerStarted","Data":"b46edd67ab2068b2f79d2ff9e81b833a64f28287c2e17a34d66c3f1579f50886"}
Jul 07 11:42:46.994415 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:46.994193    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff9decad-737f-4260-bc1a-a55386e2e26d","Type":"ContainerStarted","Data":"5658844b089b4cc7c15abef394039b7f8bccfc0b3cb97eafe96d88847efb3b16"}
Jul 07 11:42:46.994415 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:46.994223    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff9decad-737f-4260-bc1a-a55386e2e26d","Type":"ContainerStarted","Data":"c5443edaf2446230b0ff57ad744bc95924f9736c8efcb5040179cbfcf6b2c81b"}
Jul 07 11:42:47.010704 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.010681    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvg64\" (UniqueName: \"kubernetes.io/projected/2af33142-052d-442a-b5bc-e2767c7c6a14-kube-api-access-mvg64\") pod \"telemeter-client-666b56dbc-kp4zb\" (UID: \"2af33142-052d-442a-b5bc-e2767c7c6a14\") " pod="openshift-monitoring/telemeter-client-666b56dbc-kp4zb"
Jul 07 11:42:47.010841 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.010741    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2af33142-052d-442a-b5bc-e2767c7c6a14-federate-client-tls\") pod \"telemeter-client-666b56dbc-kp4zb\" (UID: \"2af33142-052d-442a-b5bc-e2767c7c6a14\") " pod="openshift-monitoring/telemeter-client-666b56dbc-kp4zb"
Jul 07 11:42:47.010841 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.010810    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2af33142-052d-442a-b5bc-e2767c7c6a14-telemeter-trusted-ca-bundle\") pod \"telemeter-client-666b56dbc-kp4zb\" (UID: \"2af33142-052d-442a-b5bc-e2767c7c6a14\") " pod="openshift-monitoring/telemeter-client-666b56dbc-kp4zb"
Jul 07 11:42:47.010940 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.010848    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2af33142-052d-442a-b5bc-e2767c7c6a14-serving-certs-ca-bundle\") pod \"telemeter-client-666b56dbc-kp4zb\" (UID: \"2af33142-052d-442a-b5bc-e2767c7c6a14\") " pod="openshift-monitoring/telemeter-client-666b56dbc-kp4zb"
Jul 07 11:42:47.010940 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.010879    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2af33142-052d-442a-b5bc-e2767c7c6a14-telemeter-client-tls\") pod \"telemeter-client-666b56dbc-kp4zb\" (UID: \"2af33142-052d-442a-b5bc-e2767c7c6a14\") " pod="openshift-monitoring/telemeter-client-666b56dbc-kp4zb"
Jul 07 11:42:47.011070 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.010933    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2af33142-052d-442a-b5bc-e2767c7c6a14-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-666b56dbc-kp4zb\" (UID: \"2af33142-052d-442a-b5bc-e2767c7c6a14\") " pod="openshift-monitoring/telemeter-client-666b56dbc-kp4zb"
Jul 07 11:42:47.011070 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.010985    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2af33142-052d-442a-b5bc-e2767c7c6a14-secret-telemeter-client\") pod \"telemeter-client-666b56dbc-kp4zb\" (UID: \"2af33142-052d-442a-b5bc-e2767c7c6a14\") " pod="openshift-monitoring/telemeter-client-666b56dbc-kp4zb"
Jul 07 11:42:47.011070 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.011030    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2af33142-052d-442a-b5bc-e2767c7c6a14-metrics-client-ca\") pod \"telemeter-client-666b56dbc-kp4zb\" (UID: \"2af33142-052d-442a-b5bc-e2767c7c6a14\") " pod="openshift-monitoring/telemeter-client-666b56dbc-kp4zb"
Jul 07 11:42:47.053043 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.052965    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.03997906 podStartE2EDuration="2.052945208s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.966148ms" isStatefulPod=false podCreationTimestamp="2026-07-07 11:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:42:47.050628041 +0000 UTC m=+157.397841675" watchObservedRunningTime="2026-07-07 11:42:47.052945208 +0000 UTC m=+157.400158833"
Jul 07 11:42:47.112216 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.112175    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2af33142-052d-442a-b5bc-e2767c7c6a14-telemeter-client-tls\") pod \"telemeter-client-666b56dbc-kp4zb\" (UID: \"2af33142-052d-442a-b5bc-e2767c7c6a14\") " pod="openshift-monitoring/telemeter-client-666b56dbc-kp4zb"
Jul 07 11:42:47.112367 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.112225    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2af33142-052d-442a-b5bc-e2767c7c6a14-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-666b56dbc-kp4zb\" (UID: \"2af33142-052d-442a-b5bc-e2767c7c6a14\") " pod="openshift-monitoring/telemeter-client-666b56dbc-kp4zb"
Jul 07 11:42:47.112367 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.112255    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2af33142-052d-442a-b5bc-e2767c7c6a14-secret-telemeter-client\") pod \"telemeter-client-666b56dbc-kp4zb\" (UID: \"2af33142-052d-442a-b5bc-e2767c7c6a14\") " pod="openshift-monitoring/telemeter-client-666b56dbc-kp4zb"
Jul 07 11:42:47.112367 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.112280    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2af33142-052d-442a-b5bc-e2767c7c6a14-metrics-client-ca\") pod \"telemeter-client-666b56dbc-kp4zb\" (UID: \"2af33142-052d-442a-b5bc-e2767c7c6a14\") " pod="openshift-monitoring/telemeter-client-666b56dbc-kp4zb"
Jul 07 11:42:47.112367 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.112353    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvg64\" (UniqueName: \"kubernetes.io/projected/2af33142-052d-442a-b5bc-e2767c7c6a14-kube-api-access-mvg64\") pod \"telemeter-client-666b56dbc-kp4zb\" (UID: \"2af33142-052d-442a-b5bc-e2767c7c6a14\") " pod="openshift-monitoring/telemeter-client-666b56dbc-kp4zb"
Jul 07 11:42:47.112591 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.112488    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2af33142-052d-442a-b5bc-e2767c7c6a14-federate-client-tls\") pod \"telemeter-client-666b56dbc-kp4zb\" (UID: \"2af33142-052d-442a-b5bc-e2767c7c6a14\") " pod="openshift-monitoring/telemeter-client-666b56dbc-kp4zb"
Jul 07 11:42:47.112591 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.112552    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2af33142-052d-442a-b5bc-e2767c7c6a14-telemeter-trusted-ca-bundle\") pod \"telemeter-client-666b56dbc-kp4zb\" (UID: \"2af33142-052d-442a-b5bc-e2767c7c6a14\") " pod="openshift-monitoring/telemeter-client-666b56dbc-kp4zb"
Jul 07 11:42:47.112689 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.112595    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2af33142-052d-442a-b5bc-e2767c7c6a14-serving-certs-ca-bundle\") pod \"telemeter-client-666b56dbc-kp4zb\" (UID: \"2af33142-052d-442a-b5bc-e2767c7c6a14\") " pod="openshift-monitoring/telemeter-client-666b56dbc-kp4zb"
Jul 07 11:42:47.116725 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.115757    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2af33142-052d-442a-b5bc-e2767c7c6a14-metrics-client-ca\") pod \"telemeter-client-666b56dbc-kp4zb\" (UID: \"2af33142-052d-442a-b5bc-e2767c7c6a14\") " pod="openshift-monitoring/telemeter-client-666b56dbc-kp4zb"
Jul 07 11:42:47.116725 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.116238    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2af33142-052d-442a-b5bc-e2767c7c6a14-telemeter-client-tls\") pod \"telemeter-client-666b56dbc-kp4zb\" (UID: \"2af33142-052d-442a-b5bc-e2767c7c6a14\") " pod="openshift-monitoring/telemeter-client-666b56dbc-kp4zb"
Jul 07 11:42:47.116725 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.116675    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2af33142-052d-442a-b5bc-e2767c7c6a14-serving-certs-ca-bundle\") pod \"telemeter-client-666b56dbc-kp4zb\" (UID: \"2af33142-052d-442a-b5bc-e2767c7c6a14\") " pod="openshift-monitoring/telemeter-client-666b56dbc-kp4zb"
Jul 07 11:42:47.116978 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.116795    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2af33142-052d-442a-b5bc-e2767c7c6a14-telemeter-trusted-ca-bundle\") pod \"telemeter-client-666b56dbc-kp4zb\" (UID: \"2af33142-052d-442a-b5bc-e2767c7c6a14\") " pod="openshift-monitoring/telemeter-client-666b56dbc-kp4zb"
Jul 07 11:42:47.117152 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.117125    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2af33142-052d-442a-b5bc-e2767c7c6a14-federate-client-tls\") pod \"telemeter-client-666b56dbc-kp4zb\" (UID: \"2af33142-052d-442a-b5bc-e2767c7c6a14\") " pod="openshift-monitoring/telemeter-client-666b56dbc-kp4zb"
Jul 07 11:42:47.117492 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.117469    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2af33142-052d-442a-b5bc-e2767c7c6a14-secret-telemeter-client\") pod \"telemeter-client-666b56dbc-kp4zb\" (UID: \"2af33142-052d-442a-b5bc-e2767c7c6a14\") " pod="openshift-monitoring/telemeter-client-666b56dbc-kp4zb"
Jul 07 11:42:47.117884 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.117865    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2af33142-052d-442a-b5bc-e2767c7c6a14-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-666b56dbc-kp4zb\" (UID: \"2af33142-052d-442a-b5bc-e2767c7c6a14\") " pod="openshift-monitoring/telemeter-client-666b56dbc-kp4zb"
Jul 07 11:42:47.134550 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.134523    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvg64\" (UniqueName: \"kubernetes.io/projected/2af33142-052d-442a-b5bc-e2767c7c6a14-kube-api-access-mvg64\") pod \"telemeter-client-666b56dbc-kp4zb\" (UID: \"2af33142-052d-442a-b5bc-e2767c7c6a14\") " pod="openshift-monitoring/telemeter-client-666b56dbc-kp4zb"
Jul 07 11:42:47.243539 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.243462    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-666b56dbc-kp4zb"
Jul 07 11:42:47.323064 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.321689    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 11:42:47.323064 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.322072    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="aaa9bb81-dc87-4b74-be29-587ab84bf1e9" containerName="prometheus" containerID="cri-o://9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4" gracePeriod=600
Jul 07 11:42:47.323064 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.322450    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="aaa9bb81-dc87-4b74-be29-587ab84bf1e9" containerName="kube-rbac-proxy-thanos" containerID="cri-o://16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7" gracePeriod=600
Jul 07 11:42:47.323064 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.322561    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="aaa9bb81-dc87-4b74-be29-587ab84bf1e9" containerName="kube-rbac-proxy" containerID="cri-o://c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6" gracePeriod=600
Jul 07 11:42:47.323064 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.322637    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="aaa9bb81-dc87-4b74-be29-587ab84bf1e9" containerName="kube-rbac-proxy-web" containerID="cri-o://f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb" gracePeriod=600
Jul 07 11:42:47.323064 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.322710    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="aaa9bb81-dc87-4b74-be29-587ab84bf1e9" containerName="thanos-sidecar" containerID="cri-o://f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5" gracePeriod=600
Jul 07 11:42:47.323064 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.322784    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="aaa9bb81-dc87-4b74-be29-587ab84bf1e9" containerName="config-reloader" containerID="cri-o://f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1" gracePeriod=600
Jul 07 11:42:47.427235 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.426879    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-666b56dbc-kp4zb"]
Jul 07 11:42:47.429252 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:42:47.429220    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2af33142_052d_442a_b5bc_e2767c7c6a14.slice/crio-866aa2b97ebd0e508727570a181485619888c96335664958e2db3e6574491f86 WatchSource:0}: Error finding container 866aa2b97ebd0e508727570a181485619888c96335664958e2db3e6574491f86: Status 404 returned error can't find the container with id 866aa2b97ebd0e508727570a181485619888c96335664958e2db3e6574491f86
Jul 07 11:42:47.686874 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.686841    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:47.828794 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.828724    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-prometheus-k8s-rulefiles-2\") pod \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") "
Jul 07 11:42:47.828794 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.828755    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") "
Jul 07 11:42:47.828794 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.828775    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-thanos-prometheus-http-client-file\") pod \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") "
Jul 07 11:42:47.829073 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.828799    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-config-out\") pod \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") "
Jul 07 11:42:47.829073 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.828822    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-kube-rbac-proxy\") pod \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") "
Jul 07 11:42:47.829073 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.828838    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-prometheus-k8s-db\") pod \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") "
Jul 07 11:42:47.829073 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.828859    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") "
Jul 07 11:42:47.829073 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.828890    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-configmap-metrics-client-ca\") pod \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") "
Jul 07 11:42:47.829073 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.828920    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-454v5\" (UniqueName: \"kubernetes.io/projected/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-kube-api-access-454v5\") pod \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") "
Jul 07 11:42:47.829073 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.828966    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-web-config\") pod \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") "
Jul 07 11:42:47.829073 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.829029    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-config\") pod \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") "
Jul 07 11:42:47.829073 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.829054    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-configmap-serving-certs-ca-bundle\") pod \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") "
Jul 07 11:42:47.829494 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.829082    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-grpc-tls\") pod \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") "
Jul 07 11:42:47.829494 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.829125    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-prometheus-k8s-rulefiles-1\") pod \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") "
Jul 07 11:42:47.829494 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.829149    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-configmap-kubelet-serving-ca-bundle\") pod \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") "
Jul 07 11:42:47.829494 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.829176    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-prometheus-k8s-rulefiles-0\") pod \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") "
Jul 07 11:42:47.829494 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.829168    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "aaa9bb81-dc87-4b74-be29-587ab84bf1e9" (UID: "aaa9bb81-dc87-4b74-be29-587ab84bf1e9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:42:47.829494 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.829214    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-prometheus-k8s-tls\") pod \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") "
Jul 07 11:42:47.829494 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.829262    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-prometheus-trusted-ca-bundle\") pod \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") "
Jul 07 11:42:47.829494 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.829291    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-metrics-client-certs\") pod \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") "
Jul 07 11:42:47.829494 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.829360    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-tls-assets\") pod \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\" (UID: \"aaa9bb81-dc87-4b74-be29-587ab84bf1e9\") "
Jul 07 11:42:47.829940 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.829635    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:42:47.830031 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.829992    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "aaa9bb81-dc87-4b74-be29-587ab84bf1e9" (UID: "aaa9bb81-dc87-4b74-be29-587ab84bf1e9"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 11:42:47.830520 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.830493    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "aaa9bb81-dc87-4b74-be29-587ab84bf1e9" (UID: "aaa9bb81-dc87-4b74-be29-587ab84bf1e9"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:42:47.831027 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.830986    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "aaa9bb81-dc87-4b74-be29-587ab84bf1e9" (UID: "aaa9bb81-dc87-4b74-be29-587ab84bf1e9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:42:47.831543 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.831507    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-config-out" (OuterVolumeSpecName: "config-out") pod "aaa9bb81-dc87-4b74-be29-587ab84bf1e9" (UID: "aaa9bb81-dc87-4b74-be29-587ab84bf1e9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 11:42:47.831543 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.831509    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "aaa9bb81-dc87-4b74-be29-587ab84bf1e9" (UID: "aaa9bb81-dc87-4b74-be29-587ab84bf1e9"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:42:47.831696 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.831534    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "aaa9bb81-dc87-4b74-be29-587ab84bf1e9" (UID: "aaa9bb81-dc87-4b74-be29-587ab84bf1e9"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:42:47.831696 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.831565    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "aaa9bb81-dc87-4b74-be29-587ab84bf1e9" (UID: "aaa9bb81-dc87-4b74-be29-587ab84bf1e9"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:42:47.831696 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.831579    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-config" (OuterVolumeSpecName: "config") pod "aaa9bb81-dc87-4b74-be29-587ab84bf1e9" (UID: "aaa9bb81-dc87-4b74-be29-587ab84bf1e9"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:42:47.831919 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.831866    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "aaa9bb81-dc87-4b74-be29-587ab84bf1e9" (UID: "aaa9bb81-dc87-4b74-be29-587ab84bf1e9"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:42:47.832431 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.832271    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "aaa9bb81-dc87-4b74-be29-587ab84bf1e9" (UID: "aaa9bb81-dc87-4b74-be29-587ab84bf1e9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 11:42:47.832431 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.832311    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "aaa9bb81-dc87-4b74-be29-587ab84bf1e9" (UID: "aaa9bb81-dc87-4b74-be29-587ab84bf1e9"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:42:47.832431 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.832361    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "aaa9bb81-dc87-4b74-be29-587ab84bf1e9" (UID: "aaa9bb81-dc87-4b74-be29-587ab84bf1e9"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:42:47.832431 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.832397    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "aaa9bb81-dc87-4b74-be29-587ab84bf1e9" (UID: "aaa9bb81-dc87-4b74-be29-587ab84bf1e9"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:42:47.833347 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.833322    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "aaa9bb81-dc87-4b74-be29-587ab84bf1e9" (UID: "aaa9bb81-dc87-4b74-be29-587ab84bf1e9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:42:47.833476 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.833456    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-kube-api-access-454v5" (OuterVolumeSpecName: "kube-api-access-454v5") pod "aaa9bb81-dc87-4b74-be29-587ab84bf1e9" (UID: "aaa9bb81-dc87-4b74-be29-587ab84bf1e9"). InnerVolumeSpecName "kube-api-access-454v5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 11:42:47.833593 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.833578    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "aaa9bb81-dc87-4b74-be29-587ab84bf1e9" (UID: "aaa9bb81-dc87-4b74-be29-587ab84bf1e9"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:42:47.833865 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.833851    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "aaa9bb81-dc87-4b74-be29-587ab84bf1e9" (UID: "aaa9bb81-dc87-4b74-be29-587ab84bf1e9"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:42:47.833967 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.833954    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "aaa9bb81-dc87-4b74-be29-587ab84bf1e9" (UID: "aaa9bb81-dc87-4b74-be29-587ab84bf1e9"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:42:47.843273 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.843255    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-web-config" (OuterVolumeSpecName: "web-config") pod "aaa9bb81-dc87-4b74-be29-587ab84bf1e9" (UID: "aaa9bb81-dc87-4b74-be29-587ab84bf1e9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:42:47.930260 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.930223    2581 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-web-config\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:42:47.930260 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.930261    2581 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-config\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:42:47.930414 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.930278    2581 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:42:47.930414 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.930293    2581 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-grpc-tls\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:42:47.930414 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.930306    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:42:47.930414 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.930321    2581 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:42:47.930414 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.930335    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:42:47.930414 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.930349    2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:42:47.930414 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.930363    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:42:47.930414 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.930376    2581 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-metrics-client-certs\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:42:47.930414 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.930390    2581 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-tls-assets\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:42:47.930414 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.930404    2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:42:47.930752 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.930418    2581 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:42:47.930752 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.930431    2581 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-config-out\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:42:47.930752 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.930447    2581 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-kube-rbac-proxy\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:42:47.930752 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.930460    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-prometheus-k8s-db\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:42:47.930752 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.930473    2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:42:47.930752 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.930486    2581 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-configmap-metrics-client-ca\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:42:47.930752 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:47.930509    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-454v5\" (UniqueName: \"kubernetes.io/projected/aaa9bb81-dc87-4b74-be29-587ab84bf1e9-kube-api-access-454v5\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:42:48.000075 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.000046    2581 generic.go:356] "Generic (PLEG): container finished" podID="aaa9bb81-dc87-4b74-be29-587ab84bf1e9" containerID="16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7" exitCode=0
Jul 07 11:42:48.000075 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.000071    2581 generic.go:356] "Generic (PLEG): container finished" podID="aaa9bb81-dc87-4b74-be29-587ab84bf1e9" containerID="c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6" exitCode=0
Jul 07 11:42:48.000075 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.000078    2581 generic.go:356] "Generic (PLEG): container finished" podID="aaa9bb81-dc87-4b74-be29-587ab84bf1e9" containerID="f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb" exitCode=0
Jul 07 11:42:48.000490 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.000083    2581 generic.go:356] "Generic (PLEG): container finished" podID="aaa9bb81-dc87-4b74-be29-587ab84bf1e9" containerID="f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5" exitCode=0
Jul 07 11:42:48.000490 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.000088    2581 generic.go:356] "Generic (PLEG): container finished" podID="aaa9bb81-dc87-4b74-be29-587ab84bf1e9" containerID="f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1" exitCode=0
Jul 07 11:42:48.000490 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.000093    2581 generic.go:356] "Generic (PLEG): container finished" podID="aaa9bb81-dc87-4b74-be29-587ab84bf1e9" containerID="9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4" exitCode=0
Jul 07 11:42:48.000490 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.000112    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.000490 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.000132    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aaa9bb81-dc87-4b74-be29-587ab84bf1e9","Type":"ContainerDied","Data":"16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7"}
Jul 07 11:42:48.000490 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.000244    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aaa9bb81-dc87-4b74-be29-587ab84bf1e9","Type":"ContainerDied","Data":"c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6"}
Jul 07 11:42:48.000490 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.000279    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aaa9bb81-dc87-4b74-be29-587ab84bf1e9","Type":"ContainerDied","Data":"f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb"}
Jul 07 11:42:48.000490 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.000319    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aaa9bb81-dc87-4b74-be29-587ab84bf1e9","Type":"ContainerDied","Data":"f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5"}
Jul 07 11:42:48.000490 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.000346    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aaa9bb81-dc87-4b74-be29-587ab84bf1e9","Type":"ContainerDied","Data":"f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1"}
Jul 07 11:42:48.000490 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.000317    2581 scope.go:117] "RemoveContainer" containerID="16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7"
Jul 07 11:42:48.000490 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.000370    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aaa9bb81-dc87-4b74-be29-587ab84bf1e9","Type":"ContainerDied","Data":"9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4"}
Jul 07 11:42:48.000490 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.000401    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aaa9bb81-dc87-4b74-be29-587ab84bf1e9","Type":"ContainerDied","Data":"083ead2cab0a8e8a90a996103ac4e5cb5e2c20eb56e9ec8c280a91e107232a33"}
Jul 07 11:42:48.001499 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.001480    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-666b56dbc-kp4zb" event={"ID":"2af33142-052d-442a-b5bc-e2767c7c6a14","Type":"ContainerStarted","Data":"866aa2b97ebd0e508727570a181485619888c96335664958e2db3e6574491f86"}
Jul 07 11:42:48.008301 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.008277    2581 scope.go:117] "RemoveContainer" containerID="c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6"
Jul 07 11:42:48.015238 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.015182    2581 scope.go:117] "RemoveContainer" containerID="f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb"
Jul 07 11:42:48.021649 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.021633    2581 scope.go:117] "RemoveContainer" containerID="f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5"
Jul 07 11:42:48.028059 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.028037    2581 scope.go:117] "RemoveContainer" containerID="f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1"
Jul 07 11:42:48.029875 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.029833    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 11:42:48.037025 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.036992    2581 scope.go:117] "RemoveContainer" containerID="9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4"
Jul 07 11:42:48.039183 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.039163    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 11:42:48.045142 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.045117    2581 scope.go:117] "RemoveContainer" containerID="fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce"
Jul 07 11:42:48.052758 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.052721    2581 scope.go:117] "RemoveContainer" containerID="16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7"
Jul 07 11:42:48.053024 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:42:48.052992    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7\": container with ID starting with 16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7 not found: ID does not exist" containerID="16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7"
Jul 07 11:42:48.053086 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.053031    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7"} err="failed to get container status \"16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7\": rpc error: code = NotFound desc = could not find container \"16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7\": container with ID starting with 16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7 not found: ID does not exist"
Jul 07 11:42:48.053086 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.053047    2581 scope.go:117] "RemoveContainer" containerID="c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6"
Jul 07 11:42:48.053304 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:42:48.053285    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6\": container with ID starting with c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6 not found: ID does not exist" containerID="c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6"
Jul 07 11:42:48.053384 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.053307    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6"} err="failed to get container status \"c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6\": rpc error: code = NotFound desc = could not find container \"c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6\": container with ID starting with c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6 not found: ID does not exist"
Jul 07 11:42:48.053384 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.053320    2581 scope.go:117] "RemoveContainer" containerID="f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb"
Jul 07 11:42:48.053564 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:42:48.053548    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb\": container with ID starting with f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb not found: ID does not exist" containerID="f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb"
Jul 07 11:42:48.053617 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.053568    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb"} err="failed to get container status \"f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb\": rpc error: code = NotFound desc = could not find container \"f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb\": container with ID starting with f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb not found: ID does not exist"
Jul 07 11:42:48.053617 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.053581    2581 scope.go:117] "RemoveContainer" containerID="f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5"
Jul 07 11:42:48.053797 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:42:48.053777    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5\": container with ID starting with f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5 not found: ID does not exist" containerID="f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5"
Jul 07 11:42:48.053846 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.053804    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5"} err="failed to get container status \"f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5\": rpc error: code = NotFound desc = could not find container \"f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5\": container with ID starting with f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5 not found: ID does not exist"
Jul 07 11:42:48.053846 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.053822    2581 scope.go:117] "RemoveContainer" containerID="f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1"
Jul 07 11:42:48.054079 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:42:48.054059    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1\": container with ID starting with f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1 not found: ID does not exist" containerID="f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1"
Jul 07 11:42:48.054165 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.054087    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1"} err="failed to get container status \"f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1\": rpc error: code = NotFound desc = could not find container \"f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1\": container with ID starting with f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1 not found: ID does not exist"
Jul 07 11:42:48.054165 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.054105    2581 scope.go:117] "RemoveContainer" containerID="9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4"
Jul 07 11:42:48.054373 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:42:48.054356    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4\": container with ID starting with 9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4 not found: ID does not exist" containerID="9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4"
Jul 07 11:42:48.054373 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.054377    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4"} err="failed to get container status \"9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4\": rpc error: code = NotFound desc = could not find container \"9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4\": container with ID starting with 9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4 not found: ID does not exist"
Jul 07 11:42:48.054478 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.054394    2581 scope.go:117] "RemoveContainer" containerID="fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce"
Jul 07 11:42:48.054678 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:42:48.054656    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce\": container with ID starting with fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce not found: ID does not exist" containerID="fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce"
Jul 07 11:42:48.054768 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.054687    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce"} err="failed to get container status \"fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce\": rpc error: code = NotFound desc = could not find container \"fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce\": container with ID starting with fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce not found: ID does not exist"
Jul 07 11:42:48.054768 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.054707    2581 scope.go:117] "RemoveContainer" containerID="16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7"
Jul 07 11:42:48.054987 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.054956    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7"} err="failed to get container status \"16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7\": rpc error: code = NotFound desc = could not find container \"16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7\": container with ID starting with 16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7 not found: ID does not exist"
Jul 07 11:42:48.054987 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.054976    2581 scope.go:117] "RemoveContainer" containerID="c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6"
Jul 07 11:42:48.055242 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.055218    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6"} err="failed to get container status \"c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6\": rpc error: code = NotFound desc = could not find container \"c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6\": container with ID starting with c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6 not found: ID does not exist"
Jul 07 11:42:48.055345 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.055243    2581 scope.go:117] "RemoveContainer" containerID="f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb"
Jul 07 11:42:48.055541 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.055515    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb"} err="failed to get container status \"f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb\": rpc error: code = NotFound desc = could not find container \"f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb\": container with ID starting with f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb not found: ID does not exist"
Jul 07 11:42:48.055645 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.055542    2581 scope.go:117] "RemoveContainer" containerID="f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5"
Jul 07 11:42:48.055850 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.055830    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5"} err="failed to get container status \"f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5\": rpc error: code = NotFound desc = could not find container \"f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5\": container with ID starting with f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5 not found: ID does not exist"
Jul 07 11:42:48.055919 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.055850    2581 scope.go:117] "RemoveContainer" containerID="f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1"
Jul 07 11:42:48.056102 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.056084    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1"} err="failed to get container status \"f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1\": rpc error: code = NotFound desc = could not find container \"f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1\": container with ID starting with f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1 not found: ID does not exist"
Jul 07 11:42:48.056181 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.056102    2581 scope.go:117] "RemoveContainer" containerID="9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4"
Jul 07 11:42:48.056349 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.056318    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4"} err="failed to get container status \"9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4\": rpc error: code = NotFound desc = could not find container \"9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4\": container with ID starting with 9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4 not found: ID does not exist"
Jul 07 11:42:48.056349 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.056346    2581 scope.go:117] "RemoveContainer" containerID="fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce"
Jul 07 11:42:48.056703 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.056668    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce"} err="failed to get container status \"fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce\": rpc error: code = NotFound desc = could not find container \"fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce\": container with ID starting with fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce not found: ID does not exist"
Jul 07 11:42:48.056703 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.056694    2581 scope.go:117] "RemoveContainer" containerID="16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7"
Jul 07 11:42:48.056948 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.056929    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7"} err="failed to get container status \"16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7\": rpc error: code = NotFound desc = could not find container \"16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7\": container with ID starting with 16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7 not found: ID does not exist"
Jul 07 11:42:48.056948 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.056946    2581 scope.go:117] "RemoveContainer" containerID="c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6"
Jul 07 11:42:48.057215 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.057196    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6"} err="failed to get container status \"c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6\": rpc error: code = NotFound desc = could not find container \"c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6\": container with ID starting with c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6 not found: ID does not exist"
Jul 07 11:42:48.057300 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.057215    2581 scope.go:117] "RemoveContainer" containerID="f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb"
Jul 07 11:42:48.057468 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.057450    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb"} err="failed to get container status \"f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb\": rpc error: code = NotFound desc = could not find container \"f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb\": container with ID starting with f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb not found: ID does not exist"
Jul 07 11:42:48.057468 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.057467    2581 scope.go:117] "RemoveContainer" containerID="f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5"
Jul 07 11:42:48.057713 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.057687    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5"} err="failed to get container status \"f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5\": rpc error: code = NotFound desc = could not find container \"f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5\": container with ID starting with f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5 not found: ID does not exist"
Jul 07 11:42:48.057713 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.057713    2581 scope.go:117] "RemoveContainer" containerID="f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1"
Jul 07 11:42:48.058040 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.058004    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1"} err="failed to get container status \"f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1\": rpc error: code = NotFound desc = could not find container \"f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1\": container with ID starting with f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1 not found: ID does not exist"
Jul 07 11:42:48.058126 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.058043    2581 scope.go:117] "RemoveContainer" containerID="9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4"
Jul 07 11:42:48.058298 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.058274    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4"} err="failed to get container status \"9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4\": rpc error: code = NotFound desc = could not find container \"9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4\": container with ID starting with 9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4 not found: ID does not exist"
Jul 07 11:42:48.058298 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.058298    2581 scope.go:117] "RemoveContainer" containerID="fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce"
Jul 07 11:42:48.058536 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.058516    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce"} err="failed to get container status \"fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce\": rpc error: code = NotFound desc = could not find container \"fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce\": container with ID starting with fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce not found: ID does not exist"
Jul 07 11:42:48.058536 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.058535    2581 scope.go:117] "RemoveContainer" containerID="16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7"
Jul 07 11:42:48.058789 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.058757    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7"} err="failed to get container status \"16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7\": rpc error: code = NotFound desc = could not find container \"16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7\": container with ID starting with 16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7 not found: ID does not exist"
Jul 07 11:42:48.058852 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.058791    2581 scope.go:117] "RemoveContainer" containerID="c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6"
Jul 07 11:42:48.059063 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.059041    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6"} err="failed to get container status \"c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6\": rpc error: code = NotFound desc = could not find container \"c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6\": container with ID starting with c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6 not found: ID does not exist"
Jul 07 11:42:48.059146 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.059066    2581 scope.go:117] "RemoveContainer" containerID="f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb"
Jul 07 11:42:48.059425 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.059403    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb"} err="failed to get container status \"f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb\": rpc error: code = NotFound desc = could not find container \"f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb\": container with ID starting with f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb not found: ID does not exist"
Jul 07 11:42:48.059522 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.059426    2581 scope.go:117] "RemoveContainer" containerID="f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5"
Jul 07 11:42:48.059692 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.059668    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5"} err="failed to get container status \"f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5\": rpc error: code = NotFound desc = could not find container \"f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5\": container with ID starting with f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5 not found: ID does not exist"
Jul 07 11:42:48.059756 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.059696    2581 scope.go:117] "RemoveContainer" containerID="f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1"
Jul 07 11:42:48.059929 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.059902    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1"} err="failed to get container status \"f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1\": rpc error: code = NotFound desc = could not find container \"f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1\": container with ID starting with f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1 not found: ID does not exist"
Jul 07 11:42:48.059987 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.059927    2581 scope.go:117] "RemoveContainer" containerID="9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4"
Jul 07 11:42:48.060181 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.060155    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4"} err="failed to get container status \"9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4\": rpc error: code = NotFound desc = could not find container \"9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4\": container with ID starting with 9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4 not found: ID does not exist"
Jul 07 11:42:48.060279 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.060182    2581 scope.go:117] "RemoveContainer" containerID="fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce"
Jul 07 11:42:48.060471 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.060454    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce"} err="failed to get container status \"fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce\": rpc error: code = NotFound desc = could not find container \"fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce\": container with ID starting with fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce not found: ID does not exist"
Jul 07 11:42:48.060535 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.060471    2581 scope.go:117] "RemoveContainer" containerID="16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7"
Jul 07 11:42:48.060723 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.060695    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7"} err="failed to get container status \"16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7\": rpc error: code = NotFound desc = could not find container \"16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7\": container with ID starting with 16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7 not found: ID does not exist"
Jul 07 11:42:48.060779 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.060724    2581 scope.go:117] "RemoveContainer" containerID="c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6"
Jul 07 11:42:48.061053 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.061023    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6"} err="failed to get container status \"c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6\": rpc error: code = NotFound desc = could not find container \"c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6\": container with ID starting with c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6 not found: ID does not exist"
Jul 07 11:42:48.061053 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.061048    2581 scope.go:117] "RemoveContainer" containerID="f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb"
Jul 07 11:42:48.061307 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.061288    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb"} err="failed to get container status \"f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb\": rpc error: code = NotFound desc = could not find container \"f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb\": container with ID starting with f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb not found: ID does not exist"
Jul 07 11:42:48.061390 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.061307    2581 scope.go:117] "RemoveContainer" containerID="f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5"
Jul 07 11:42:48.061569 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.061547    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5"} err="failed to get container status \"f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5\": rpc error: code = NotFound desc = could not find container \"f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5\": container with ID starting with f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5 not found: ID does not exist"
Jul 07 11:42:48.061671 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.061573    2581 scope.go:117] "RemoveContainer" containerID="f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1"
Jul 07 11:42:48.061824 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.061802    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1"} err="failed to get container status \"f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1\": rpc error: code = NotFound desc = could not find container \"f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1\": container with ID starting with f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1 not found: ID does not exist"
Jul 07 11:42:48.061879 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.061823    2581 scope.go:117] "RemoveContainer" containerID="9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4"
Jul 07 11:42:48.062124 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.062101    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4"} err="failed to get container status \"9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4\": rpc error: code = NotFound desc = could not find container \"9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4\": container with ID starting with 9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4 not found: ID does not exist"
Jul 07 11:42:48.062124 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.062124    2581 scope.go:117] "RemoveContainer" containerID="fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce"
Jul 07 11:42:48.062351 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.062332    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce"} err="failed to get container status \"fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce\": rpc error: code = NotFound desc = could not find container \"fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce\": container with ID starting with fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce not found: ID does not exist"
Jul 07 11:42:48.062406 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.062353    2581 scope.go:117] "RemoveContainer" containerID="16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7"
Jul 07 11:42:48.062658 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.062633    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7"} err="failed to get container status \"16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7\": rpc error: code = NotFound desc = could not find container \"16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7\": container with ID starting with 16fe290bf31859eefe1415000b54e61fd507a2ecfe23c29e0e5d05f8f9d0ccc7 not found: ID does not exist"
Jul 07 11:42:48.062730 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.062661    2581 scope.go:117] "RemoveContainer" containerID="c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6"
Jul 07 11:42:48.062907 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.062882    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6"} err="failed to get container status \"c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6\": rpc error: code = NotFound desc = could not find container \"c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6\": container with ID starting with c123632e49da479b7ea3b7d25b1bd5854bba2ec65da252f03d42da82c3f55ba6 not found: ID does not exist"
Jul 07 11:42:48.062907 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.062905    2581 scope.go:117] "RemoveContainer" containerID="f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb"
Jul 07 11:42:48.063199 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.063178    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb"} err="failed to get container status \"f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb\": rpc error: code = NotFound desc = could not find container \"f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb\": container with ID starting with f5d3728f5a5ac8b90792ee0f8c0cfd14e6b55f72f1e4113e58b39f1fef4f78eb not found: ID does not exist"
Jul 07 11:42:48.063263 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.063200    2581 scope.go:117] "RemoveContainer" containerID="f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5"
Jul 07 11:42:48.063425 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.063403    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5"} err="failed to get container status \"f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5\": rpc error: code = NotFound desc = could not find container \"f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5\": container with ID starting with f99fe1c451333b50cbfcbfe5152c760a88465fe5c713eda6bcb873283e79baf5 not found: ID does not exist"
Jul 07 11:42:48.063487 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.063427    2581 scope.go:117] "RemoveContainer" containerID="f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1"
Jul 07 11:42:48.063682 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.063661    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1"} err="failed to get container status \"f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1\": rpc error: code = NotFound desc = could not find container \"f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1\": container with ID starting with f98782a78a04c8165b0fccfb5eacabe481f0065b404c8f08db48feb54e7f3ff1 not found: ID does not exist"
Jul 07 11:42:48.063682 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.063683    2581 scope.go:117] "RemoveContainer" containerID="9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4"
Jul 07 11:42:48.063914 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.063897    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4"} err="failed to get container status \"9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4\": rpc error: code = NotFound desc = could not find container \"9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4\": container with ID starting with 9617d047c75b2df0ddde2e95afc0eef5f24f6ed7a169141acb7c8751e1990ca4 not found: ID does not exist"
Jul 07 11:42:48.063971 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.063915    2581 scope.go:117] "RemoveContainer" containerID="fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce"
Jul 07 11:42:48.064156 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.064136    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce"} err="failed to get container status \"fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce\": rpc error: code = NotFound desc = could not find container \"fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce\": container with ID starting with fd65d8c95187294cb3d7a5b13b6ebd4dda0b44060a750d841d48f071c884f4ce not found: ID does not exist"
Jul 07 11:42:48.109313 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.108177    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 11:42:48.109313 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.108553    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaa9bb81-dc87-4b74-be29-587ab84bf1e9" containerName="thanos-sidecar"
Jul 07 11:42:48.109313 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.108568    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaa9bb81-dc87-4b74-be29-587ab84bf1e9" containerName="thanos-sidecar"
Jul 07 11:42:48.109313 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.108582    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaa9bb81-dc87-4b74-be29-587ab84bf1e9" containerName="prometheus"
Jul 07 11:42:48.109313 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.108590    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaa9bb81-dc87-4b74-be29-587ab84bf1e9" containerName="prometheus"
Jul 07 11:42:48.109313 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.108610    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaa9bb81-dc87-4b74-be29-587ab84bf1e9" containerName="kube-rbac-proxy-thanos"
Jul 07 11:42:48.109313 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.108618    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaa9bb81-dc87-4b74-be29-587ab84bf1e9" containerName="kube-rbac-proxy-thanos"
Jul 07 11:42:48.109313 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.108627    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaa9bb81-dc87-4b74-be29-587ab84bf1e9" containerName="config-reloader"
Jul 07 11:42:48.109313 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.108635    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaa9bb81-dc87-4b74-be29-587ab84bf1e9" containerName="config-reloader"
Jul 07 11:42:48.109313 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.108643    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaa9bb81-dc87-4b74-be29-587ab84bf1e9" containerName="kube-rbac-proxy-web"
Jul 07 11:42:48.109313 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.108651    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaa9bb81-dc87-4b74-be29-587ab84bf1e9" containerName="kube-rbac-proxy-web"
Jul 07 11:42:48.109313 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.108664    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaa9bb81-dc87-4b74-be29-587ab84bf1e9" containerName="kube-rbac-proxy"
Jul 07 11:42:48.109313 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.108671    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaa9bb81-dc87-4b74-be29-587ab84bf1e9" containerName="kube-rbac-proxy"
Jul 07 11:42:48.109313 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.108681    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaa9bb81-dc87-4b74-be29-587ab84bf1e9" containerName="init-config-reloader"
Jul 07 11:42:48.109313 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.108689    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaa9bb81-dc87-4b74-be29-587ab84bf1e9" containerName="init-config-reloader"
Jul 07 11:42:48.109313 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.108752    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaa9bb81-dc87-4b74-be29-587ab84bf1e9" containerName="kube-rbac-proxy-thanos"
Jul 07 11:42:48.109313 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.108763    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaa9bb81-dc87-4b74-be29-587ab84bf1e9" containerName="config-reloader"
Jul 07 11:42:48.109313 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.108774    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaa9bb81-dc87-4b74-be29-587ab84bf1e9" containerName="kube-rbac-proxy-web"
Jul 07 11:42:48.109313 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.108783    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaa9bb81-dc87-4b74-be29-587ab84bf1e9" containerName="kube-rbac-proxy"
Jul 07 11:42:48.109313 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.108793    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaa9bb81-dc87-4b74-be29-587ab84bf1e9" containerName="prometheus"
Jul 07 11:42:48.109313 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.108801    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaa9bb81-dc87-4b74-be29-587ab84bf1e9" containerName="thanos-sidecar"
Jul 07 11:42:48.113097 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.113077    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.119089 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.119069    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 07 11:42:48.119089 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.119082    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 07 11:42:48.119245 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.119082    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 07 11:42:48.119401 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.119387    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 07 11:42:48.119529 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.119514    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 07 11:42:48.119780 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.119762    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 07 11:42:48.121055 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.120851    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 07 11:42:48.121168 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.121030    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 07 11:42:48.121578 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.121537    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cm6d30d9a76h0\""
Jul 07 11:42:48.121673 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.121650    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-pv2bn\""
Jul 07 11:42:48.122203 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.121982    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 07 11:42:48.123428 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.123407    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 07 11:42:48.123809 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.123790    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 07 11:42:48.123916 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.123877    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 07 11:42:48.126799 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.126781    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 07 11:42:48.131271 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.131251    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 07 11:42:48.140300 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.140278    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 11:42:48.171072 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.171050    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aaa9bb81-dc87-4b74-be29-587ab84bf1e9" path="/var/lib/kubelet/pods/aaa9bb81-dc87-4b74-be29-587ab84bf1e9/volumes"
Jul 07 11:42:48.233533 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.233509    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-config\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.233636 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.233558    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.233636 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.233589    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.233636 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.233615    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.233807 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.233643    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-web-config\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.233807 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.233672    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.233807 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.233721    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.233807 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.233770    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-config-out\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.233807 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.233804    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.234057 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.233824    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.234057 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.233849    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.234057 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.233953    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.234057 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.233988    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.234057 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.234033    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.234233 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.234061    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.234233 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.234089    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkggb\" (UniqueName: \"kubernetes.io/projected/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-kube-api-access-xkggb\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.234233 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.234114    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.234233 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.234138    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.234233 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.234171    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.234233 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.234225    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.334887 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.334849    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.335069 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.334894    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.335069 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.334915    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.335069 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.334933    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.335069 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.334966    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkggb\" (UniqueName: \"kubernetes.io/projected/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-kube-api-access-xkggb\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.335069 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.334990    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.335561 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.335364    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.335561 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.335412    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.335561 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.335497    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.335561 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.335529    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-config\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.335846 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.335571    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.335846 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.335628    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.335846 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.335673    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.335846 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.335702    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-web-config\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.335846 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.335704    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.336134 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.335902    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.336683 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.336357    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.336683 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.336428    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.336683 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.336469    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.336683 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.336507    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-config-out\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.336683 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.336542    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.336683 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.336568    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.336683 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.336598    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.339258 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.338230    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.339258 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.338302    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.339258 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.338752    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.339258 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.338876    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.340076 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.339818    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.340306 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.340284    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-config\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.340374 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.340310    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.340562 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.340539    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.341125 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.341069    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.341125 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.341086    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.341274 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.341239    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.341909 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.341865    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.342186 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.342142    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.342352 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.342330    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-config-out\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.342662 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.342639    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-web-config\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.343333 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.343317    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.346560 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.346540    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkggb\" (UniqueName: \"kubernetes.io/projected/3ffe3c1b-fea5-4bfd-8a62-11590159a96a-kube-api-access-xkggb\") pod \"prometheus-k8s-0\" (UID: \"3ffe3c1b-fea5-4bfd-8a62-11590159a96a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.424922 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.424840    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:42:48.584037 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:48.583986    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 11:42:48.585960 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:42:48.585931    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ffe3c1b_fea5_4bfd_8a62_11590159a96a.slice/crio-1c2d364e269cf886e191b3ec2a3fe10864fa863cb9688c639cbfd005a073d3e2 WatchSource:0}: Error finding container 1c2d364e269cf886e191b3ec2a3fe10864fa863cb9688c639cbfd005a073d3e2: Status 404 returned error can't find the container with id 1c2d364e269cf886e191b3ec2a3fe10864fa863cb9688c639cbfd005a073d3e2
Jul 07 11:42:49.008167 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:49.008135    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ffe3c1b-fea5-4bfd-8a62-11590159a96a","Type":"ContainerStarted","Data":"57c71e5f844c8500e69957d430ea90863265e740383ba4d6a842e0f2c94cb3ea"}
Jul 07 11:42:49.008604 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:49.008211    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ffe3c1b-fea5-4bfd-8a62-11590159a96a","Type":"ContainerStarted","Data":"1c2d364e269cf886e191b3ec2a3fe10864fa863cb9688c639cbfd005a073d3e2"}
Jul 07 11:42:50.013253 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:50.013215    2581 generic.go:356] "Generic (PLEG): container finished" podID="3ffe3c1b-fea5-4bfd-8a62-11590159a96a" containerID="57c71e5f844c8500e69957d430ea90863265e740383ba4d6a842e0f2c94cb3ea" exitCode=0
Jul 07 11:42:50.013661 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:50.013305    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ffe3c1b-fea5-4bfd-8a62-11590159a96a","Type":"ContainerDied","Data":"57c71e5f844c8500e69957d430ea90863265e740383ba4d6a842e0f2c94cb3ea"}
Jul 07 11:42:50.015533 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:50.015504    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-666b56dbc-kp4zb" event={"ID":"2af33142-052d-442a-b5bc-e2767c7c6a14","Type":"ContainerStarted","Data":"d4f6dfdff83e5393f62ddd27f073e1fc03898c91c37e2b80d231a2d266c37074"}
Jul 07 11:42:50.015646 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:50.015556    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-666b56dbc-kp4zb" event={"ID":"2af33142-052d-442a-b5bc-e2767c7c6a14","Type":"ContainerStarted","Data":"420e59f462547037dd89bc846a3d0666cb14ea6bb2151733aeb51701c076c125"}
Jul 07 11:42:50.015646 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:50.015584    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-666b56dbc-kp4zb" event={"ID":"2af33142-052d-442a-b5bc-e2767c7c6a14","Type":"ContainerStarted","Data":"00ce7d719539b037d9f4af7372fbff621bfb4786ea25b16c52b031bfe1a4dca7"}
Jul 07 11:42:50.070906 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:50.070867    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-666b56dbc-kp4zb" podStartSLOduration=2.4291804470000002 podStartE2EDuration="4.070850358s" totalImagesPullingTime="1.641669911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:42:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:42:50.068563351 +0000 UTC m=+160.415777000" watchObservedRunningTime="2026-07-07 11:42:50.070850358 +0000 UTC m=+160.418063975"
Jul 07 11:42:51.024072 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:51.024033    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ffe3c1b-fea5-4bfd-8a62-11590159a96a","Type":"ContainerStarted","Data":"8b6904e7914f87bd5227a46f86ae15e69d81dc75872522a9509cbe89b39ba904"}
Jul 07 11:42:51.024485 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:51.024116    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ffe3c1b-fea5-4bfd-8a62-11590159a96a","Type":"ContainerStarted","Data":"7b5d4ed4201a4e42257a6f3f052681b88c53d161b9dd27c6336d2fa26c09d9ef"}
Jul 07 11:42:51.024485 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:51.024160    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ffe3c1b-fea5-4bfd-8a62-11590159a96a","Type":"ContainerStarted","Data":"f8425e943c60190c24dbae7b04c4b399e8dc53b180fd8ce6f7f487cbafe371d7"}
Jul 07 11:42:51.024485 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:51.024194    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ffe3c1b-fea5-4bfd-8a62-11590159a96a","Type":"ContainerStarted","Data":"400a6760cc75cecdd9c9db276fa491f90ae4ff9887ea246fb9318b695052a5ab"}
Jul 07 11:42:51.024485 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:51.024224    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ffe3c1b-fea5-4bfd-8a62-11590159a96a","Type":"ContainerStarted","Data":"ab3e3249af5257eac6dc5de4d60d11b2360145acfd04db9d51c826fb7f7d731d"}
Jul 07 11:42:51.024485 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:51.024251    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ffe3c1b-fea5-4bfd-8a62-11590159a96a","Type":"ContainerStarted","Data":"b3ca6b6d192bb0fb7fa336b0effff423db5785074e7d83c0aa089077d6f0175a"}
Jul 07 11:42:51.072597 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:51.072530    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.052524837 podStartE2EDuration="3.072503784s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.978947ms" isStatefulPod=false podCreationTimestamp="2026-07-07 11:42:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:42:51.071079298 +0000 UTC m=+161.418292925" watchObservedRunningTime="2026-07-07 11:42:51.072503784 +0000 UTC m=+161.419717410"
Jul 07 11:42:53.424960 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:42:53.424925    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:43:48.425265 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:43:48.425225    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:43:48.440615 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:43:48.440591    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:43:49.249339 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:43:49.249315    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 11:45:12.014624 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:12.014601    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kf48j_4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801/ovn-acl-logging/0.log"
Jul 07 11:45:12.017625 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:12.017605    2581 kubelet.go:1648] "Image garbage collection succeeded"
Jul 07 11:45:12.017753 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:12.017727    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kf48j_4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801/ovn-acl-logging/0.log"
Jul 07 11:45:34.490050 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:34.490003    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589kj49w"]
Jul 07 11:45:34.492345 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:34.492329    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589kj49w"
Jul 07 11:45:34.495933 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:34.495911    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 07 11:45:34.496053 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:34.495939    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 07 11:45:34.497337 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:34.497316    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hrsbq\""
Jul 07 11:45:34.504829 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:34.504806    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589kj49w"]
Jul 07 11:45:34.606100 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:34.606069    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x66gv\" (UniqueName: \"kubernetes.io/projected/329e9efe-2a7a-4881-8cfe-0599a2a34b39-kube-api-access-x66gv\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589kj49w\" (UID: \"329e9efe-2a7a-4881-8cfe-0599a2a34b39\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589kj49w"
Jul 07 11:45:34.606250 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:34.606106    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/329e9efe-2a7a-4881-8cfe-0599a2a34b39-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589kj49w\" (UID: \"329e9efe-2a7a-4881-8cfe-0599a2a34b39\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589kj49w"
Jul 07 11:45:34.606250 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:34.606143    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/329e9efe-2a7a-4881-8cfe-0599a2a34b39-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589kj49w\" (UID: \"329e9efe-2a7a-4881-8cfe-0599a2a34b39\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589kj49w"
Jul 07 11:45:34.707005 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:34.706977    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x66gv\" (UniqueName: \"kubernetes.io/projected/329e9efe-2a7a-4881-8cfe-0599a2a34b39-kube-api-access-x66gv\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589kj49w\" (UID: \"329e9efe-2a7a-4881-8cfe-0599a2a34b39\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589kj49w"
Jul 07 11:45:34.707181 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:34.707035    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/329e9efe-2a7a-4881-8cfe-0599a2a34b39-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589kj49w\" (UID: \"329e9efe-2a7a-4881-8cfe-0599a2a34b39\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589kj49w"
Jul 07 11:45:34.707181 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:34.707086    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/329e9efe-2a7a-4881-8cfe-0599a2a34b39-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589kj49w\" (UID: \"329e9efe-2a7a-4881-8cfe-0599a2a34b39\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589kj49w"
Jul 07 11:45:34.707451 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:34.707429    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/329e9efe-2a7a-4881-8cfe-0599a2a34b39-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589kj49w\" (UID: \"329e9efe-2a7a-4881-8cfe-0599a2a34b39\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589kj49w"
Jul 07 11:45:34.707513 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:34.707463    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/329e9efe-2a7a-4881-8cfe-0599a2a34b39-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589kj49w\" (UID: \"329e9efe-2a7a-4881-8cfe-0599a2a34b39\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589kj49w"
Jul 07 11:45:34.716706 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:34.716686    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x66gv\" (UniqueName: \"kubernetes.io/projected/329e9efe-2a7a-4881-8cfe-0599a2a34b39-kube-api-access-x66gv\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589kj49w\" (UID: \"329e9efe-2a7a-4881-8cfe-0599a2a34b39\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589kj49w"
Jul 07 11:45:34.800787 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:34.800721    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589kj49w"
Jul 07 11:45:34.949410 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:34.949386    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589kj49w"]
Jul 07 11:45:34.951865 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:45:34.951828    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod329e9efe_2a7a_4881_8cfe_0599a2a34b39.slice/crio-39e796f62593e0c4fa83918a11142a353473461d912f3c86353c5de8c2e3bd22 WatchSource:0}: Error finding container 39e796f62593e0c4fa83918a11142a353473461d912f3c86353c5de8c2e3bd22: Status 404 returned error can't find the container with id 39e796f62593e0c4fa83918a11142a353473461d912f3c86353c5de8c2e3bd22
Jul 07 11:45:34.953750 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:34.953729    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 11:45:35.617282 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:35.617247    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589kj49w" event={"ID":"329e9efe-2a7a-4881-8cfe-0599a2a34b39","Type":"ContainerStarted","Data":"39e796f62593e0c4fa83918a11142a353473461d912f3c86353c5de8c2e3bd22"}
Jul 07 11:45:40.641000 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:40.640961    2581 generic.go:356] "Generic (PLEG): container finished" podID="329e9efe-2a7a-4881-8cfe-0599a2a34b39" containerID="7f7b978ea05e630aa02a505c89eabac0c0e989bc34597d19d640a5b7c2feadb3" exitCode=0
Jul 07 11:45:40.641417 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:40.641033    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589kj49w" event={"ID":"329e9efe-2a7a-4881-8cfe-0599a2a34b39","Type":"ContainerDied","Data":"7f7b978ea05e630aa02a505c89eabac0c0e989bc34597d19d640a5b7c2feadb3"}
Jul 07 11:45:42.652407 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:42.652370    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589kj49w" event={"ID":"329e9efe-2a7a-4881-8cfe-0599a2a34b39","Type":"ContainerStarted","Data":"2f1b60cf6d5dcee87fb583fb42c4bddd63328b737cf596fe6b32679bd1c062fa"}
Jul 07 11:45:43.661838 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:43.661801    2581 generic.go:356] "Generic (PLEG): container finished" podID="329e9efe-2a7a-4881-8cfe-0599a2a34b39" containerID="2f1b60cf6d5dcee87fb583fb42c4bddd63328b737cf596fe6b32679bd1c062fa" exitCode=0
Jul 07 11:45:43.662247 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:43.661854    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589kj49w" event={"ID":"329e9efe-2a7a-4881-8cfe-0599a2a34b39","Type":"ContainerDied","Data":"2f1b60cf6d5dcee87fb583fb42c4bddd63328b737cf596fe6b32679bd1c062fa"}
Jul 07 11:45:49.695580 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:49.695493    2581 generic.go:356] "Generic (PLEG): container finished" podID="329e9efe-2a7a-4881-8cfe-0599a2a34b39" containerID="849b1066e3358bbc0bc65f89baf780f67750979e916f86e4917fe93a86c739a3" exitCode=0
Jul 07 11:45:49.695580 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:49.695541    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589kj49w" event={"ID":"329e9efe-2a7a-4881-8cfe-0599a2a34b39","Type":"ContainerDied","Data":"849b1066e3358bbc0bc65f89baf780f67750979e916f86e4917fe93a86c739a3"}
Jul 07 11:45:50.829259 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:50.829234    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589kj49w"
Jul 07 11:45:50.950109 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:50.950032    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/329e9efe-2a7a-4881-8cfe-0599a2a34b39-bundle\") pod \"329e9efe-2a7a-4881-8cfe-0599a2a34b39\" (UID: \"329e9efe-2a7a-4881-8cfe-0599a2a34b39\") "
Jul 07 11:45:50.950109 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:50.950099    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x66gv\" (UniqueName: \"kubernetes.io/projected/329e9efe-2a7a-4881-8cfe-0599a2a34b39-kube-api-access-x66gv\") pod \"329e9efe-2a7a-4881-8cfe-0599a2a34b39\" (UID: \"329e9efe-2a7a-4881-8cfe-0599a2a34b39\") "
Jul 07 11:45:50.950332 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:50.950143    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/329e9efe-2a7a-4881-8cfe-0599a2a34b39-util\") pod \"329e9efe-2a7a-4881-8cfe-0599a2a34b39\" (UID: \"329e9efe-2a7a-4881-8cfe-0599a2a34b39\") "
Jul 07 11:45:50.950629 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:50.950605    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/329e9efe-2a7a-4881-8cfe-0599a2a34b39-bundle" (OuterVolumeSpecName: "bundle") pod "329e9efe-2a7a-4881-8cfe-0599a2a34b39" (UID: "329e9efe-2a7a-4881-8cfe-0599a2a34b39"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 11:45:50.952253 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:50.952232    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/329e9efe-2a7a-4881-8cfe-0599a2a34b39-kube-api-access-x66gv" (OuterVolumeSpecName: "kube-api-access-x66gv") pod "329e9efe-2a7a-4881-8cfe-0599a2a34b39" (UID: "329e9efe-2a7a-4881-8cfe-0599a2a34b39"). InnerVolumeSpecName "kube-api-access-x66gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 11:45:50.954155 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:50.954130    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/329e9efe-2a7a-4881-8cfe-0599a2a34b39-util" (OuterVolumeSpecName: "util") pod "329e9efe-2a7a-4881-8cfe-0599a2a34b39" (UID: "329e9efe-2a7a-4881-8cfe-0599a2a34b39"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 11:45:51.050675 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:51.050645    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x66gv\" (UniqueName: \"kubernetes.io/projected/329e9efe-2a7a-4881-8cfe-0599a2a34b39-kube-api-access-x66gv\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:45:51.050675 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:51.050668    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/329e9efe-2a7a-4881-8cfe-0599a2a34b39-util\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:45:51.050675 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:51.050678    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/329e9efe-2a7a-4881-8cfe-0599a2a34b39-bundle\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:45:51.707482 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:51.707445    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589kj49w"
Jul 07 11:45:51.707482 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:51.707457    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589kj49w" event={"ID":"329e9efe-2a7a-4881-8cfe-0599a2a34b39","Type":"ContainerDied","Data":"39e796f62593e0c4fa83918a11142a353473461d912f3c86353c5de8c2e3bd22"}
Jul 07 11:45:51.707685 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:45:51.707507    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39e796f62593e0c4fa83918a11142a353473461d912f3c86353c5de8c2e3bd22"
Jul 07 11:47:08.664859 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:08.664828    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-57676b4bd8-lb458"]
Jul 07 11:47:08.665298 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:08.665181    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="329e9efe-2a7a-4881-8cfe-0599a2a34b39" containerName="pull"
Jul 07 11:47:08.665298 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:08.665193    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="329e9efe-2a7a-4881-8cfe-0599a2a34b39" containerName="pull"
Jul 07 11:47:08.665298 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:08.665211    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="329e9efe-2a7a-4881-8cfe-0599a2a34b39" containerName="extract"
Jul 07 11:47:08.665298 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:08.665219    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="329e9efe-2a7a-4881-8cfe-0599a2a34b39" containerName="extract"
Jul 07 11:47:08.665298 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:08.665230    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="329e9efe-2a7a-4881-8cfe-0599a2a34b39" containerName="util"
Jul 07 11:47:08.665298 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:08.665235    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="329e9efe-2a7a-4881-8cfe-0599a2a34b39" containerName="util"
Jul 07 11:47:08.665298 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:08.665285    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="329e9efe-2a7a-4881-8cfe-0599a2a34b39" containerName="extract"
Jul 07 11:47:08.667205 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:08.667189    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-57676b4bd8-lb458"
Jul 07 11:47:08.670157 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:08.670135    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 07 11:47:08.670284 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:08.670168    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 07 11:47:08.670337 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:08.670283    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-fx4v5\""
Jul 07 11:47:08.671136 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:08.671116    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdb6l\" (UniqueName: \"kubernetes.io/projected/6e950872-c47b-451a-8993-74a2f99d5eea-kube-api-access-sdb6l\") pod \"kserve-controller-manager-57676b4bd8-lb458\" (UID: \"6e950872-c47b-451a-8993-74a2f99d5eea\") " pod="kserve/kserve-controller-manager-57676b4bd8-lb458"
Jul 07 11:47:08.671248 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:08.671170    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e950872-c47b-451a-8993-74a2f99d5eea-cert\") pod \"kserve-controller-manager-57676b4bd8-lb458\" (UID: \"6e950872-c47b-451a-8993-74a2f99d5eea\") " pod="kserve/kserve-controller-manager-57676b4bd8-lb458"
Jul 07 11:47:08.671431 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:08.671418    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 07 11:47:08.679247 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:08.679224    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-57676b4bd8-lb458"]
Jul 07 11:47:08.772129 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:08.772100    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e950872-c47b-451a-8993-74a2f99d5eea-cert\") pod \"kserve-controller-manager-57676b4bd8-lb458\" (UID: \"6e950872-c47b-451a-8993-74a2f99d5eea\") " pod="kserve/kserve-controller-manager-57676b4bd8-lb458"
Jul 07 11:47:08.772266 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:08.772152    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdb6l\" (UniqueName: \"kubernetes.io/projected/6e950872-c47b-451a-8993-74a2f99d5eea-kube-api-access-sdb6l\") pod \"kserve-controller-manager-57676b4bd8-lb458\" (UID: \"6e950872-c47b-451a-8993-74a2f99d5eea\") " pod="kserve/kserve-controller-manager-57676b4bd8-lb458"
Jul 07 11:47:08.772266 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:47:08.772233    2581 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 07 11:47:08.772341 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:47:08.772303    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e950872-c47b-451a-8993-74a2f99d5eea-cert podName:6e950872-c47b-451a-8993-74a2f99d5eea nodeName:}" failed. No retries permitted until 2026-07-07 11:47:09.272287318 +0000 UTC m=+419.619500919 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6e950872-c47b-451a-8993-74a2f99d5eea-cert") pod "kserve-controller-manager-57676b4bd8-lb458" (UID: "6e950872-c47b-451a-8993-74a2f99d5eea") : secret "kserve-webhook-server-cert" not found
Jul 07 11:47:08.783932 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:08.783902    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdb6l\" (UniqueName: \"kubernetes.io/projected/6e950872-c47b-451a-8993-74a2f99d5eea-kube-api-access-sdb6l\") pod \"kserve-controller-manager-57676b4bd8-lb458\" (UID: \"6e950872-c47b-451a-8993-74a2f99d5eea\") " pod="kserve/kserve-controller-manager-57676b4bd8-lb458"
Jul 07 11:47:09.276328 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:09.276288    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e950872-c47b-451a-8993-74a2f99d5eea-cert\") pod \"kserve-controller-manager-57676b4bd8-lb458\" (UID: \"6e950872-c47b-451a-8993-74a2f99d5eea\") " pod="kserve/kserve-controller-manager-57676b4bd8-lb458"
Jul 07 11:47:09.278403 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:09.278381    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e950872-c47b-451a-8993-74a2f99d5eea-cert\") pod \"kserve-controller-manager-57676b4bd8-lb458\" (UID: \"6e950872-c47b-451a-8993-74a2f99d5eea\") " pod="kserve/kserve-controller-manager-57676b4bd8-lb458"
Jul 07 11:47:09.576771 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:09.576691    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-57676b4bd8-lb458"
Jul 07 11:47:09.700506 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:09.700483    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-57676b4bd8-lb458"]
Jul 07 11:47:09.702367 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:47:09.702343    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e950872_c47b_451a_8993_74a2f99d5eea.slice/crio-363996891a4bd47daabcffcd4be61fc7584ee0a3bad822a0c80b988646df3694 WatchSource:0}: Error finding container 363996891a4bd47daabcffcd4be61fc7584ee0a3bad822a0c80b988646df3694: Status 404 returned error can't find the container with id 363996891a4bd47daabcffcd4be61fc7584ee0a3bad822a0c80b988646df3694
Jul 07 11:47:10.004175 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:10.004143    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-57676b4bd8-lb458" event={"ID":"6e950872-c47b-451a-8993-74a2f99d5eea","Type":"ContainerStarted","Data":"363996891a4bd47daabcffcd4be61fc7584ee0a3bad822a0c80b988646df3694"}
Jul 07 11:47:13.019435 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:13.019401    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-57676b4bd8-lb458" event={"ID":"6e950872-c47b-451a-8993-74a2f99d5eea","Type":"ContainerStarted","Data":"bd33486b633855f4c275a7cce16972221f04dab7a5a11e90d1e50a193b56b117"}
Jul 07 11:47:13.019798 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:13.019494    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-57676b4bd8-lb458"
Jul 07 11:47:13.037262 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:13.037219    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-57676b4bd8-lb458" podStartSLOduration=2.745514404 podStartE2EDuration="5.037206788s" totalImagesPullingTime="2.291692384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:47:13.036640821 +0000 UTC m=+423.383854444" watchObservedRunningTime="2026-07-07 11:47:13.037206788 +0000 UTC m=+423.384420411"
Jul 07 11:47:44.028381 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:44.028350    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-57676b4bd8-lb458"
Jul 07 11:47:44.637911 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:44.637877    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-57676b4bd8-lb458"]
Jul 07 11:47:44.638102 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:44.638073    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/kserve-controller-manager-57676b4bd8-lb458" podUID="6e950872-c47b-451a-8993-74a2f99d5eea" containerName="manager" containerID="cri-o://bd33486b633855f4c275a7cce16972221f04dab7a5a11e90d1e50a193b56b117" gracePeriod=10
Jul 07 11:47:44.661662 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:44.661638    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-57676b4bd8-c4mpb"]
Jul 07 11:47:44.663642 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:44.663622    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-57676b4bd8-c4mpb"
Jul 07 11:47:44.675387 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:44.675359    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-57676b4bd8-c4mpb"]
Jul 07 11:47:44.771517 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:44.771489    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f63dc8ee-c323-4fe4-931f-627df2115c9f-cert\") pod \"kserve-controller-manager-57676b4bd8-c4mpb\" (UID: \"f63dc8ee-c323-4fe4-931f-627df2115c9f\") " pod="kserve/kserve-controller-manager-57676b4bd8-c4mpb"
Jul 07 11:47:44.771649 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:44.771532    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj2xv\" (UniqueName: \"kubernetes.io/projected/f63dc8ee-c323-4fe4-931f-627df2115c9f-kube-api-access-xj2xv\") pod \"kserve-controller-manager-57676b4bd8-c4mpb\" (UID: \"f63dc8ee-c323-4fe4-931f-627df2115c9f\") " pod="kserve/kserve-controller-manager-57676b4bd8-c4mpb"
Jul 07 11:47:44.872572 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:44.872538    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f63dc8ee-c323-4fe4-931f-627df2115c9f-cert\") pod \"kserve-controller-manager-57676b4bd8-c4mpb\" (UID: \"f63dc8ee-c323-4fe4-931f-627df2115c9f\") " pod="kserve/kserve-controller-manager-57676b4bd8-c4mpb"
Jul 07 11:47:44.872684 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:44.872592    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj2xv\" (UniqueName: \"kubernetes.io/projected/f63dc8ee-c323-4fe4-931f-627df2115c9f-kube-api-access-xj2xv\") pod \"kserve-controller-manager-57676b4bd8-c4mpb\" (UID: \"f63dc8ee-c323-4fe4-931f-627df2115c9f\") " pod="kserve/kserve-controller-manager-57676b4bd8-c4mpb"
Jul 07 11:47:44.874871 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:44.874848    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f63dc8ee-c323-4fe4-931f-627df2115c9f-cert\") pod \"kserve-controller-manager-57676b4bd8-c4mpb\" (UID: \"f63dc8ee-c323-4fe4-931f-627df2115c9f\") " pod="kserve/kserve-controller-manager-57676b4bd8-c4mpb"
Jul 07 11:47:44.875323 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:44.875304    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-57676b4bd8-lb458"
Jul 07 11:47:44.881458 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:44.881439    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj2xv\" (UniqueName: \"kubernetes.io/projected/f63dc8ee-c323-4fe4-931f-627df2115c9f-kube-api-access-xj2xv\") pod \"kserve-controller-manager-57676b4bd8-c4mpb\" (UID: \"f63dc8ee-c323-4fe4-931f-627df2115c9f\") " pod="kserve/kserve-controller-manager-57676b4bd8-c4mpb"
Jul 07 11:47:44.973402 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:44.973328    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e950872-c47b-451a-8993-74a2f99d5eea-cert\") pod \"6e950872-c47b-451a-8993-74a2f99d5eea\" (UID: \"6e950872-c47b-451a-8993-74a2f99d5eea\") "
Jul 07 11:47:44.973554 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:44.973422    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdb6l\" (UniqueName: \"kubernetes.io/projected/6e950872-c47b-451a-8993-74a2f99d5eea-kube-api-access-sdb6l\") pod \"6e950872-c47b-451a-8993-74a2f99d5eea\" (UID: \"6e950872-c47b-451a-8993-74a2f99d5eea\") "
Jul 07 11:47:44.975415 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:44.975386    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e950872-c47b-451a-8993-74a2f99d5eea-cert" (OuterVolumeSpecName: "cert") pod "6e950872-c47b-451a-8993-74a2f99d5eea" (UID: "6e950872-c47b-451a-8993-74a2f99d5eea"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:47:44.975521 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:44.975424    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e950872-c47b-451a-8993-74a2f99d5eea-kube-api-access-sdb6l" (OuterVolumeSpecName: "kube-api-access-sdb6l") pod "6e950872-c47b-451a-8993-74a2f99d5eea" (UID: "6e950872-c47b-451a-8993-74a2f99d5eea"). InnerVolumeSpecName "kube-api-access-sdb6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 11:47:45.026215 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:45.026177    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-57676b4bd8-c4mpb"
Jul 07 11:47:45.074530 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:45.074495    2581 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e950872-c47b-451a-8993-74a2f99d5eea-cert\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:47:45.074946 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:45.074532    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdb6l\" (UniqueName: \"kubernetes.io/projected/6e950872-c47b-451a-8993-74a2f99d5eea-kube-api-access-sdb6l\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:47:45.144644 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:45.144609    2581 generic.go:356] "Generic (PLEG): container finished" podID="6e950872-c47b-451a-8993-74a2f99d5eea" containerID="bd33486b633855f4c275a7cce16972221f04dab7a5a11e90d1e50a193b56b117" exitCode=0
Jul 07 11:47:45.144805 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:45.144655    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-57676b4bd8-lb458"
Jul 07 11:47:45.144805 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:45.144695    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-57676b4bd8-lb458" event={"ID":"6e950872-c47b-451a-8993-74a2f99d5eea","Type":"ContainerDied","Data":"bd33486b633855f4c275a7cce16972221f04dab7a5a11e90d1e50a193b56b117"}
Jul 07 11:47:45.144805 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:45.144751    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-57676b4bd8-lb458" event={"ID":"6e950872-c47b-451a-8993-74a2f99d5eea","Type":"ContainerDied","Data":"363996891a4bd47daabcffcd4be61fc7584ee0a3bad822a0c80b988646df3694"}
Jul 07 11:47:45.144805 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:45.144780    2581 scope.go:117] "RemoveContainer" containerID="bd33486b633855f4c275a7cce16972221f04dab7a5a11e90d1e50a193b56b117"
Jul 07 11:47:45.148974 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:45.148953    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-57676b4bd8-c4mpb"]
Jul 07 11:47:45.151663 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:47:45.151630    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf63dc8ee_c323_4fe4_931f_627df2115c9f.slice/crio-ea150b0122cb59cbfb8ffb844f26a219178e0a2d39b4abc7900b456629f473ca WatchSource:0}: Error finding container ea150b0122cb59cbfb8ffb844f26a219178e0a2d39b4abc7900b456629f473ca: Status 404 returned error can't find the container with id ea150b0122cb59cbfb8ffb844f26a219178e0a2d39b4abc7900b456629f473ca
Jul 07 11:47:45.154193 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:45.154167    2581 scope.go:117] "RemoveContainer" containerID="bd33486b633855f4c275a7cce16972221f04dab7a5a11e90d1e50a193b56b117"
Jul 07 11:47:45.154457 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:47:45.154439    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd33486b633855f4c275a7cce16972221f04dab7a5a11e90d1e50a193b56b117\": container with ID starting with bd33486b633855f4c275a7cce16972221f04dab7a5a11e90d1e50a193b56b117 not found: ID does not exist" containerID="bd33486b633855f4c275a7cce16972221f04dab7a5a11e90d1e50a193b56b117"
Jul 07 11:47:45.154522 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:45.154470    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd33486b633855f4c275a7cce16972221f04dab7a5a11e90d1e50a193b56b117"} err="failed to get container status \"bd33486b633855f4c275a7cce16972221f04dab7a5a11e90d1e50a193b56b117\": rpc error: code = NotFound desc = could not find container \"bd33486b633855f4c275a7cce16972221f04dab7a5a11e90d1e50a193b56b117\": container with ID starting with bd33486b633855f4c275a7cce16972221f04dab7a5a11e90d1e50a193b56b117 not found: ID does not exist"
Jul 07 11:47:45.169066 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:45.169044    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-57676b4bd8-lb458"]
Jul 07 11:47:45.172447 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:45.172428    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-57676b4bd8-lb458"]
Jul 07 11:47:46.149788 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:46.149759    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-57676b4bd8-c4mpb" event={"ID":"f63dc8ee-c323-4fe4-931f-627df2115c9f","Type":"ContainerStarted","Data":"ea150b0122cb59cbfb8ffb844f26a219178e0a2d39b4abc7900b456629f473ca"}
Jul 07 11:47:46.171212 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:46.171183    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e950872-c47b-451a-8993-74a2f99d5eea" path="/var/lib/kubelet/pods/6e950872-c47b-451a-8993-74a2f99d5eea/volumes"
Jul 07 11:47:47.155636 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:47.155602    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-57676b4bd8-c4mpb" event={"ID":"f63dc8ee-c323-4fe4-931f-627df2115c9f","Type":"ContainerStarted","Data":"de5118a534c6c1426e59ce2d587dee2567f040602e39d1f56ee91c80d0abd743"}
Jul 07 11:47:47.155995 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:47.155671    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-57676b4bd8-c4mpb"
Jul 07 11:47:47.177057 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:47:47.176972    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-57676b4bd8-c4mpb" podStartSLOduration=1.982029561 podStartE2EDuration="3.176953948s" totalImagesPullingTime="1.194924387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:47:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:47:47.176050115 +0000 UTC m=+457.523263739" watchObservedRunningTime="2026-07-07 11:47:47.176953948 +0000 UTC m=+457.524167572"
Jul 07 11:48:18.170592 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:18.170563    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-57676b4bd8-c4mpb"
Jul 07 11:48:19.158222 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:19.158181    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-kbsc8"]
Jul 07 11:48:19.158503 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:19.158491    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e950872-c47b-451a-8993-74a2f99d5eea" containerName="manager"
Jul 07 11:48:19.158547 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:19.158506    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e950872-c47b-451a-8993-74a2f99d5eea" containerName="manager"
Jul 07 11:48:19.158580 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:19.158557    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e950872-c47b-451a-8993-74a2f99d5eea" containerName="manager"
Jul 07 11:48:19.161300 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:19.161282    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-kbsc8"
Jul 07 11:48:19.164292 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:19.164268    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-vx5dn\""
Jul 07 11:48:19.164388 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:19.164269    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 07 11:48:19.172632 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:19.172613    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-kbsc8"]
Jul 07 11:48:19.177360 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:19.177341    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-bqh87"]
Jul 07 11:48:19.179331 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:19.179316    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-bqh87"
Jul 07 11:48:19.182075 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:19.182058    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-gjqxh\""
Jul 07 11:48:19.182292 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:19.182269    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 07 11:48:19.188873 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:19.188851    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-bqh87"]
Jul 07 11:48:19.248299 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:19.248275    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq5xt\" (UniqueName: \"kubernetes.io/projected/47645c7d-9e9e-4682-a7ab-36dc6a0f0c9b-kube-api-access-sq5xt\") pod \"model-serving-api-fd65d7d6-kbsc8\" (UID: \"47645c7d-9e9e-4682-a7ab-36dc6a0f0c9b\") " pod="kserve/model-serving-api-fd65d7d6-kbsc8"
Jul 07 11:48:19.248442 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:19.248318    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47645c7d-9e9e-4682-a7ab-36dc6a0f0c9b-tls-certs\") pod \"model-serving-api-fd65d7d6-kbsc8\" (UID: \"47645c7d-9e9e-4682-a7ab-36dc6a0f0c9b\") " pod="kserve/model-serving-api-fd65d7d6-kbsc8"
Jul 07 11:48:19.248442 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:19.248394    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f02ca4e5-50d1-4b05-9a9b-41e28e5db5eb-cert\") pod \"odh-model-controller-84b54b7f97-bqh87\" (UID: \"f02ca4e5-50d1-4b05-9a9b-41e28e5db5eb\") " pod="kserve/odh-model-controller-84b54b7f97-bqh87"
Jul 07 11:48:19.248442 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:19.248430    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qdm8\" (UniqueName: \"kubernetes.io/projected/f02ca4e5-50d1-4b05-9a9b-41e28e5db5eb-kube-api-access-2qdm8\") pod \"odh-model-controller-84b54b7f97-bqh87\" (UID: \"f02ca4e5-50d1-4b05-9a9b-41e28e5db5eb\") " pod="kserve/odh-model-controller-84b54b7f97-bqh87"
Jul 07 11:48:19.349209 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:19.349177    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47645c7d-9e9e-4682-a7ab-36dc6a0f0c9b-tls-certs\") pod \"model-serving-api-fd65d7d6-kbsc8\" (UID: \"47645c7d-9e9e-4682-a7ab-36dc6a0f0c9b\") " pod="kserve/model-serving-api-fd65d7d6-kbsc8"
Jul 07 11:48:19.349358 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:19.349228    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f02ca4e5-50d1-4b05-9a9b-41e28e5db5eb-cert\") pod \"odh-model-controller-84b54b7f97-bqh87\" (UID: \"f02ca4e5-50d1-4b05-9a9b-41e28e5db5eb\") " pod="kserve/odh-model-controller-84b54b7f97-bqh87"
Jul 07 11:48:19.349358 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:19.349260    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qdm8\" (UniqueName: \"kubernetes.io/projected/f02ca4e5-50d1-4b05-9a9b-41e28e5db5eb-kube-api-access-2qdm8\") pod \"odh-model-controller-84b54b7f97-bqh87\" (UID: \"f02ca4e5-50d1-4b05-9a9b-41e28e5db5eb\") " pod="kserve/odh-model-controller-84b54b7f97-bqh87"
Jul 07 11:48:19.349358 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:48:19.349322    2581 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 07 11:48:19.349358 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:19.349334    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq5xt\" (UniqueName: \"kubernetes.io/projected/47645c7d-9e9e-4682-a7ab-36dc6a0f0c9b-kube-api-access-sq5xt\") pod \"model-serving-api-fd65d7d6-kbsc8\" (UID: \"47645c7d-9e9e-4682-a7ab-36dc6a0f0c9b\") " pod="kserve/model-serving-api-fd65d7d6-kbsc8"
Jul 07 11:48:19.349553 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:48:19.349376    2581 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 07 11:48:19.349553 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:48:19.349385    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47645c7d-9e9e-4682-a7ab-36dc6a0f0c9b-tls-certs podName:47645c7d-9e9e-4682-a7ab-36dc6a0f0c9b nodeName:}" failed. No retries permitted until 2026-07-07 11:48:19.849367422 +0000 UTC m=+490.196581024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/47645c7d-9e9e-4682-a7ab-36dc6a0f0c9b-tls-certs") pod "model-serving-api-fd65d7d6-kbsc8" (UID: "47645c7d-9e9e-4682-a7ab-36dc6a0f0c9b") : secret "model-serving-api-tls" not found
Jul 07 11:48:19.349553 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:48:19.349457    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f02ca4e5-50d1-4b05-9a9b-41e28e5db5eb-cert podName:f02ca4e5-50d1-4b05-9a9b-41e28e5db5eb nodeName:}" failed. No retries permitted until 2026-07-07 11:48:19.849439382 +0000 UTC m=+490.196652990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f02ca4e5-50d1-4b05-9a9b-41e28e5db5eb-cert") pod "odh-model-controller-84b54b7f97-bqh87" (UID: "f02ca4e5-50d1-4b05-9a9b-41e28e5db5eb") : secret "odh-model-controller-webhook-cert" not found
Jul 07 11:48:19.358327 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:19.358300    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qdm8\" (UniqueName: \"kubernetes.io/projected/f02ca4e5-50d1-4b05-9a9b-41e28e5db5eb-kube-api-access-2qdm8\") pod \"odh-model-controller-84b54b7f97-bqh87\" (UID: \"f02ca4e5-50d1-4b05-9a9b-41e28e5db5eb\") " pod="kserve/odh-model-controller-84b54b7f97-bqh87"
Jul 07 11:48:19.358418 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:19.358361    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq5xt\" (UniqueName: \"kubernetes.io/projected/47645c7d-9e9e-4682-a7ab-36dc6a0f0c9b-kube-api-access-sq5xt\") pod \"model-serving-api-fd65d7d6-kbsc8\" (UID: \"47645c7d-9e9e-4682-a7ab-36dc6a0f0c9b\") " pod="kserve/model-serving-api-fd65d7d6-kbsc8"
Jul 07 11:48:19.853827 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:19.853789    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47645c7d-9e9e-4682-a7ab-36dc6a0f0c9b-tls-certs\") pod \"model-serving-api-fd65d7d6-kbsc8\" (UID: \"47645c7d-9e9e-4682-a7ab-36dc6a0f0c9b\") " pod="kserve/model-serving-api-fd65d7d6-kbsc8"
Jul 07 11:48:19.854058 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:19.853839    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f02ca4e5-50d1-4b05-9a9b-41e28e5db5eb-cert\") pod \"odh-model-controller-84b54b7f97-bqh87\" (UID: \"f02ca4e5-50d1-4b05-9a9b-41e28e5db5eb\") " pod="kserve/odh-model-controller-84b54b7f97-bqh87"
Jul 07 11:48:19.856234 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:19.856210    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f02ca4e5-50d1-4b05-9a9b-41e28e5db5eb-cert\") pod \"odh-model-controller-84b54b7f97-bqh87\" (UID: \"f02ca4e5-50d1-4b05-9a9b-41e28e5db5eb\") " pod="kserve/odh-model-controller-84b54b7f97-bqh87"
Jul 07 11:48:19.856331 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:19.856210    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47645c7d-9e9e-4682-a7ab-36dc6a0f0c9b-tls-certs\") pod \"model-serving-api-fd65d7d6-kbsc8\" (UID: \"47645c7d-9e9e-4682-a7ab-36dc6a0f0c9b\") " pod="kserve/model-serving-api-fd65d7d6-kbsc8"
Jul 07 11:48:20.071289 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:20.071254    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-kbsc8"
Jul 07 11:48:20.089132 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:20.089110    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-bqh87"
Jul 07 11:48:20.212307 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:20.212282    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-kbsc8"]
Jul 07 11:48:20.214082 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:48:20.214057    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47645c7d_9e9e_4682_a7ab_36dc6a0f0c9b.slice/crio-81d510ea984fbb0c1f6fb7b0cb5b5923f8f7296c27800f8a0537a575c848a0e9 WatchSource:0}: Error finding container 81d510ea984fbb0c1f6fb7b0cb5b5923f8f7296c27800f8a0537a575c848a0e9: Status 404 returned error can't find the container with id 81d510ea984fbb0c1f6fb7b0cb5b5923f8f7296c27800f8a0537a575c848a0e9
Jul 07 11:48:20.229797 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:20.229778    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-bqh87"]
Jul 07 11:48:20.232076 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:48:20.232051    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf02ca4e5_50d1_4b05_9a9b_41e28e5db5eb.slice/crio-2473071f9d76a65825ed09edb2a34a05354909416a78e6f0559c78f057ef024c WatchSource:0}: Error finding container 2473071f9d76a65825ed09edb2a34a05354909416a78e6f0559c78f057ef024c: Status 404 returned error can't find the container with id 2473071f9d76a65825ed09edb2a34a05354909416a78e6f0559c78f057ef024c
Jul 07 11:48:20.281615 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:20.281584    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-kbsc8" event={"ID":"47645c7d-9e9e-4682-a7ab-36dc6a0f0c9b","Type":"ContainerStarted","Data":"81d510ea984fbb0c1f6fb7b0cb5b5923f8f7296c27800f8a0537a575c848a0e9"}
Jul 07 11:48:20.282778 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:20.282755    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-bqh87" event={"ID":"f02ca4e5-50d1-4b05-9a9b-41e28e5db5eb","Type":"ContainerStarted","Data":"2473071f9d76a65825ed09edb2a34a05354909416a78e6f0559c78f057ef024c"}
Jul 07 11:48:23.299255 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:23.299153    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-kbsc8" event={"ID":"47645c7d-9e9e-4682-a7ab-36dc6a0f0c9b","Type":"ContainerStarted","Data":"13ddc627b3e56d9633a57a1c8c32baa0727d6cd296c5945b1e4326d946847467"}
Jul 07 11:48:23.299711 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:23.299274    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-kbsc8"
Jul 07 11:48:23.300681 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:23.300659    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-bqh87" event={"ID":"f02ca4e5-50d1-4b05-9a9b-41e28e5db5eb","Type":"ContainerStarted","Data":"a0f5860a218f9f12210c133abdcdd149ba0734f01c009385bef9e437859b6876"}
Jul 07 11:48:23.300789 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:23.300711    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-bqh87"
Jul 07 11:48:23.316715 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:23.316668    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-kbsc8" podStartSLOduration=1.666603313 podStartE2EDuration="4.316656657s" totalImagesPullingTime="2.650053344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:48:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:48:23.315902804 +0000 UTC m=+493.663116429" watchObservedRunningTime="2026-07-07 11:48:23.316656657 +0000 UTC m=+493.663870280"
Jul 07 11:48:23.333247 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:23.333196    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-bqh87" podStartSLOduration=1.647537332 podStartE2EDuration="4.333179781s" totalImagesPullingTime="2.685642449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:48:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:48:23.331440558 +0000 UTC m=+493.678654182" watchObservedRunningTime="2026-07-07 11:48:23.333179781 +0000 UTC m=+493.680393405"
Jul 07 11:48:34.306937 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:34.306910    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-bqh87"
Jul 07 11:48:34.308829 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:34.308807    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-kbsc8"
Jul 07 11:48:35.121093 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:35.121061    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-pnrfw"]
Jul 07 11:48:35.123187 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:35.123171    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pnrfw"
Jul 07 11:48:35.126044 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:35.126025    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 07 11:48:35.126127 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:35.126059    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-tmcjj\""
Jul 07 11:48:35.132895 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:35.132872    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-pnrfw"]
Jul 07 11:48:35.180818 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:35.180782    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccebb7d4-9e60-421d-9c09-b43a19b0f2c0-model-cache\") pod \"s3-init-pnrfw\" (UID: \"ccebb7d4-9e60-421d-9c09-b43a19b0f2c0\") " pod="kserve/s3-init-pnrfw"
Jul 07 11:48:35.180933 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:35.180832    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4zmt\" (UniqueName: \"kubernetes.io/projected/ccebb7d4-9e60-421d-9c09-b43a19b0f2c0-kube-api-access-c4zmt\") pod \"s3-init-pnrfw\" (UID: \"ccebb7d4-9e60-421d-9c09-b43a19b0f2c0\") " pod="kserve/s3-init-pnrfw"
Jul 07 11:48:35.281245 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:35.281217    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccebb7d4-9e60-421d-9c09-b43a19b0f2c0-model-cache\") pod \"s3-init-pnrfw\" (UID: \"ccebb7d4-9e60-421d-9c09-b43a19b0f2c0\") " pod="kserve/s3-init-pnrfw"
Jul 07 11:48:35.281245 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:35.281246    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4zmt\" (UniqueName: \"kubernetes.io/projected/ccebb7d4-9e60-421d-9c09-b43a19b0f2c0-kube-api-access-c4zmt\") pod \"s3-init-pnrfw\" (UID: \"ccebb7d4-9e60-421d-9c09-b43a19b0f2c0\") " pod="kserve/s3-init-pnrfw"
Jul 07 11:48:35.281633 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:35.281617    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccebb7d4-9e60-421d-9c09-b43a19b0f2c0-model-cache\") pod \"s3-init-pnrfw\" (UID: \"ccebb7d4-9e60-421d-9c09-b43a19b0f2c0\") " pod="kserve/s3-init-pnrfw"
Jul 07 11:48:35.289972 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:35.289947    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4zmt\" (UniqueName: \"kubernetes.io/projected/ccebb7d4-9e60-421d-9c09-b43a19b0f2c0-kube-api-access-c4zmt\") pod \"s3-init-pnrfw\" (UID: \"ccebb7d4-9e60-421d-9c09-b43a19b0f2c0\") " pod="kserve/s3-init-pnrfw"
Jul 07 11:48:35.445762 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:35.445693    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pnrfw"
Jul 07 11:48:35.568468 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:35.568200    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-pnrfw"]
Jul 07 11:48:35.571451 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:48:35.571402    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccebb7d4_9e60_421d_9c09_b43a19b0f2c0.slice/crio-c8af5bde919a7f1741f7547a2831b8842c37ed1f8ef04aed67ade8fb9805be98 WatchSource:0}: Error finding container c8af5bde919a7f1741f7547a2831b8842c37ed1f8ef04aed67ade8fb9805be98: Status 404 returned error can't find the container with id c8af5bde919a7f1741f7547a2831b8842c37ed1f8ef04aed67ade8fb9805be98
Jul 07 11:48:36.353562 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:36.353529    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pnrfw" event={"ID":"ccebb7d4-9e60-421d-9c09-b43a19b0f2c0","Type":"ContainerStarted","Data":"c8af5bde919a7f1741f7547a2831b8842c37ed1f8ef04aed67ade8fb9805be98"}
Jul 07 11:48:39.371091 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:39.371051    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pnrfw" event={"ID":"ccebb7d4-9e60-421d-9c09-b43a19b0f2c0","Type":"ContainerStarted","Data":"c2aa378aa7ca385981a9d7c769c0e5330158f93d264216aeccb161269113b747"}
Jul 07 11:48:41.380911 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:41.380876    2581 generic.go:356] "Generic (PLEG): container finished" podID="ccebb7d4-9e60-421d-9c09-b43a19b0f2c0" containerID="c2aa378aa7ca385981a9d7c769c0e5330158f93d264216aeccb161269113b747" exitCode=0
Jul 07 11:48:41.381292 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:41.380951    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pnrfw" event={"ID":"ccebb7d4-9e60-421d-9c09-b43a19b0f2c0","Type":"ContainerDied","Data":"c2aa378aa7ca385981a9d7c769c0e5330158f93d264216aeccb161269113b747"}
Jul 07 11:48:46.406403 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:46.406368    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pnrfw" event={"ID":"ccebb7d4-9e60-421d-9c09-b43a19b0f2c0","Type":"ContainerStarted","Data":"18c3a6d422ceeb2b76f4838d4a9b7a6b1e38e43d52613fdd6c2e89d19b25e604"}
Jul 07 11:48:46.425704 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:48:46.425631    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-pnrfw" podStartSLOduration=1.740122794 podStartE2EDuration="11.425614621s" totalImagesPullingTime="8.204406372s" totalInitContainerRuntime="1.481085455s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:48:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:48:46.425150117 +0000 UTC m=+516.772363741" watchObservedRunningTime="2026-07-07 11:48:46.425614621 +0000 UTC m=+516.772828243"
Jul 07 11:49:51.678896 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:49:51.678861    2581 generic.go:356] "Generic (PLEG): container finished" podID="ccebb7d4-9e60-421d-9c09-b43a19b0f2c0" containerID="18c3a6d422ceeb2b76f4838d4a9b7a6b1e38e43d52613fdd6c2e89d19b25e604" exitCode=0
Jul 07 11:49:51.679406 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:49:51.678938    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pnrfw" event={"ID":"ccebb7d4-9e60-421d-9c09-b43a19b0f2c0","Type":"ContainerDied","Data":"18c3a6d422ceeb2b76f4838d4a9b7a6b1e38e43d52613fdd6c2e89d19b25e604"}
Jul 07 11:49:52.811167 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:49:52.811145    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pnrfw"
Jul 07 11:49:52.827718 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:49:52.827695    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccebb7d4-9e60-421d-9c09-b43a19b0f2c0-model-cache\") pod \"ccebb7d4-9e60-421d-9c09-b43a19b0f2c0\" (UID: \"ccebb7d4-9e60-421d-9c09-b43a19b0f2c0\") "
Jul 07 11:49:52.827810 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:49:52.827754    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4zmt\" (UniqueName: \"kubernetes.io/projected/ccebb7d4-9e60-421d-9c09-b43a19b0f2c0-kube-api-access-c4zmt\") pod \"ccebb7d4-9e60-421d-9c09-b43a19b0f2c0\" (UID: \"ccebb7d4-9e60-421d-9c09-b43a19b0f2c0\") "
Jul 07 11:49:52.829710 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:49:52.829687    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccebb7d4-9e60-421d-9c09-b43a19b0f2c0-kube-api-access-c4zmt" (OuterVolumeSpecName: "kube-api-access-c4zmt") pod "ccebb7d4-9e60-421d-9c09-b43a19b0f2c0" (UID: "ccebb7d4-9e60-421d-9c09-b43a19b0f2c0"). InnerVolumeSpecName "kube-api-access-c4zmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 11:49:52.847995 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:49:52.847949    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccebb7d4-9e60-421d-9c09-b43a19b0f2c0-model-cache" (OuterVolumeSpecName: "model-cache") pod "ccebb7d4-9e60-421d-9c09-b43a19b0f2c0" (UID: "ccebb7d4-9e60-421d-9c09-b43a19b0f2c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 11:49:52.928668 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:49:52.928637    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccebb7d4-9e60-421d-9c09-b43a19b0f2c0-model-cache\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:49:52.928668 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:49:52.928667    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4zmt\" (UniqueName: \"kubernetes.io/projected/ccebb7d4-9e60-421d-9c09-b43a19b0f2c0-kube-api-access-c4zmt\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:49:53.688127 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:49:53.688101    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pnrfw"
Jul 07 11:49:53.688287 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:49:53.688136    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pnrfw" event={"ID":"ccebb7d4-9e60-421d-9c09-b43a19b0f2c0","Type":"ContainerDied","Data":"c8af5bde919a7f1741f7547a2831b8842c37ed1f8ef04aed67ade8fb9805be98"}
Jul 07 11:49:53.688287 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:49:53.688168    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8af5bde919a7f1741f7547a2831b8842c37ed1f8ef04aed67ade8fb9805be98"
Jul 07 11:50:01.529165 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:01.529131    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-rmcgf"]
Jul 07 11:50:01.529598 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:01.529466    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccebb7d4-9e60-421d-9c09-b43a19b0f2c0" containerName="s3-init"
Jul 07 11:50:01.529598 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:01.529477    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccebb7d4-9e60-421d-9c09-b43a19b0f2c0" containerName="s3-init"
Jul 07 11:50:01.529598 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:01.529487    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccebb7d4-9e60-421d-9c09-b43a19b0f2c0" containerName="download-hf-model"
Jul 07 11:50:01.529598 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:01.529493    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccebb7d4-9e60-421d-9c09-b43a19b0f2c0" containerName="download-hf-model"
Jul 07 11:50:01.529598 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:01.529540    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccebb7d4-9e60-421d-9c09-b43a19b0f2c0" containerName="s3-init"
Jul 07 11:50:01.531628 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:01.531612    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-rmcgf"
Jul 07 11:50:01.534993 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:01.534936    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 07 11:50:01.535321 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:01.535305    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-tmcjj\""
Jul 07 11:50:01.544695 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:01.544668    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-rmcgf"]
Jul 07 11:50:01.595353 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:01.595321    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3b659ee-e419-41cc-9958-c651dd1273bd-model-cache\") pod \"s3-tls-init-custom-rmcgf\" (UID: \"f3b659ee-e419-41cc-9958-c651dd1273bd\") " pod="kserve/s3-tls-init-custom-rmcgf"
Jul 07 11:50:01.595476 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:01.595383    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69n75\" (UniqueName: \"kubernetes.io/projected/f3b659ee-e419-41cc-9958-c651dd1273bd-kube-api-access-69n75\") pod \"s3-tls-init-custom-rmcgf\" (UID: \"f3b659ee-e419-41cc-9958-c651dd1273bd\") " pod="kserve/s3-tls-init-custom-rmcgf"
Jul 07 11:50:01.695971 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:01.695938    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69n75\" (UniqueName: \"kubernetes.io/projected/f3b659ee-e419-41cc-9958-c651dd1273bd-kube-api-access-69n75\") pod \"s3-tls-init-custom-rmcgf\" (UID: \"f3b659ee-e419-41cc-9958-c651dd1273bd\") " pod="kserve/s3-tls-init-custom-rmcgf"
Jul 07 11:50:01.696142 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:01.695987    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3b659ee-e419-41cc-9958-c651dd1273bd-model-cache\") pod \"s3-tls-init-custom-rmcgf\" (UID: \"f3b659ee-e419-41cc-9958-c651dd1273bd\") " pod="kserve/s3-tls-init-custom-rmcgf"
Jul 07 11:50:01.696319 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:01.696300    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3b659ee-e419-41cc-9958-c651dd1273bd-model-cache\") pod \"s3-tls-init-custom-rmcgf\" (UID: \"f3b659ee-e419-41cc-9958-c651dd1273bd\") " pod="kserve/s3-tls-init-custom-rmcgf"
Jul 07 11:50:01.705756 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:01.705736    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69n75\" (UniqueName: \"kubernetes.io/projected/f3b659ee-e419-41cc-9958-c651dd1273bd-kube-api-access-69n75\") pod \"s3-tls-init-custom-rmcgf\" (UID: \"f3b659ee-e419-41cc-9958-c651dd1273bd\") " pod="kserve/s3-tls-init-custom-rmcgf"
Jul 07 11:50:01.850379 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:01.850349    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-rmcgf"
Jul 07 11:50:01.989934 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:01.989902    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-rmcgf"]
Jul 07 11:50:01.992739 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:50:01.992710    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3b659ee_e419_41cc_9958_c651dd1273bd.slice/crio-c20b8c106d32b6a9514d892ac7415a363742fe569a89a149eee22d02af65dfff WatchSource:0}: Error finding container c20b8c106d32b6a9514d892ac7415a363742fe569a89a149eee22d02af65dfff: Status 404 returned error can't find the container with id c20b8c106d32b6a9514d892ac7415a363742fe569a89a149eee22d02af65dfff
Jul 07 11:50:02.727216 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:02.727181    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-rmcgf" event={"ID":"f3b659ee-e419-41cc-9958-c651dd1273bd","Type":"ContainerStarted","Data":"fe6d0595e977f032f97da7bc0a545e4031b4bbad5b303c55a867c771ca5f03b0"}
Jul 07 11:50:02.727584 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:02.727224    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-rmcgf" event={"ID":"f3b659ee-e419-41cc-9958-c651dd1273bd","Type":"ContainerStarted","Data":"c20b8c106d32b6a9514d892ac7415a363742fe569a89a149eee22d02af65dfff"}
Jul 07 11:50:03.732174 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:03.732137    2581 generic.go:356] "Generic (PLEG): container finished" podID="f3b659ee-e419-41cc-9958-c651dd1273bd" containerID="fe6d0595e977f032f97da7bc0a545e4031b4bbad5b303c55a867c771ca5f03b0" exitCode=0
Jul 07 11:50:03.732623 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:03.732225    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-rmcgf" event={"ID":"f3b659ee-e419-41cc-9958-c651dd1273bd","Type":"ContainerDied","Data":"fe6d0595e977f032f97da7bc0a545e4031b4bbad5b303c55a867c771ca5f03b0"}
Jul 07 11:50:04.737635 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:04.737603    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-rmcgf" event={"ID":"f3b659ee-e419-41cc-9958-c651dd1273bd","Type":"ContainerStarted","Data":"87dba2d7d467462df5ea6649561a79e75042a97066704b35a1236c52dc0e9d73"}
Jul 07 11:50:04.757777 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:04.757720    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-rmcgf" podStartSLOduration=2.2825725390000002 podStartE2EDuration="3.757704358s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.475131819s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:50:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:50:04.756158189 +0000 UTC m=+595.103371811" watchObservedRunningTime="2026-07-07 11:50:04.757704358 +0000 UTC m=+595.104917981"
Jul 07 11:50:10.765637 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:10.765601    2581 generic.go:356] "Generic (PLEG): container finished" podID="f3b659ee-e419-41cc-9958-c651dd1273bd" containerID="87dba2d7d467462df5ea6649561a79e75042a97066704b35a1236c52dc0e9d73" exitCode=0
Jul 07 11:50:10.766057 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:10.765647    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-rmcgf" event={"ID":"f3b659ee-e419-41cc-9958-c651dd1273bd","Type":"ContainerDied","Data":"87dba2d7d467462df5ea6649561a79e75042a97066704b35a1236c52dc0e9d73"}
Jul 07 11:50:11.910260 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:11.910237    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-rmcgf"
Jul 07 11:50:11.984718 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:11.984691    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3b659ee-e419-41cc-9958-c651dd1273bd-model-cache\") pod \"f3b659ee-e419-41cc-9958-c651dd1273bd\" (UID: \"f3b659ee-e419-41cc-9958-c651dd1273bd\") "
Jul 07 11:50:11.984954 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:11.984746    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69n75\" (UniqueName: \"kubernetes.io/projected/f3b659ee-e419-41cc-9958-c651dd1273bd-kube-api-access-69n75\") pod \"f3b659ee-e419-41cc-9958-c651dd1273bd\" (UID: \"f3b659ee-e419-41cc-9958-c651dd1273bd\") "
Jul 07 11:50:11.986854 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:11.986824    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b659ee-e419-41cc-9958-c651dd1273bd-kube-api-access-69n75" (OuterVolumeSpecName: "kube-api-access-69n75") pod "f3b659ee-e419-41cc-9958-c651dd1273bd" (UID: "f3b659ee-e419-41cc-9958-c651dd1273bd"). InnerVolumeSpecName "kube-api-access-69n75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 11:50:12.008251 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:12.008224    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b659ee-e419-41cc-9958-c651dd1273bd-model-cache" (OuterVolumeSpecName: "model-cache") pod "f3b659ee-e419-41cc-9958-c651dd1273bd" (UID: "f3b659ee-e419-41cc-9958-c651dd1273bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 11:50:12.042865 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:12.042810    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kf48j_4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801/ovn-acl-logging/0.log"
Jul 07 11:50:12.043971 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:12.043951    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kf48j_4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801/ovn-acl-logging/0.log"
Jul 07 11:50:12.085924 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:12.085902    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3b659ee-e419-41cc-9958-c651dd1273bd-model-cache\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:50:12.085924 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:12.085921    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69n75\" (UniqueName: \"kubernetes.io/projected/f3b659ee-e419-41cc-9958-c651dd1273bd-kube-api-access-69n75\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:50:12.775409 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:12.775383    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-rmcgf"
Jul 07 11:50:12.775569 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:12.775380    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-rmcgf" event={"ID":"f3b659ee-e419-41cc-9958-c651dd1273bd","Type":"ContainerDied","Data":"c20b8c106d32b6a9514d892ac7415a363742fe569a89a149eee22d02af65dfff"}
Jul 07 11:50:12.775569 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:12.775490    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c20b8c106d32b6a9514d892ac7415a363742fe569a89a149eee22d02af65dfff"
Jul 07 11:50:18.552604 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:18.552565    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-4p794"]
Jul 07 11:50:18.552965 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:18.552902    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3b659ee-e419-41cc-9958-c651dd1273bd" containerName="download-hf-model"
Jul 07 11:50:18.552965 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:18.552916    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3b659ee-e419-41cc-9958-c651dd1273bd" containerName="download-hf-model"
Jul 07 11:50:18.552965 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:18.552926    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3b659ee-e419-41cc-9958-c651dd1273bd" containerName="s3-tls-init-custom"
Jul 07 11:50:18.552965 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:18.552931    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3b659ee-e419-41cc-9958-c651dd1273bd" containerName="s3-tls-init-custom"
Jul 07 11:50:18.553129 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:18.552989    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3b659ee-e419-41cc-9958-c651dd1273bd" containerName="s3-tls-init-custom"
Jul 07 11:50:18.555004 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:18.554988    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-4p794"
Jul 07 11:50:18.557732 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:18.557706    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 07 11:50:18.557883 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:18.557782    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-tmcjj\""
Jul 07 11:50:18.564434 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:18.564409    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-4p794"]
Jul 07 11:50:18.643447 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:18.643413    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50714eb9-46b4-4893-9c55-c2167edc6505-model-cache\") pod \"s3-tls-init-serving-4p794\" (UID: \"50714eb9-46b4-4893-9c55-c2167edc6505\") " pod="kserve/s3-tls-init-serving-4p794"
Jul 07 11:50:18.643586 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:18.643458    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz4v2\" (UniqueName: \"kubernetes.io/projected/50714eb9-46b4-4893-9c55-c2167edc6505-kube-api-access-zz4v2\") pod \"s3-tls-init-serving-4p794\" (UID: \"50714eb9-46b4-4893-9c55-c2167edc6505\") " pod="kserve/s3-tls-init-serving-4p794"
Jul 07 11:50:18.744421 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:18.744398    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz4v2\" (UniqueName: \"kubernetes.io/projected/50714eb9-46b4-4893-9c55-c2167edc6505-kube-api-access-zz4v2\") pod \"s3-tls-init-serving-4p794\" (UID: \"50714eb9-46b4-4893-9c55-c2167edc6505\") " pod="kserve/s3-tls-init-serving-4p794"
Jul 07 11:50:18.744552 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:18.744462    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50714eb9-46b4-4893-9c55-c2167edc6505-model-cache\") pod \"s3-tls-init-serving-4p794\" (UID: \"50714eb9-46b4-4893-9c55-c2167edc6505\") " pod="kserve/s3-tls-init-serving-4p794"
Jul 07 11:50:18.744763 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:18.744748    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50714eb9-46b4-4893-9c55-c2167edc6505-model-cache\") pod \"s3-tls-init-serving-4p794\" (UID: \"50714eb9-46b4-4893-9c55-c2167edc6505\") " pod="kserve/s3-tls-init-serving-4p794"
Jul 07 11:50:18.755567 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:18.755540    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz4v2\" (UniqueName: \"kubernetes.io/projected/50714eb9-46b4-4893-9c55-c2167edc6505-kube-api-access-zz4v2\") pod \"s3-tls-init-serving-4p794\" (UID: \"50714eb9-46b4-4893-9c55-c2167edc6505\") " pod="kserve/s3-tls-init-serving-4p794"
Jul 07 11:50:18.869964 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:18.869934    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-4p794"
Jul 07 11:50:18.995364 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:18.995341    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-4p794"]
Jul 07 11:50:18.997734 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:50:18.997708    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50714eb9_46b4_4893_9c55_c2167edc6505.slice/crio-9221c61b92c6ac9305eff0731fc0784e3fb787f2ed1d4ace827507930f442134 WatchSource:0}: Error finding container 9221c61b92c6ac9305eff0731fc0784e3fb787f2ed1d4ace827507930f442134: Status 404 returned error can't find the container with id 9221c61b92c6ac9305eff0731fc0784e3fb787f2ed1d4ace827507930f442134
Jul 07 11:50:19.805795 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:19.805752    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4p794" event={"ID":"50714eb9-46b4-4893-9c55-c2167edc6505","Type":"ContainerStarted","Data":"b734df0e5ebfcf07f01088ff200764ebfc3ca2316040a3b330f45188929cc43b"}
Jul 07 11:50:19.805795 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:19.805802    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4p794" event={"ID":"50714eb9-46b4-4893-9c55-c2167edc6505","Type":"ContainerStarted","Data":"9221c61b92c6ac9305eff0731fc0784e3fb787f2ed1d4ace827507930f442134"}
Jul 07 11:50:20.811101 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:20.811004    2581 generic.go:356] "Generic (PLEG): container finished" podID="50714eb9-46b4-4893-9c55-c2167edc6505" containerID="b734df0e5ebfcf07f01088ff200764ebfc3ca2316040a3b330f45188929cc43b" exitCode=0
Jul 07 11:50:20.811101 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:20.811074    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4p794" event={"ID":"50714eb9-46b4-4893-9c55-c2167edc6505","Type":"ContainerDied","Data":"b734df0e5ebfcf07f01088ff200764ebfc3ca2316040a3b330f45188929cc43b"}
Jul 07 11:50:21.816828 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:21.816788    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4p794" event={"ID":"50714eb9-46b4-4893-9c55-c2167edc6505","Type":"ContainerStarted","Data":"0e7e7915d44a73146898f5ea0354194a6d4ba23dc78b5e5c1724d4b18b033bda"}
Jul 07 11:50:21.836740 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:21.836690    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-4p794" podStartSLOduration=2.3466671740000002 podStartE2EDuration="3.836677462s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.490010288s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:50:21.835764133 +0000 UTC m=+612.182977756" watchObservedRunningTime="2026-07-07 11:50:21.836677462 +0000 UTC m=+612.183891085"
Jul 07 11:50:26.839312 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:26.839278    2581 generic.go:356] "Generic (PLEG): container finished" podID="50714eb9-46b4-4893-9c55-c2167edc6505" containerID="0e7e7915d44a73146898f5ea0354194a6d4ba23dc78b5e5c1724d4b18b033bda" exitCode=0
Jul 07 11:50:26.839671 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:26.839356    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4p794" event={"ID":"50714eb9-46b4-4893-9c55-c2167edc6505","Type":"ContainerDied","Data":"0e7e7915d44a73146898f5ea0354194a6d4ba23dc78b5e5c1724d4b18b033bda"}
Jul 07 11:50:28.002620 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:28.002599    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-4p794"
Jul 07 11:50:28.130971 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:28.130947    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50714eb9-46b4-4893-9c55-c2167edc6505-model-cache\") pod \"50714eb9-46b4-4893-9c55-c2167edc6505\" (UID: \"50714eb9-46b4-4893-9c55-c2167edc6505\") "
Jul 07 11:50:28.131155 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:28.130983    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zz4v2\" (UniqueName: \"kubernetes.io/projected/50714eb9-46b4-4893-9c55-c2167edc6505-kube-api-access-zz4v2\") pod \"50714eb9-46b4-4893-9c55-c2167edc6505\" (UID: \"50714eb9-46b4-4893-9c55-c2167edc6505\") "
Jul 07 11:50:28.132994 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:28.132968    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50714eb9-46b4-4893-9c55-c2167edc6505-kube-api-access-zz4v2" (OuterVolumeSpecName: "kube-api-access-zz4v2") pod "50714eb9-46b4-4893-9c55-c2167edc6505" (UID: "50714eb9-46b4-4893-9c55-c2167edc6505"). InnerVolumeSpecName "kube-api-access-zz4v2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 11:50:28.153422 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:28.153393    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50714eb9-46b4-4893-9c55-c2167edc6505-model-cache" (OuterVolumeSpecName: "model-cache") pod "50714eb9-46b4-4893-9c55-c2167edc6505" (UID: "50714eb9-46b4-4893-9c55-c2167edc6505"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 11:50:28.231856 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:28.231833    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50714eb9-46b4-4893-9c55-c2167edc6505-model-cache\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:50:28.231856 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:28.231855    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz4v2\" (UniqueName: \"kubernetes.io/projected/50714eb9-46b4-4893-9c55-c2167edc6505-kube-api-access-zz4v2\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:50:28.848396 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:28.848362    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-4p794"
Jul 07 11:50:28.848396 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:28.848375    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4p794" event={"ID":"50714eb9-46b4-4893-9c55-c2167edc6505","Type":"ContainerDied","Data":"9221c61b92c6ac9305eff0731fc0784e3fb787f2ed1d4ace827507930f442134"}
Jul 07 11:50:28.848596 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:28.848411    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9221c61b92c6ac9305eff0731fc0784e3fb787f2ed1d4ace827507930f442134"
Jul 07 11:50:41.462376 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:41.462345    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8"]
Jul 07 11:50:41.463714 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:41.462700    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50714eb9-46b4-4893-9c55-c2167edc6505" containerName="s3-tls-init-serving"
Jul 07 11:50:41.463714 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:41.462712    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="50714eb9-46b4-4893-9c55-c2167edc6505" containerName="s3-tls-init-serving"
Jul 07 11:50:41.463714 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:41.462729    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50714eb9-46b4-4893-9c55-c2167edc6505" containerName="download-hf-model"
Jul 07 11:50:41.463714 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:41.462735    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="50714eb9-46b4-4893-9c55-c2167edc6505" containerName="download-hf-model"
Jul 07 11:50:41.463714 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:41.462785    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="50714eb9-46b4-4893-9c55-c2167edc6505" containerName="s3-tls-init-serving"
Jul 07 11:50:41.464671 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:41.464655    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8"
Jul 07 11:50:41.467918 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:41.467890    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 07 11:50:41.467918 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:41.467895    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 11:50:41.468108 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:41.467943    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 11:50:41.468108 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:41.467963    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-p2bwt\""
Jul 07 11:50:41.468228 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:41.468208    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 07 11:50:41.468347 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:41.468247    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 11:50:41.468347 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:41.468319    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 11:50:41.481245 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:41.481223    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8"]
Jul 07 11:50:41.544695 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:41.544667    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb97329a-c3b8-47e2-b4d8-76e520b61610-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-656ff6f494-qzwq8\" (UID: \"fb97329a-c3b8-47e2-b4d8-76e520b61610\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8"
Jul 07 11:50:41.544831 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:41.544704    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb97329a-c3b8-47e2-b4d8-76e520b61610-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-656ff6f494-qzwq8\" (UID: \"fb97329a-c3b8-47e2-b4d8-76e520b61610\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8"
Jul 07 11:50:41.544831 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:41.544731    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69cv5\" (UniqueName: \"kubernetes.io/projected/fb97329a-c3b8-47e2-b4d8-76e520b61610-kube-api-access-69cv5\") pod \"isvc-sklearn-batcher-predictor-656ff6f494-qzwq8\" (UID: \"fb97329a-c3b8-47e2-b4d8-76e520b61610\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8"
Jul 07 11:50:41.544831 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:41.544824    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb97329a-c3b8-47e2-b4d8-76e520b61610-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-656ff6f494-qzwq8\" (UID: \"fb97329a-c3b8-47e2-b4d8-76e520b61610\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8"
Jul 07 11:50:41.544962 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:41.544847    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb97329a-c3b8-47e2-b4d8-76e520b61610-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-656ff6f494-qzwq8\" (UID: \"fb97329a-c3b8-47e2-b4d8-76e520b61610\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8"
Jul 07 11:50:41.645257 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:41.645229    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb97329a-c3b8-47e2-b4d8-76e520b61610-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-656ff6f494-qzwq8\" (UID: \"fb97329a-c3b8-47e2-b4d8-76e520b61610\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8"
Jul 07 11:50:41.645416 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:41.645263    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb97329a-c3b8-47e2-b4d8-76e520b61610-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-656ff6f494-qzwq8\" (UID: \"fb97329a-c3b8-47e2-b4d8-76e520b61610\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8"
Jul 07 11:50:41.645416 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:41.645292    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69cv5\" (UniqueName: \"kubernetes.io/projected/fb97329a-c3b8-47e2-b4d8-76e520b61610-kube-api-access-69cv5\") pod \"isvc-sklearn-batcher-predictor-656ff6f494-qzwq8\" (UID: \"fb97329a-c3b8-47e2-b4d8-76e520b61610\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8"
Jul 07 11:50:41.645416 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:41.645333    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb97329a-c3b8-47e2-b4d8-76e520b61610-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-656ff6f494-qzwq8\" (UID: \"fb97329a-c3b8-47e2-b4d8-76e520b61610\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8"
Jul 07 11:50:41.645416 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:41.645388    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb97329a-c3b8-47e2-b4d8-76e520b61610-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-656ff6f494-qzwq8\" (UID: \"fb97329a-c3b8-47e2-b4d8-76e520b61610\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8"
Jul 07 11:50:41.645717 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:41.645695    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb97329a-c3b8-47e2-b4d8-76e520b61610-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-656ff6f494-qzwq8\" (UID: \"fb97329a-c3b8-47e2-b4d8-76e520b61610\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8"
Jul 07 11:50:41.646075 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:41.646054    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb97329a-c3b8-47e2-b4d8-76e520b61610-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-656ff6f494-qzwq8\" (UID: \"fb97329a-c3b8-47e2-b4d8-76e520b61610\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8"
Jul 07 11:50:41.646164 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:41.646100    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb97329a-c3b8-47e2-b4d8-76e520b61610-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-656ff6f494-qzwq8\" (UID: \"fb97329a-c3b8-47e2-b4d8-76e520b61610\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8"
Jul 07 11:50:41.647997 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:41.647975    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb97329a-c3b8-47e2-b4d8-76e520b61610-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-656ff6f494-qzwq8\" (UID: \"fb97329a-c3b8-47e2-b4d8-76e520b61610\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8"
Jul 07 11:50:41.654798 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:41.654778    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69cv5\" (UniqueName: \"kubernetes.io/projected/fb97329a-c3b8-47e2-b4d8-76e520b61610-kube-api-access-69cv5\") pod \"isvc-sklearn-batcher-predictor-656ff6f494-qzwq8\" (UID: \"fb97329a-c3b8-47e2-b4d8-76e520b61610\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8"
Jul 07 11:50:41.776176 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:41.776107    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8"
Jul 07 11:50:41.906419 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:41.906387    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8"]
Jul 07 11:50:41.908434 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:50:41.908410    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb97329a_c3b8_47e2_b4d8_76e520b61610.slice/crio-ed4c147100d1d838568da6a4bfc8be2d0e71166699f45d54a60e3d5067fe0c31 WatchSource:0}: Error finding container ed4c147100d1d838568da6a4bfc8be2d0e71166699f45d54a60e3d5067fe0c31: Status 404 returned error can't find the container with id ed4c147100d1d838568da6a4bfc8be2d0e71166699f45d54a60e3d5067fe0c31
Jul 07 11:50:42.915254 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:42.915211    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" event={"ID":"fb97329a-c3b8-47e2-b4d8-76e520b61610","Type":"ContainerStarted","Data":"48c9c2d1c73c394d8649899a2a6135c1a81d3ac71895fd42b9e2ad9e5f7679c5"}
Jul 07 11:50:42.915613 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:42.915285    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" event={"ID":"fb97329a-c3b8-47e2-b4d8-76e520b61610","Type":"ContainerStarted","Data":"ed4c147100d1d838568da6a4bfc8be2d0e71166699f45d54a60e3d5067fe0c31"}
Jul 07 11:50:45.928940 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:45.928901    2581 generic.go:356] "Generic (PLEG): container finished" podID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerID="48c9c2d1c73c394d8649899a2a6135c1a81d3ac71895fd42b9e2ad9e5f7679c5" exitCode=0
Jul 07 11:50:45.929415 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:45.928944    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" event={"ID":"fb97329a-c3b8-47e2-b4d8-76e520b61610","Type":"ContainerDied","Data":"48c9c2d1c73c394d8649899a2a6135c1a81d3ac71895fd42b9e2ad9e5f7679c5"}
Jul 07 11:50:45.930273 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:50:45.930253    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 11:51:00.010499 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:51:00.010456    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" event={"ID":"fb97329a-c3b8-47e2-b4d8-76e520b61610","Type":"ContainerStarted","Data":"878f0da76ff6e0ac224cef390889abe81a4cadebc72ed2b8842a79c1d22709bd"}
Jul 07 11:51:02.021534 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:51:02.021453    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" event={"ID":"fb97329a-c3b8-47e2-b4d8-76e520b61610","Type":"ContainerStarted","Data":"b624bc5eedd2fad09e3557190bafb85440ed5fe73a4f189029fdc4b7f636eac1"}
Jul 07 11:51:05.040136 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:51:05.040099    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" event={"ID":"fb97329a-c3b8-47e2-b4d8-76e520b61610","Type":"ContainerStarted","Data":"6e5d97c4905ac4f55a21ec9df851eafbd20791f18e008ccc6a73d68974502d48"}
Jul 07 11:51:05.040631 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:51:05.040282    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8"
Jul 07 11:51:05.065431 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:51:05.065387    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" podStartSLOduration=2.067580477 podStartE2EDuration="24.065374026s" totalImagesPullingTime="18.118214793s" totalInitContainerRuntime="3.879578756s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:50:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:51:05.063401758 +0000 UTC m=+655.410615381" watchObservedRunningTime="2026-07-07 11:51:05.065374026 +0000 UTC m=+655.412587706"
Jul 07 11:51:06.044608 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:51:06.044557    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8"
Jul 07 11:51:06.045121 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:51:06.044635    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8"
Jul 07 11:51:06.046281 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:51:06.046245    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 07 11:51:06.046971 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:51:06.046947    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:51:06.049781 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:51:06.049761    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8"
Jul 07 11:51:07.048581 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:51:07.048545    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 07 11:51:07.048988 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:51:07.048769    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:51:08.053176 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:51:08.053140    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 07 11:51:08.053581 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:51:08.053370    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:51:18.053923 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:51:18.053877    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 07 11:51:18.054374 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:51:18.054347    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:51:28.053470 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:51:28.053422    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 07 11:51:28.053936 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:51:28.053911    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:51:38.053736 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:51:38.053688    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 07 11:51:38.054251 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:51:38.054154    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:51:48.053888 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:51:48.053810    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 07 11:51:48.054295 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:51:48.054246    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:51:58.053449 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:51:58.053403    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 07 11:51:58.053875 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:51:58.053849    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:52:08.053373 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:08.053273    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 07 11:52:08.053742 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:08.053685    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:52:18.053902 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:18.053873    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8"
Jul 07 11:52:18.054516 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:18.054108    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8"
Jul 07 11:52:26.602520 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:26.602485    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8"]
Jul 07 11:52:26.602906 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:26.602764    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="kserve-container" containerID="cri-o://878f0da76ff6e0ac224cef390889abe81a4cadebc72ed2b8842a79c1d22709bd" gracePeriod=30
Jul 07 11:52:26.602906 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:26.602829    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="kube-rbac-proxy" containerID="cri-o://b624bc5eedd2fad09e3557190bafb85440ed5fe73a4f189029fdc4b7f636eac1" gracePeriod=30
Jul 07 11:52:26.602906 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:26.602824    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="agent" containerID="cri-o://6e5d97c4905ac4f55a21ec9df851eafbd20791f18e008ccc6a73d68974502d48" gracePeriod=30
Jul 07 11:52:26.647366 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:26.647338    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4"]
Jul 07 11:52:26.651063 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:26.651045    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4"
Jul 07 11:52:26.653750 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:26.653731    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 07 11:52:26.654783 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:26.654761    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 07 11:52:26.666824 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:26.666797    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4"]
Jul 07 11:52:26.751218 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:26.751191    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0484788-3062-45f5-9f46-44a0688db68f-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4\" (UID: \"e0484788-3062-45f5-9f46-44a0688db68f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4"
Jul 07 11:52:26.751378 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:26.751239    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0484788-3062-45f5-9f46-44a0688db68f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4\" (UID: \"e0484788-3062-45f5-9f46-44a0688db68f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4"
Jul 07 11:52:26.751378 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:26.751277    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0484788-3062-45f5-9f46-44a0688db68f-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4\" (UID: \"e0484788-3062-45f5-9f46-44a0688db68f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4"
Jul 07 11:52:26.751378 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:26.751304    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0484788-3062-45f5-9f46-44a0688db68f-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4\" (UID: \"e0484788-3062-45f5-9f46-44a0688db68f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4"
Jul 07 11:52:26.751562 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:26.751381    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl62p\" (UniqueName: \"kubernetes.io/projected/e0484788-3062-45f5-9f46-44a0688db68f-kube-api-access-wl62p\") pod \"isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4\" (UID: \"e0484788-3062-45f5-9f46-44a0688db68f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4"
Jul 07 11:52:26.852631 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:26.852554    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0484788-3062-45f5-9f46-44a0688db68f-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4\" (UID: \"e0484788-3062-45f5-9f46-44a0688db68f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4"
Jul 07 11:52:26.852631 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:26.852601    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0484788-3062-45f5-9f46-44a0688db68f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4\" (UID: \"e0484788-3062-45f5-9f46-44a0688db68f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4"
Jul 07 11:52:26.852838 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:26.852654    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0484788-3062-45f5-9f46-44a0688db68f-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4\" (UID: \"e0484788-3062-45f5-9f46-44a0688db68f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4"
Jul 07 11:52:26.852838 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:26.852690    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0484788-3062-45f5-9f46-44a0688db68f-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4\" (UID: \"e0484788-3062-45f5-9f46-44a0688db68f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4"
Jul 07 11:52:26.852838 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:26.852747    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl62p\" (UniqueName: \"kubernetes.io/projected/e0484788-3062-45f5-9f46-44a0688db68f-kube-api-access-wl62p\") pod \"isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4\" (UID: \"e0484788-3062-45f5-9f46-44a0688db68f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4"
Jul 07 11:52:26.853001 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:52:26.852863    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-serving-cert: secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Jul 07 11:52:26.853001 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:52:26.852959    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0484788-3062-45f5-9f46-44a0688db68f-proxy-tls podName:e0484788-3062-45f5-9f46-44a0688db68f nodeName:}" failed. No retries permitted until 2026-07-07 11:52:27.352940326 +0000 UTC m=+737.700153934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e0484788-3062-45f5-9f46-44a0688db68f-proxy-tls") pod "isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" (UID: "e0484788-3062-45f5-9f46-44a0688db68f") : secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Jul 07 11:52:26.853154 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:26.853121    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0484788-3062-45f5-9f46-44a0688db68f-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4\" (UID: \"e0484788-3062-45f5-9f46-44a0688db68f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4"
Jul 07 11:52:26.853347 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:26.853325    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0484788-3062-45f5-9f46-44a0688db68f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4\" (UID: \"e0484788-3062-45f5-9f46-44a0688db68f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4"
Jul 07 11:52:26.853404 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:26.853361    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0484788-3062-45f5-9f46-44a0688db68f-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4\" (UID: \"e0484788-3062-45f5-9f46-44a0688db68f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4"
Jul 07 11:52:26.862141 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:26.862124    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl62p\" (UniqueName: \"kubernetes.io/projected/e0484788-3062-45f5-9f46-44a0688db68f-kube-api-access-wl62p\") pod \"isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4\" (UID: \"e0484788-3062-45f5-9f46-44a0688db68f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4"
Jul 07 11:52:27.355737 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:27.355711    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0484788-3062-45f5-9f46-44a0688db68f-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4\" (UID: \"e0484788-3062-45f5-9f46-44a0688db68f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4"
Jul 07 11:52:27.358249 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:27.358224    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0484788-3062-45f5-9f46-44a0688db68f-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4\" (UID: \"e0484788-3062-45f5-9f46-44a0688db68f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4"
Jul 07 11:52:27.388234 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:27.388207    2581 generic.go:356] "Generic (PLEG): container finished" podID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerID="b624bc5eedd2fad09e3557190bafb85440ed5fe73a4f189029fdc4b7f636eac1" exitCode=2
Jul 07 11:52:27.388347 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:27.388270    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" event={"ID":"fb97329a-c3b8-47e2-b4d8-76e520b61610","Type":"ContainerDied","Data":"b624bc5eedd2fad09e3557190bafb85440ed5fe73a4f189029fdc4b7f636eac1"}
Jul 07 11:52:27.561502 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:27.561469    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4"
Jul 07 11:52:27.693532 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:27.693453    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4"]
Jul 07 11:52:27.695790 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:52:27.695760    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0484788_3062_45f5_9f46_44a0688db68f.slice/crio-9e6b17d3080886ed1b258f4b5fbe480213df685627c60be6ebcb28da7108c4f2 WatchSource:0}: Error finding container 9e6b17d3080886ed1b258f4b5fbe480213df685627c60be6ebcb28da7108c4f2: Status 404 returned error can't find the container with id 9e6b17d3080886ed1b258f4b5fbe480213df685627c60be6ebcb28da7108c4f2
Jul 07 11:52:28.054030 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:28.053930    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 07 11:52:28.054305 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:28.054281    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:52:28.394538 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:28.394498    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" event={"ID":"e0484788-3062-45f5-9f46-44a0688db68f","Type":"ContainerStarted","Data":"1117e4ad6f34899abfd61b38aed9babe045090e254a19d2de125a2c8dbdda531"}
Jul 07 11:52:28.394701 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:28.394574    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" event={"ID":"e0484788-3062-45f5-9f46-44a0688db68f","Type":"ContainerStarted","Data":"9e6b17d3080886ed1b258f4b5fbe480213df685627c60be6ebcb28da7108c4f2"}
Jul 07 11:52:31.045561 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:31.045525    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 07 11:52:31.410356 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:31.410326    2581 generic.go:356] "Generic (PLEG): container finished" podID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerID="878f0da76ff6e0ac224cef390889abe81a4cadebc72ed2b8842a79c1d22709bd" exitCode=0
Jul 07 11:52:31.410514 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:31.410398    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" event={"ID":"fb97329a-c3b8-47e2-b4d8-76e520b61610","Type":"ContainerDied","Data":"878f0da76ff6e0ac224cef390889abe81a4cadebc72ed2b8842a79c1d22709bd"}
Jul 07 11:52:32.415564 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:32.415530    2581 generic.go:356] "Generic (PLEG): container finished" podID="e0484788-3062-45f5-9f46-44a0688db68f" containerID="1117e4ad6f34899abfd61b38aed9babe045090e254a19d2de125a2c8dbdda531" exitCode=0
Jul 07 11:52:32.415950 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:32.415588    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" event={"ID":"e0484788-3062-45f5-9f46-44a0688db68f","Type":"ContainerDied","Data":"1117e4ad6f34899abfd61b38aed9babe045090e254a19d2de125a2c8dbdda531"}
Jul 07 11:52:33.422271 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:33.422238    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" event={"ID":"e0484788-3062-45f5-9f46-44a0688db68f","Type":"ContainerStarted","Data":"cdee6433c0287b079338976358ae494c862c09addbdd2b5f8b43aa555fa867ee"}
Jul 07 11:52:33.422631 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:33.422328    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" event={"ID":"e0484788-3062-45f5-9f46-44a0688db68f","Type":"ContainerStarted","Data":"7ec9a121ca1a763f4808677957507c225b0576c396e7d56604c0f5a539d3b76a"}
Jul 07 11:52:33.422631 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:33.422354    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" event={"ID":"e0484788-3062-45f5-9f46-44a0688db68f","Type":"ContainerStarted","Data":"5a96e6726c5cb14695fced3c0e62a54ab172c4b17df82df389e21f5eb064321d"}
Jul 07 11:52:33.422746 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:33.422669    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4"
Jul 07 11:52:33.422849 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:33.422822    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4"
Jul 07 11:52:33.424161 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:33.424136    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:5000: connect: connection refused"
Jul 07 11:52:33.446649 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:33.446607    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" podStartSLOduration=3.644225168 podStartE2EDuration="7.446593528s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.80236836s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:52:33.444464722 +0000 UTC m=+743.791678346" watchObservedRunningTime="2026-07-07 11:52:33.446593528 +0000 UTC m=+743.793807150"
Jul 07 11:52:34.426381 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:34.426356    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4"
Jul 07 11:52:34.426843 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:34.426468    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:5000: connect: connection refused"
Jul 07 11:52:34.427476 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:34.427441    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:52:35.430882 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:35.430842    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:5000: connect: connection refused"
Jul 07 11:52:35.431327 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:35.431304    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:52:36.045771 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:36.045732    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 07 11:52:38.053129 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:38.053082    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 07 11:52:38.053506 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:38.053336    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:52:40.435191 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:40.435164    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4"
Jul 07 11:52:40.435771 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:40.435733    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:5000: connect: connection refused"
Jul 07 11:52:40.436181 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:40.436155    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:52:41.045216 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:41.045169    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 07 11:52:41.045404 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:41.045310    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8"
Jul 07 11:52:46.045841 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:46.045790    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 07 11:52:48.053268 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:48.053211    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 07 11:52:48.053680 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:48.053427    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8"
Jul 07 11:52:48.053680 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:48.053509    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:52:48.053680 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:48.053626    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8"
Jul 07 11:52:50.435986 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:50.435942    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:5000: connect: connection refused"
Jul 07 11:52:50.436500 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:50.436459    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:52:51.045864 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:51.045828    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 07 11:52:56.044963 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:56.044917    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 07 11:52:56.752734 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:56.752712    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8"
Jul 07 11:52:56.786105 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:56.786077    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb97329a-c3b8-47e2-b4d8-76e520b61610-cabundle-cert\") pod \"fb97329a-c3b8-47e2-b4d8-76e520b61610\" (UID: \"fb97329a-c3b8-47e2-b4d8-76e520b61610\") "
Jul 07 11:52:56.786250 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:56.786125    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69cv5\" (UniqueName: \"kubernetes.io/projected/fb97329a-c3b8-47e2-b4d8-76e520b61610-kube-api-access-69cv5\") pod \"fb97329a-c3b8-47e2-b4d8-76e520b61610\" (UID: \"fb97329a-c3b8-47e2-b4d8-76e520b61610\") "
Jul 07 11:52:56.786250 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:56.786164    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb97329a-c3b8-47e2-b4d8-76e520b61610-proxy-tls\") pod \"fb97329a-c3b8-47e2-b4d8-76e520b61610\" (UID: \"fb97329a-c3b8-47e2-b4d8-76e520b61610\") "
Jul 07 11:52:56.786250 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:56.786241    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb97329a-c3b8-47e2-b4d8-76e520b61610-kserve-provision-location\") pod \"fb97329a-c3b8-47e2-b4d8-76e520b61610\" (UID: \"fb97329a-c3b8-47e2-b4d8-76e520b61610\") "
Jul 07 11:52:56.786423 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:56.786269    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb97329a-c3b8-47e2-b4d8-76e520b61610-kube-rbac-proxy-sar-config\") pod \"fb97329a-c3b8-47e2-b4d8-76e520b61610\" (UID: \"fb97329a-c3b8-47e2-b4d8-76e520b61610\") "
Jul 07 11:52:56.786716 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:56.786488    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb97329a-c3b8-47e2-b4d8-76e520b61610-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fb97329a-c3b8-47e2-b4d8-76e520b61610" (UID: "fb97329a-c3b8-47e2-b4d8-76e520b61610"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:52:56.786716 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:56.786674    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb97329a-c3b8-47e2-b4d8-76e520b61610-cabundle-cert\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:52:56.786936 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:56.786887    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb97329a-c3b8-47e2-b4d8-76e520b61610-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fb97329a-c3b8-47e2-b4d8-76e520b61610" (UID: "fb97329a-c3b8-47e2-b4d8-76e520b61610"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 11:52:56.787076 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:56.786995    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb97329a-c3b8-47e2-b4d8-76e520b61610-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fb97329a-c3b8-47e2-b4d8-76e520b61610" (UID: "fb97329a-c3b8-47e2-b4d8-76e520b61610"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:52:56.788571 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:56.788539    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb97329a-c3b8-47e2-b4d8-76e520b61610-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fb97329a-c3b8-47e2-b4d8-76e520b61610" (UID: "fb97329a-c3b8-47e2-b4d8-76e520b61610"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:52:56.788671 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:56.788614    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb97329a-c3b8-47e2-b4d8-76e520b61610-kube-api-access-69cv5" (OuterVolumeSpecName: "kube-api-access-69cv5") pod "fb97329a-c3b8-47e2-b4d8-76e520b61610" (UID: "fb97329a-c3b8-47e2-b4d8-76e520b61610"). InnerVolumeSpecName "kube-api-access-69cv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 11:52:56.887481 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:56.887412    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69cv5\" (UniqueName: \"kubernetes.io/projected/fb97329a-c3b8-47e2-b4d8-76e520b61610-kube-api-access-69cv5\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:52:56.887481 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:56.887436    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb97329a-c3b8-47e2-b4d8-76e520b61610-proxy-tls\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:52:56.887481 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:56.887447    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb97329a-c3b8-47e2-b4d8-76e520b61610-kserve-provision-location\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:52:56.887481 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:56.887456    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb97329a-c3b8-47e2-b4d8-76e520b61610-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:52:57.532108 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:57.532068    2581 generic.go:356] "Generic (PLEG): container finished" podID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerID="6e5d97c4905ac4f55a21ec9df851eafbd20791f18e008ccc6a73d68974502d48" exitCode=0
Jul 07 11:52:57.532484 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:57.532129    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8"
Jul 07 11:52:57.532484 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:57.532150    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" event={"ID":"fb97329a-c3b8-47e2-b4d8-76e520b61610","Type":"ContainerDied","Data":"6e5d97c4905ac4f55a21ec9df851eafbd20791f18e008ccc6a73d68974502d48"}
Jul 07 11:52:57.532484 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:57.532217    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8" event={"ID":"fb97329a-c3b8-47e2-b4d8-76e520b61610","Type":"ContainerDied","Data":"ed4c147100d1d838568da6a4bfc8be2d0e71166699f45d54a60e3d5067fe0c31"}
Jul 07 11:52:57.532484 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:57.532252    2581 scope.go:117] "RemoveContainer" containerID="6e5d97c4905ac4f55a21ec9df851eafbd20791f18e008ccc6a73d68974502d48"
Jul 07 11:52:57.541668 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:57.541646    2581 scope.go:117] "RemoveContainer" containerID="b624bc5eedd2fad09e3557190bafb85440ed5fe73a4f189029fdc4b7f636eac1"
Jul 07 11:52:57.549231 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:57.549216    2581 scope.go:117] "RemoveContainer" containerID="878f0da76ff6e0ac224cef390889abe81a4cadebc72ed2b8842a79c1d22709bd"
Jul 07 11:52:57.556386 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:57.556369    2581 scope.go:117] "RemoveContainer" containerID="48c9c2d1c73c394d8649899a2a6135c1a81d3ac71895fd42b9e2ad9e5f7679c5"
Jul 07 11:52:57.556870 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:57.556851    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8"]
Jul 07 11:52:57.561244 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:57.561223    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-qzwq8"]
Jul 07 11:52:57.563757 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:57.563743    2581 scope.go:117] "RemoveContainer" containerID="6e5d97c4905ac4f55a21ec9df851eafbd20791f18e008ccc6a73d68974502d48"
Jul 07 11:52:57.564032 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:52:57.563996    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e5d97c4905ac4f55a21ec9df851eafbd20791f18e008ccc6a73d68974502d48\": container with ID starting with 6e5d97c4905ac4f55a21ec9df851eafbd20791f18e008ccc6a73d68974502d48 not found: ID does not exist" containerID="6e5d97c4905ac4f55a21ec9df851eafbd20791f18e008ccc6a73d68974502d48"
Jul 07 11:52:57.564113 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:57.564042    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e5d97c4905ac4f55a21ec9df851eafbd20791f18e008ccc6a73d68974502d48"} err="failed to get container status \"6e5d97c4905ac4f55a21ec9df851eafbd20791f18e008ccc6a73d68974502d48\": rpc error: code = NotFound desc = could not find container \"6e5d97c4905ac4f55a21ec9df851eafbd20791f18e008ccc6a73d68974502d48\": container with ID starting with 6e5d97c4905ac4f55a21ec9df851eafbd20791f18e008ccc6a73d68974502d48 not found: ID does not exist"
Jul 07 11:52:57.564113 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:57.564067    2581 scope.go:117] "RemoveContainer" containerID="b624bc5eedd2fad09e3557190bafb85440ed5fe73a4f189029fdc4b7f636eac1"
Jul 07 11:52:57.564323 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:52:57.564307    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b624bc5eedd2fad09e3557190bafb85440ed5fe73a4f189029fdc4b7f636eac1\": container with ID starting with b624bc5eedd2fad09e3557190bafb85440ed5fe73a4f189029fdc4b7f636eac1 not found: ID does not exist" containerID="b624bc5eedd2fad09e3557190bafb85440ed5fe73a4f189029fdc4b7f636eac1"
Jul 07 11:52:57.564364 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:57.564329    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b624bc5eedd2fad09e3557190bafb85440ed5fe73a4f189029fdc4b7f636eac1"} err="failed to get container status \"b624bc5eedd2fad09e3557190bafb85440ed5fe73a4f189029fdc4b7f636eac1\": rpc error: code = NotFound desc = could not find container \"b624bc5eedd2fad09e3557190bafb85440ed5fe73a4f189029fdc4b7f636eac1\": container with ID starting with b624bc5eedd2fad09e3557190bafb85440ed5fe73a4f189029fdc4b7f636eac1 not found: ID does not exist"
Jul 07 11:52:57.564364 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:57.564345    2581 scope.go:117] "RemoveContainer" containerID="878f0da76ff6e0ac224cef390889abe81a4cadebc72ed2b8842a79c1d22709bd"
Jul 07 11:52:57.564568 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:52:57.564549    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"878f0da76ff6e0ac224cef390889abe81a4cadebc72ed2b8842a79c1d22709bd\": container with ID starting with 878f0da76ff6e0ac224cef390889abe81a4cadebc72ed2b8842a79c1d22709bd not found: ID does not exist" containerID="878f0da76ff6e0ac224cef390889abe81a4cadebc72ed2b8842a79c1d22709bd"
Jul 07 11:52:57.564615 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:57.564577    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"878f0da76ff6e0ac224cef390889abe81a4cadebc72ed2b8842a79c1d22709bd"} err="failed to get container status \"878f0da76ff6e0ac224cef390889abe81a4cadebc72ed2b8842a79c1d22709bd\": rpc error: code = NotFound desc = could not find container \"878f0da76ff6e0ac224cef390889abe81a4cadebc72ed2b8842a79c1d22709bd\": container with ID starting with 878f0da76ff6e0ac224cef390889abe81a4cadebc72ed2b8842a79c1d22709bd not found: ID does not exist"
Jul 07 11:52:57.564615 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:57.564596    2581 scope.go:117] "RemoveContainer" containerID="48c9c2d1c73c394d8649899a2a6135c1a81d3ac71895fd42b9e2ad9e5f7679c5"
Jul 07 11:52:57.564812 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:52:57.564796    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48c9c2d1c73c394d8649899a2a6135c1a81d3ac71895fd42b9e2ad9e5f7679c5\": container with ID starting with 48c9c2d1c73c394d8649899a2a6135c1a81d3ac71895fd42b9e2ad9e5f7679c5 not found: ID does not exist" containerID="48c9c2d1c73c394d8649899a2a6135c1a81d3ac71895fd42b9e2ad9e5f7679c5"
Jul 07 11:52:57.564855 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:57.564817    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48c9c2d1c73c394d8649899a2a6135c1a81d3ac71895fd42b9e2ad9e5f7679c5"} err="failed to get container status \"48c9c2d1c73c394d8649899a2a6135c1a81d3ac71895fd42b9e2ad9e5f7679c5\": rpc error: code = NotFound desc = could not find container \"48c9c2d1c73c394d8649899a2a6135c1a81d3ac71895fd42b9e2ad9e5f7679c5\": container with ID starting with 48c9c2d1c73c394d8649899a2a6135c1a81d3ac71895fd42b9e2ad9e5f7679c5 not found: ID does not exist"
Jul 07 11:52:58.172429 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:52:58.172382    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" path="/var/lib/kubelet/pods/fb97329a-c3b8-47e2-b4d8-76e520b61610/volumes"
Jul 07 11:53:00.435656 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:53:00.435614    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:5000: connect: connection refused"
Jul 07 11:53:00.436110 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:53:00.436066    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:53:10.436243 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:53:10.436200    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:5000: connect: connection refused"
Jul 07 11:53:10.436705 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:53:10.436607    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:53:20.436073 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:53:20.436022    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:5000: connect: connection refused"
Jul 07 11:53:20.436491 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:53:20.436466    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:53:30.436328 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:53:30.436282    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:5000: connect: connection refused"
Jul 07 11:53:30.436866 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:53:30.436750    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:53:40.436005 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:53:40.435907    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:5000: connect: connection refused"
Jul 07 11:53:40.436449 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:53:40.436356    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:53:47.166945 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:53:47.166909    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4"
Jul 07 11:53:47.167383 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:53:47.167172    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4"
Jul 07 11:54:01.810269 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:01.810223    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9nrhc"]
Jul 07 11:54:01.810810 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:01.810653    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="kube-rbac-proxy"
Jul 07 11:54:01.810810 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:01.810670    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="kube-rbac-proxy"
Jul 07 11:54:01.810810 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:01.810691    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="storage-initializer"
Jul 07 11:54:01.810810 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:01.810696    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="storage-initializer"
Jul 07 11:54:01.810810 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:01.810704    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="kserve-container"
Jul 07 11:54:01.810810 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:01.810710    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="kserve-container"
Jul 07 11:54:01.810810 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:01.810716    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="agent"
Jul 07 11:54:01.810810 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:01.810721    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="agent"
Jul 07 11:54:01.810810 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:01.810769    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="agent"
Jul 07 11:54:01.810810 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:01.810777    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="kube-rbac-proxy"
Jul 07 11:54:01.810810 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:01.810785    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb97329a-c3b8-47e2-b4d8-76e520b61610" containerName="kserve-container"
Jul 07 11:54:01.813827 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:01.813811    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9nrhc"
Jul 07 11:54:01.817170 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:01.817147    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jul 07 11:54:01.817318 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:01.817183    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jul 07 11:54:01.835324 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:01.835297    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4"]
Jul 07 11:54:01.835599 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:01.835569    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="kserve-container" containerID="cri-o://5a96e6726c5cb14695fced3c0e62a54ab172c4b17df82df389e21f5eb064321d" gracePeriod=30
Jul 07 11:54:01.835716 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:01.835652    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="agent" containerID="cri-o://cdee6433c0287b079338976358ae494c862c09addbdd2b5f8b43aa555fa867ee" gracePeriod=30
Jul 07 11:54:01.835841 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:01.835809    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="kube-rbac-proxy" containerID="cri-o://7ec9a121ca1a763f4808677957507c225b0576c396e7d56604c0f5a539d3b76a" gracePeriod=30
Jul 07 11:54:01.837774 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:01.837751    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9nrhc"]
Jul 07 11:54:01.930021 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:01.929984    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6frf\" (UniqueName: \"kubernetes.io/projected/4b929818-519c-47b4-a528-911d0e368001-kube-api-access-t6frf\") pod \"message-dumper-predictor-6575bcfdfd-9nrhc\" (UID: \"4b929818-519c-47b4-a528-911d0e368001\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9nrhc"
Jul 07 11:54:01.930206 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:01.930115    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b929818-519c-47b4-a528-911d0e368001-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-9nrhc\" (UID: \"4b929818-519c-47b4-a528-911d0e368001\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9nrhc"
Jul 07 11:54:01.930206 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:01.930197    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b929818-519c-47b4-a528-911d0e368001-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-9nrhc\" (UID: \"4b929818-519c-47b4-a528-911d0e368001\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9nrhc"
Jul 07 11:54:02.031394 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:02.031354    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b929818-519c-47b4-a528-911d0e368001-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-9nrhc\" (UID: \"4b929818-519c-47b4-a528-911d0e368001\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9nrhc"
Jul 07 11:54:02.031593 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:02.031408    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6frf\" (UniqueName: \"kubernetes.io/projected/4b929818-519c-47b4-a528-911d0e368001-kube-api-access-t6frf\") pod \"message-dumper-predictor-6575bcfdfd-9nrhc\" (UID: \"4b929818-519c-47b4-a528-911d0e368001\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9nrhc"
Jul 07 11:54:02.031593 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:02.031477    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b929818-519c-47b4-a528-911d0e368001-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-9nrhc\" (UID: \"4b929818-519c-47b4-a528-911d0e368001\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9nrhc"
Jul 07 11:54:02.032257 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:02.032229    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b929818-519c-47b4-a528-911d0e368001-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-9nrhc\" (UID: \"4b929818-519c-47b4-a528-911d0e368001\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9nrhc"
Jul 07 11:54:02.033826 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:02.033803    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b929818-519c-47b4-a528-911d0e368001-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-9nrhc\" (UID: \"4b929818-519c-47b4-a528-911d0e368001\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9nrhc"
Jul 07 11:54:02.042206 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:02.042183    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6frf\" (UniqueName: \"kubernetes.io/projected/4b929818-519c-47b4-a528-911d0e368001-kube-api-access-t6frf\") pod \"message-dumper-predictor-6575bcfdfd-9nrhc\" (UID: \"4b929818-519c-47b4-a528-911d0e368001\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9nrhc"
Jul 07 11:54:02.123874 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:02.123841    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9nrhc"
Jul 07 11:54:02.262925 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:02.262900    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9nrhc"]
Jul 07 11:54:02.265451 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:54:02.265413    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b929818_519c_47b4_a528_911d0e368001.slice/crio-7a808a814ec3001de99a1be960c219bf725fbf7ae0a00d23661b2d50fe8eb3ba WatchSource:0}: Error finding container 7a808a814ec3001de99a1be960c219bf725fbf7ae0a00d23661b2d50fe8eb3ba: Status 404 returned error can't find the container with id 7a808a814ec3001de99a1be960c219bf725fbf7ae0a00d23661b2d50fe8eb3ba
Jul 07 11:54:02.814698 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:02.814663    2581 generic.go:356] "Generic (PLEG): container finished" podID="e0484788-3062-45f5-9f46-44a0688db68f" containerID="7ec9a121ca1a763f4808677957507c225b0576c396e7d56604c0f5a539d3b76a" exitCode=2
Jul 07 11:54:02.815130 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:02.814738    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" event={"ID":"e0484788-3062-45f5-9f46-44a0688db68f","Type":"ContainerDied","Data":"7ec9a121ca1a763f4808677957507c225b0576c396e7d56604c0f5a539d3b76a"}
Jul 07 11:54:02.815930 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:02.815905    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9nrhc" event={"ID":"4b929818-519c-47b4-a528-911d0e368001","Type":"ContainerStarted","Data":"7a808a814ec3001de99a1be960c219bf725fbf7ae0a00d23661b2d50fe8eb3ba"}
Jul 07 11:54:03.822605 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:03.822567    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9nrhc" event={"ID":"4b929818-519c-47b4-a528-911d0e368001","Type":"ContainerStarted","Data":"be5d3a03a2e96f5ef7c541b56184a93363f7ac9926d226125a9b2901e8f28446"}
Jul 07 11:54:03.823124 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:03.822626    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9nrhc" event={"ID":"4b929818-519c-47b4-a528-911d0e368001","Type":"ContainerStarted","Data":"3ee18cc11e84669712a4ed36a45c585b2538ddbdf33b6225223266465ad60fc6"}
Jul 07 11:54:03.823124 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:03.822749    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9nrhc"
Jul 07 11:54:03.823124 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:03.822875    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9nrhc"
Jul 07 11:54:03.823940 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:03.823916    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9nrhc" podUID="4b929818-519c-47b4-a528-911d0e368001" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 07 11:54:03.843938 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:03.843878    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9nrhc" podStartSLOduration=1.5152760170000001 podStartE2EDuration="2.843862701s" totalImagesPullingTime="1.328586684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:54:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:54:03.842672886 +0000 UTC m=+834.189886510" watchObservedRunningTime="2026-07-07 11:54:03.843862701 +0000 UTC m=+834.191076324"
Jul 07 11:54:04.828756 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:04.828727    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9nrhc"
Jul 07 11:54:05.431188 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:05.431152    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 07 11:54:06.838376 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:06.838343    2581 generic.go:356] "Generic (PLEG): container finished" podID="e0484788-3062-45f5-9f46-44a0688db68f" containerID="5a96e6726c5cb14695fced3c0e62a54ab172c4b17df82df389e21f5eb064321d" exitCode=0
Jul 07 11:54:06.838767 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:06.838410    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" event={"ID":"e0484788-3062-45f5-9f46-44a0688db68f","Type":"ContainerDied","Data":"5a96e6726c5cb14695fced3c0e62a54ab172c4b17df82df389e21f5eb064321d"}
Jul 07 11:54:07.167240 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:07.167198    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:5000: connect: connection refused"
Jul 07 11:54:07.167548 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:07.167526    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:54:10.431545 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:10.431507    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 07 11:54:10.836729 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:10.836644    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9nrhc"
Jul 07 11:54:11.832912 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:11.832878    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc"]
Jul 07 11:54:11.836689 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:11.836666    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc"
Jul 07 11:54:11.839342 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:11.839322    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 07 11:54:11.839453 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:11.839366    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 07 11:54:11.849360 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:11.849338    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc"]
Jul 07 11:54:11.912902 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:11.912872    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb2zt\" (UniqueName: \"kubernetes.io/projected/83e4b486-4385-4e1f-854c-d4470a305428-kube-api-access-xb2zt\") pod \"isvc-logger-predictor-6697b4c4cc-sflbc\" (UID: \"83e4b486-4385-4e1f-854c-d4470a305428\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc"
Jul 07 11:54:11.913082 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:11.912931    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83e4b486-4385-4e1f-854c-d4470a305428-kserve-provision-location\") pod \"isvc-logger-predictor-6697b4c4cc-sflbc\" (UID: \"83e4b486-4385-4e1f-854c-d4470a305428\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc"
Jul 07 11:54:11.913082 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:11.912999    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83e4b486-4385-4e1f-854c-d4470a305428-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6697b4c4cc-sflbc\" (UID: \"83e4b486-4385-4e1f-854c-d4470a305428\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc"
Jul 07 11:54:11.913082 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:11.913046    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83e4b486-4385-4e1f-854c-d4470a305428-cabundle-cert\") pod \"isvc-logger-predictor-6697b4c4cc-sflbc\" (UID: \"83e4b486-4385-4e1f-854c-d4470a305428\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc"
Jul 07 11:54:11.913082 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:11.913076    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83e4b486-4385-4e1f-854c-d4470a305428-proxy-tls\") pod \"isvc-logger-predictor-6697b4c4cc-sflbc\" (UID: \"83e4b486-4385-4e1f-854c-d4470a305428\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc"
Jul 07 11:54:12.013866 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:12.013824    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83e4b486-4385-4e1f-854c-d4470a305428-kserve-provision-location\") pod \"isvc-logger-predictor-6697b4c4cc-sflbc\" (UID: \"83e4b486-4385-4e1f-854c-d4470a305428\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc"
Jul 07 11:54:12.014056 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:12.013899    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83e4b486-4385-4e1f-854c-d4470a305428-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6697b4c4cc-sflbc\" (UID: \"83e4b486-4385-4e1f-854c-d4470a305428\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc"
Jul 07 11:54:12.014056 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:12.013927    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83e4b486-4385-4e1f-854c-d4470a305428-cabundle-cert\") pod \"isvc-logger-predictor-6697b4c4cc-sflbc\" (UID: \"83e4b486-4385-4e1f-854c-d4470a305428\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc"
Jul 07 11:54:12.014056 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:12.013970    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83e4b486-4385-4e1f-854c-d4470a305428-proxy-tls\") pod \"isvc-logger-predictor-6697b4c4cc-sflbc\" (UID: \"83e4b486-4385-4e1f-854c-d4470a305428\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc"
Jul 07 11:54:12.014056 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:12.014029    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb2zt\" (UniqueName: \"kubernetes.io/projected/83e4b486-4385-4e1f-854c-d4470a305428-kube-api-access-xb2zt\") pod \"isvc-logger-predictor-6697b4c4cc-sflbc\" (UID: \"83e4b486-4385-4e1f-854c-d4470a305428\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc"
Jul 07 11:54:12.014305 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:12.014283    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83e4b486-4385-4e1f-854c-d4470a305428-kserve-provision-location\") pod \"isvc-logger-predictor-6697b4c4cc-sflbc\" (UID: \"83e4b486-4385-4e1f-854c-d4470a305428\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc"
Jul 07 11:54:12.014718 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:12.014695    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83e4b486-4385-4e1f-854c-d4470a305428-cabundle-cert\") pod \"isvc-logger-predictor-6697b4c4cc-sflbc\" (UID: \"83e4b486-4385-4e1f-854c-d4470a305428\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc"
Jul 07 11:54:12.016909 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:12.016892    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 07 11:54:12.016909 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:12.016902    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 07 11:54:12.022840 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:12.022823    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb2zt\" (UniqueName: \"kubernetes.io/projected/83e4b486-4385-4e1f-854c-d4470a305428-kube-api-access-xb2zt\") pod \"isvc-logger-predictor-6697b4c4cc-sflbc\" (UID: \"83e4b486-4385-4e1f-854c-d4470a305428\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc"
Jul 07 11:54:12.024671 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:54:12.024653    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Jul 07 11:54:12.024725 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:54:12.024710    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83e4b486-4385-4e1f-854c-d4470a305428-proxy-tls podName:83e4b486-4385-4e1f-854c-d4470a305428 nodeName:}" failed. No retries permitted until 2026-07-07 11:54:12.524697133 +0000 UTC m=+842.871910738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/83e4b486-4385-4e1f-854c-d4470a305428-proxy-tls") pod "isvc-logger-predictor-6697b4c4cc-sflbc" (UID: "83e4b486-4385-4e1f-854c-d4470a305428") : secret "isvc-logger-predictor-serving-cert" not found
Jul 07 11:54:12.025164 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:12.025145    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83e4b486-4385-4e1f-854c-d4470a305428-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6697b4c4cc-sflbc\" (UID: \"83e4b486-4385-4e1f-854c-d4470a305428\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc"
Jul 07 11:54:12.618395 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:12.618352    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83e4b486-4385-4e1f-854c-d4470a305428-proxy-tls\") pod \"isvc-logger-predictor-6697b4c4cc-sflbc\" (UID: \"83e4b486-4385-4e1f-854c-d4470a305428\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc"
Jul 07 11:54:12.620730 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:12.620705    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83e4b486-4385-4e1f-854c-d4470a305428-proxy-tls\") pod \"isvc-logger-predictor-6697b4c4cc-sflbc\" (UID: \"83e4b486-4385-4e1f-854c-d4470a305428\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc"
Jul 07 11:54:12.747958 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:12.747928    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc"
Jul 07 11:54:12.881518 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:12.881496    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc"]
Jul 07 11:54:12.883366 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:54:12.883342    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83e4b486_4385_4e1f_854c_d4470a305428.slice/crio-97f5931f1deca833697cff4533905ce53ca1f6d18aee60e3f420ea8009f3c923 WatchSource:0}: Error finding container 97f5931f1deca833697cff4533905ce53ca1f6d18aee60e3f420ea8009f3c923: Status 404 returned error can't find the container with id 97f5931f1deca833697cff4533905ce53ca1f6d18aee60e3f420ea8009f3c923
Jul 07 11:54:13.871791 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:13.871755    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" event={"ID":"83e4b486-4385-4e1f-854c-d4470a305428","Type":"ContainerStarted","Data":"56110ea244a091c53c4ec5ec556cd64fbf84bf1477b134eee990d04498f5d0ad"}
Jul 07 11:54:13.871937 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:13.871855    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" event={"ID":"83e4b486-4385-4e1f-854c-d4470a305428","Type":"ContainerStarted","Data":"97f5931f1deca833697cff4533905ce53ca1f6d18aee60e3f420ea8009f3c923"}
Jul 07 11:54:15.431744 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:15.431694    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 07 11:54:15.432202 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:15.431850    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4"
Jul 07 11:54:16.886917 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:16.886884    2581 generic.go:356] "Generic (PLEG): container finished" podID="83e4b486-4385-4e1f-854c-d4470a305428" containerID="56110ea244a091c53c4ec5ec556cd64fbf84bf1477b134eee990d04498f5d0ad" exitCode=0
Jul 07 11:54:16.887250 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:16.886922    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" event={"ID":"83e4b486-4385-4e1f-854c-d4470a305428","Type":"ContainerDied","Data":"56110ea244a091c53c4ec5ec556cd64fbf84bf1477b134eee990d04498f5d0ad"}
Jul 07 11:54:17.167169 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:17.167133    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:5000: connect: connection refused"
Jul 07 11:54:17.167447 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:17.167425    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:54:17.894330 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:17.894292    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" event={"ID":"83e4b486-4385-4e1f-854c-d4470a305428","Type":"ContainerStarted","Data":"927774dcbffa40af1d281bf7451c7dc9ed3826a4fd0585d4bd1bdc2559d8f1c2"}
Jul 07 11:54:17.894709 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:17.894363    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" event={"ID":"83e4b486-4385-4e1f-854c-d4470a305428","Type":"ContainerStarted","Data":"d97323a9a8dd25e133646dd259a14775ce687351e2d2e09c63cebc87040a835a"}
Jul 07 11:54:17.894709 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:17.894388    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" event={"ID":"83e4b486-4385-4e1f-854c-d4470a305428","Type":"ContainerStarted","Data":"51389dc5ccd0161d7a56e6c731d9e1c6798987a841683f397b2e48c204eaae17"}
Jul 07 11:54:17.894709 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:17.894659    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc"
Jul 07 11:54:17.894840 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:17.894817    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc"
Jul 07 11:54:17.896096 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:17.896068    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 07 11:54:17.921054 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:17.920998    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" podStartSLOduration=3.111679062 podStartE2EDuration="6.920985923s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.809306861s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:54:17.918454727 +0000 UTC m=+848.265668352" watchObservedRunningTime="2026-07-07 11:54:17.920985923 +0000 UTC m=+848.268199546"
Jul 07 11:54:18.898771 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:18.898738    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc"
Jul 07 11:54:18.899203 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:18.898832    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 07 11:54:18.899903 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:18.899873    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:54:19.903432 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:19.903387    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 07 11:54:19.903879 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:19.903783    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:54:20.431891 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:20.431845    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 07 11:54:24.908134 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:24.908101    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc"
Jul 07 11:54:24.908754 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:24.908722    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 07 11:54:24.908949 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:24.908923    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:54:25.431445 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:25.431402    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 07 11:54:27.166436 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:27.166384    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:5000: connect: connection refused"
Jul 07 11:54:27.166848 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:27.166537    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4"
Jul 07 11:54:27.166848 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:27.166725    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:54:27.166848 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:27.166824    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4"
Jul 07 11:54:30.431583 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:30.431543    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 07 11:54:31.966810 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:31.966781    2581 generic.go:356] "Generic (PLEG): container finished" podID="e0484788-3062-45f5-9f46-44a0688db68f" containerID="cdee6433c0287b079338976358ae494c862c09addbdd2b5f8b43aa555fa867ee" exitCode=0
Jul 07 11:54:31.967185 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:31.966817    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" event={"ID":"e0484788-3062-45f5-9f46-44a0688db68f","Type":"ContainerDied","Data":"cdee6433c0287b079338976358ae494c862c09addbdd2b5f8b43aa555fa867ee"}
Jul 07 11:54:31.985935 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:31.985912    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4"
Jul 07 11:54:31.992329 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:31.992310    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0484788-3062-45f5-9f46-44a0688db68f-proxy-tls\") pod \"e0484788-3062-45f5-9f46-44a0688db68f\" (UID: \"e0484788-3062-45f5-9f46-44a0688db68f\") "
Jul 07 11:54:31.992432 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:31.992352    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0484788-3062-45f5-9f46-44a0688db68f-kube-rbac-proxy-sar-config\") pod \"e0484788-3062-45f5-9f46-44a0688db68f\" (UID: \"e0484788-3062-45f5-9f46-44a0688db68f\") "
Jul 07 11:54:31.992432 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:31.992385    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0484788-3062-45f5-9f46-44a0688db68f-cabundle-cert\") pod \"e0484788-3062-45f5-9f46-44a0688db68f\" (UID: \"e0484788-3062-45f5-9f46-44a0688db68f\") "
Jul 07 11:54:31.992714 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:31.992687    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0484788-3062-45f5-9f46-44a0688db68f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e0484788-3062-45f5-9f46-44a0688db68f" (UID: "e0484788-3062-45f5-9f46-44a0688db68f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:54:31.992819 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:31.992766    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0484788-3062-45f5-9f46-44a0688db68f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e0484788-3062-45f5-9f46-44a0688db68f" (UID: "e0484788-3062-45f5-9f46-44a0688db68f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:54:31.994294 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:31.994272    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0484788-3062-45f5-9f46-44a0688db68f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e0484788-3062-45f5-9f46-44a0688db68f" (UID: "e0484788-3062-45f5-9f46-44a0688db68f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:54:32.093139 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:32.093067    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0484788-3062-45f5-9f46-44a0688db68f-kserve-provision-location\") pod \"e0484788-3062-45f5-9f46-44a0688db68f\" (UID: \"e0484788-3062-45f5-9f46-44a0688db68f\") "
Jul 07 11:54:32.093139 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:32.093107    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wl62p\" (UniqueName: \"kubernetes.io/projected/e0484788-3062-45f5-9f46-44a0688db68f-kube-api-access-wl62p\") pod \"e0484788-3062-45f5-9f46-44a0688db68f\" (UID: \"e0484788-3062-45f5-9f46-44a0688db68f\") "
Jul 07 11:54:32.093411 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:32.093391    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0484788-3062-45f5-9f46-44a0688db68f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e0484788-3062-45f5-9f46-44a0688db68f" (UID: "e0484788-3062-45f5-9f46-44a0688db68f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 11:54:32.093448 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:32.093401    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0484788-3062-45f5-9f46-44a0688db68f-proxy-tls\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:54:32.093448 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:32.093440    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0484788-3062-45f5-9f46-44a0688db68f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:54:32.093517 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:32.093451    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0484788-3062-45f5-9f46-44a0688db68f-cabundle-cert\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:54:32.095096 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:32.095073    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0484788-3062-45f5-9f46-44a0688db68f-kube-api-access-wl62p" (OuterVolumeSpecName: "kube-api-access-wl62p") pod "e0484788-3062-45f5-9f46-44a0688db68f" (UID: "e0484788-3062-45f5-9f46-44a0688db68f"). InnerVolumeSpecName "kube-api-access-wl62p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 11:54:32.193963 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:32.193939    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0484788-3062-45f5-9f46-44a0688db68f-kserve-provision-location\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:54:32.193963 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:32.193961    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl62p\" (UniqueName: \"kubernetes.io/projected/e0484788-3062-45f5-9f46-44a0688db68f-kube-api-access-wl62p\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:54:32.973488 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:32.973454    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4" event={"ID":"e0484788-3062-45f5-9f46-44a0688db68f","Type":"ContainerDied","Data":"9e6b17d3080886ed1b258f4b5fbe480213df685627c60be6ebcb28da7108c4f2"}
Jul 07 11:54:32.973488 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:32.973478    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4"
Jul 07 11:54:32.974035 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:32.973544    2581 scope.go:117] "RemoveContainer" containerID="cdee6433c0287b079338976358ae494c862c09addbdd2b5f8b43aa555fa867ee"
Jul 07 11:54:32.981556 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:32.981537    2581 scope.go:117] "RemoveContainer" containerID="7ec9a121ca1a763f4808677957507c225b0576c396e7d56604c0f5a539d3b76a"
Jul 07 11:54:32.988704 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:32.988683    2581 scope.go:117] "RemoveContainer" containerID="5a96e6726c5cb14695fced3c0e62a54ab172c4b17df82df389e21f5eb064321d"
Jul 07 11:54:32.994571 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:32.994549    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4"]
Jul 07 11:54:32.997105 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:32.997084    2581 scope.go:117] "RemoveContainer" containerID="1117e4ad6f34899abfd61b38aed9babe045090e254a19d2de125a2c8dbdda531"
Jul 07 11:54:33.001404 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:33.001382    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-6nlf4"]
Jul 07 11:54:34.171142 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:34.171109    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0484788-3062-45f5-9f46-44a0688db68f" path="/var/lib/kubelet/pods/e0484788-3062-45f5-9f46-44a0688db68f/volumes"
Jul 07 11:54:34.909214 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:34.909175    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 07 11:54:34.909623 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:34.909597    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:54:44.909217 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:44.909180    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 07 11:54:44.909707 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:44.909688    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:54:54.909526 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:54.909486    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 07 11:54:54.909968 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:54:54.909936    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:55:04.909585 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:04.909484    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 07 11:55:04.910150 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:04.909931    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:55:12.074360 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:12.074333    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kf48j_4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801/ovn-acl-logging/0.log"
Jul 07 11:55:12.075216 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:12.075196    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kf48j_4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801/ovn-acl-logging/0.log"
Jul 07 11:55:14.909094 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:14.909056    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 07 11:55:14.909536 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:14.909510    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:55:24.909359 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:24.909319    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 07 11:55:24.909818 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:24.909790    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:55:34.909345 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:34.909310    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc"
Jul 07 11:55:34.909718 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:34.909519    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc"
Jul 07 11:55:46.875907 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:46.875855    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-9nrhc_4b929818-519c-47b4-a528-911d0e368001/kserve-container/0.log"
Jul 07 11:55:47.205817 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.205740    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc"]
Jul 07 11:55:47.206078 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.206052    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="kserve-container" containerID="cri-o://51389dc5ccd0161d7a56e6c731d9e1c6798987a841683f397b2e48c204eaae17" gracePeriod=30
Jul 07 11:55:47.206186 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.206115    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="agent" containerID="cri-o://927774dcbffa40af1d281bf7451c7dc9ed3826a4fd0585d4bd1bdc2559d8f1c2" gracePeriod=30
Jul 07 11:55:47.206250 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.206148    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="kube-rbac-proxy" containerID="cri-o://d97323a9a8dd25e133646dd259a14775ce687351e2d2e09c63cebc87040a835a" gracePeriod=30
Jul 07 11:55:47.237536 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.237509    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk"]
Jul 07 11:55:47.238046 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.238028    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="storage-initializer"
Jul 07 11:55:47.238138 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.238050    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="storage-initializer"
Jul 07 11:55:47.238138 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.238069    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="kserve-container"
Jul 07 11:55:47.238138 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.238077    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="kserve-container"
Jul 07 11:55:47.238138 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.238090    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="kube-rbac-proxy"
Jul 07 11:55:47.238138 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.238098    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="kube-rbac-proxy"
Jul 07 11:55:47.238138 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.238111    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="agent"
Jul 07 11:55:47.238138 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.238119    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="agent"
Jul 07 11:55:47.238489 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.238214    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="kserve-container"
Jul 07 11:55:47.238489 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.238227    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="agent"
Jul 07 11:55:47.238489 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.238236    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0484788-3062-45f5-9f46-44a0688db68f" containerName="kube-rbac-proxy"
Jul 07 11:55:47.241687 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.241670    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk"
Jul 07 11:55:47.244923 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.244729    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 07 11:55:47.245103 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.245083    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 07 11:55:47.256159 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.256137    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk"]
Jul 07 11:55:47.258280 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.258254    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9nrhc"]
Jul 07 11:55:47.258519 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.258493    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9nrhc" podUID="4b929818-519c-47b4-a528-911d0e368001" containerName="kserve-container" containerID="cri-o://3ee18cc11e84669712a4ed36a45c585b2538ddbdf33b6225223266465ad60fc6" gracePeriod=30
Jul 07 11:55:47.258594 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.258572    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9nrhc" podUID="4b929818-519c-47b4-a528-911d0e368001" containerName="kube-rbac-proxy" containerID="cri-o://be5d3a03a2e96f5ef7c541b56184a93363f7ac9926d226125a9b2901e8f28446" gracePeriod=30
Jul 07 11:55:47.315887 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.315863    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92m68\" (UniqueName: \"kubernetes.io/projected/2a1155fd-96fe-41b1-813d-df25f6ee3a40-kube-api-access-92m68\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk\" (UID: \"2a1155fd-96fe-41b1-813d-df25f6ee3a40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk"
Jul 07 11:55:47.316032 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.315892    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a1155fd-96fe-41b1-813d-df25f6ee3a40-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk\" (UID: \"2a1155fd-96fe-41b1-813d-df25f6ee3a40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk"
Jul 07 11:55:47.316032 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.315922    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a1155fd-96fe-41b1-813d-df25f6ee3a40-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk\" (UID: \"2a1155fd-96fe-41b1-813d-df25f6ee3a40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk"
Jul 07 11:55:47.316032 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.315982    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a1155fd-96fe-41b1-813d-df25f6ee3a40-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk\" (UID: \"2a1155fd-96fe-41b1-813d-df25f6ee3a40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk"
Jul 07 11:55:47.316158 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.316084    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a1155fd-96fe-41b1-813d-df25f6ee3a40-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk\" (UID: \"2a1155fd-96fe-41b1-813d-df25f6ee3a40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk"
Jul 07 11:55:47.417138 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.417094    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a1155fd-96fe-41b1-813d-df25f6ee3a40-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk\" (UID: \"2a1155fd-96fe-41b1-813d-df25f6ee3a40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk"
Jul 07 11:55:47.417322 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.417164    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a1155fd-96fe-41b1-813d-df25f6ee3a40-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk\" (UID: \"2a1155fd-96fe-41b1-813d-df25f6ee3a40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk"
Jul 07 11:55:47.417322 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.417193    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92m68\" (UniqueName: \"kubernetes.io/projected/2a1155fd-96fe-41b1-813d-df25f6ee3a40-kube-api-access-92m68\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk\" (UID: \"2a1155fd-96fe-41b1-813d-df25f6ee3a40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk"
Jul 07 11:55:47.417322 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.417215    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a1155fd-96fe-41b1-813d-df25f6ee3a40-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk\" (UID: \"2a1155fd-96fe-41b1-813d-df25f6ee3a40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk"
Jul 07 11:55:47.417322 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.417261    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a1155fd-96fe-41b1-813d-df25f6ee3a40-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk\" (UID: \"2a1155fd-96fe-41b1-813d-df25f6ee3a40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk"
Jul 07 11:55:47.417543 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:55:47.417400    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v1-predictor-serving-cert: secret "isvc-autogluon-v1-predictor-serving-cert" not found
Jul 07 11:55:47.417543 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:55:47.417460    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a1155fd-96fe-41b1-813d-df25f6ee3a40-proxy-tls podName:2a1155fd-96fe-41b1-813d-df25f6ee3a40 nodeName:}" failed. No retries permitted until 2026-07-07 11:55:47.917440222 +0000 UTC m=+938.264653825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2a1155fd-96fe-41b1-813d-df25f6ee3a40-proxy-tls") pod "isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk" (UID: "2a1155fd-96fe-41b1-813d-df25f6ee3a40") : secret "isvc-autogluon-v1-predictor-serving-cert" not found
Jul 07 11:55:47.417676 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.417647    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a1155fd-96fe-41b1-813d-df25f6ee3a40-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk\" (UID: \"2a1155fd-96fe-41b1-813d-df25f6ee3a40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk"
Jul 07 11:55:47.417895 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.417870    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a1155fd-96fe-41b1-813d-df25f6ee3a40-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk\" (UID: \"2a1155fd-96fe-41b1-813d-df25f6ee3a40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk"
Jul 07 11:55:47.417965 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.417870    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a1155fd-96fe-41b1-813d-df25f6ee3a40-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk\" (UID: \"2a1155fd-96fe-41b1-813d-df25f6ee3a40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk"
Jul 07 11:55:47.426520 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.426488    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92m68\" (UniqueName: \"kubernetes.io/projected/2a1155fd-96fe-41b1-813d-df25f6ee3a40-kube-api-access-92m68\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk\" (UID: \"2a1155fd-96fe-41b1-813d-df25f6ee3a40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk"
Jul 07 11:55:47.522217 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.522194    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9nrhc"
Jul 07 11:55:47.618192 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.618159    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b929818-519c-47b4-a528-911d0e368001-kube-rbac-proxy-sar-config\") pod \"4b929818-519c-47b4-a528-911d0e368001\" (UID: \"4b929818-519c-47b4-a528-911d0e368001\") "
Jul 07 11:55:47.618355 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.618232    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6frf\" (UniqueName: \"kubernetes.io/projected/4b929818-519c-47b4-a528-911d0e368001-kube-api-access-t6frf\") pod \"4b929818-519c-47b4-a528-911d0e368001\" (UID: \"4b929818-519c-47b4-a528-911d0e368001\") "
Jul 07 11:55:47.618422 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.618398    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b929818-519c-47b4-a528-911d0e368001-proxy-tls\") pod \"4b929818-519c-47b4-a528-911d0e368001\" (UID: \"4b929818-519c-47b4-a528-911d0e368001\") "
Jul 07 11:55:47.618557 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.618535    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b929818-519c-47b4-a528-911d0e368001-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4b929818-519c-47b4-a528-911d0e368001" (UID: "4b929818-519c-47b4-a528-911d0e368001"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:55:47.618689 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.618673    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b929818-519c-47b4-a528-911d0e368001-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:55:47.620225 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.620202    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b929818-519c-47b4-a528-911d0e368001-kube-api-access-t6frf" (OuterVolumeSpecName: "kube-api-access-t6frf") pod "4b929818-519c-47b4-a528-911d0e368001" (UID: "4b929818-519c-47b4-a528-911d0e368001"). InnerVolumeSpecName "kube-api-access-t6frf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 11:55:47.620305 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.620273    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b929818-519c-47b4-a528-911d0e368001-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4b929818-519c-47b4-a528-911d0e368001" (UID: "4b929818-519c-47b4-a528-911d0e368001"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:55:47.719689 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.719651    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6frf\" (UniqueName: \"kubernetes.io/projected/4b929818-519c-47b4-a528-911d0e368001-kube-api-access-t6frf\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:55:47.719689 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.719682    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b929818-519c-47b4-a528-911d0e368001-proxy-tls\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:55:47.922287 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.922254    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a1155fd-96fe-41b1-813d-df25f6ee3a40-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk\" (UID: \"2a1155fd-96fe-41b1-813d-df25f6ee3a40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk"
Jul 07 11:55:47.924561 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:47.924542    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a1155fd-96fe-41b1-813d-df25f6ee3a40-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk\" (UID: \"2a1155fd-96fe-41b1-813d-df25f6ee3a40\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk"
Jul 07 11:55:48.156419 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:48.156376    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk"
Jul 07 11:55:48.290791 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:48.290751    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk"]
Jul 07 11:55:48.313022 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:48.312985    2581 generic.go:356] "Generic (PLEG): container finished" podID="83e4b486-4385-4e1f-854c-d4470a305428" containerID="d97323a9a8dd25e133646dd259a14775ce687351e2d2e09c63cebc87040a835a" exitCode=2
Jul 07 11:55:48.313117 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:48.313039    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" event={"ID":"83e4b486-4385-4e1f-854c-d4470a305428","Type":"ContainerDied","Data":"d97323a9a8dd25e133646dd259a14775ce687351e2d2e09c63cebc87040a835a"}
Jul 07 11:55:48.314591 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:48.314568    2581 generic.go:356] "Generic (PLEG): container finished" podID="4b929818-519c-47b4-a528-911d0e368001" containerID="be5d3a03a2e96f5ef7c541b56184a93363f7ac9926d226125a9b2901e8f28446" exitCode=2
Jul 07 11:55:48.314591 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:48.314588    2581 generic.go:356] "Generic (PLEG): container finished" podID="4b929818-519c-47b4-a528-911d0e368001" containerID="3ee18cc11e84669712a4ed36a45c585b2538ddbdf33b6225223266465ad60fc6" exitCode=2
Jul 07 11:55:48.314831 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:48.314629    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9nrhc"
Jul 07 11:55:48.314831 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:48.314660    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9nrhc" event={"ID":"4b929818-519c-47b4-a528-911d0e368001","Type":"ContainerDied","Data":"be5d3a03a2e96f5ef7c541b56184a93363f7ac9926d226125a9b2901e8f28446"}
Jul 07 11:55:48.314831 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:48.314733    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9nrhc" event={"ID":"4b929818-519c-47b4-a528-911d0e368001","Type":"ContainerDied","Data":"3ee18cc11e84669712a4ed36a45c585b2538ddbdf33b6225223266465ad60fc6"}
Jul 07 11:55:48.314831 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:48.314777    2581 scope.go:117] "RemoveContainer" containerID="be5d3a03a2e96f5ef7c541b56184a93363f7ac9926d226125a9b2901e8f28446"
Jul 07 11:55:48.314831 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:48.314797    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9nrhc" event={"ID":"4b929818-519c-47b4-a528-911d0e368001","Type":"ContainerDied","Data":"7a808a814ec3001de99a1be960c219bf725fbf7ae0a00d23661b2d50fe8eb3ba"}
Jul 07 11:55:48.316287 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:48.316267    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk" event={"ID":"2a1155fd-96fe-41b1-813d-df25f6ee3a40","Type":"ContainerStarted","Data":"05c30af1b9b05b746a8dc5889f1ae4b065c4adb1f273b99af999e5327b183703"}
Jul 07 11:55:48.335141 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:48.335103    2581 scope.go:117] "RemoveContainer" containerID="3ee18cc11e84669712a4ed36a45c585b2538ddbdf33b6225223266465ad60fc6"
Jul 07 11:55:48.344481 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:48.344458    2581 scope.go:117] "RemoveContainer" containerID="be5d3a03a2e96f5ef7c541b56184a93363f7ac9926d226125a9b2901e8f28446"
Jul 07 11:55:48.344719 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:55:48.344699    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be5d3a03a2e96f5ef7c541b56184a93363f7ac9926d226125a9b2901e8f28446\": container with ID starting with be5d3a03a2e96f5ef7c541b56184a93363f7ac9926d226125a9b2901e8f28446 not found: ID does not exist" containerID="be5d3a03a2e96f5ef7c541b56184a93363f7ac9926d226125a9b2901e8f28446"
Jul 07 11:55:48.344789 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:48.344727    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be5d3a03a2e96f5ef7c541b56184a93363f7ac9926d226125a9b2901e8f28446"} err="failed to get container status \"be5d3a03a2e96f5ef7c541b56184a93363f7ac9926d226125a9b2901e8f28446\": rpc error: code = NotFound desc = could not find container \"be5d3a03a2e96f5ef7c541b56184a93363f7ac9926d226125a9b2901e8f28446\": container with ID starting with be5d3a03a2e96f5ef7c541b56184a93363f7ac9926d226125a9b2901e8f28446 not found: ID does not exist"
Jul 07 11:55:48.344789 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:48.344744    2581 scope.go:117] "RemoveContainer" containerID="3ee18cc11e84669712a4ed36a45c585b2538ddbdf33b6225223266465ad60fc6"
Jul 07 11:55:48.345031 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:55:48.344993    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ee18cc11e84669712a4ed36a45c585b2538ddbdf33b6225223266465ad60fc6\": container with ID starting with 3ee18cc11e84669712a4ed36a45c585b2538ddbdf33b6225223266465ad60fc6 not found: ID does not exist" containerID="3ee18cc11e84669712a4ed36a45c585b2538ddbdf33b6225223266465ad60fc6"
Jul 07 11:55:48.345107 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:48.345029    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ee18cc11e84669712a4ed36a45c585b2538ddbdf33b6225223266465ad60fc6"} err="failed to get container status \"3ee18cc11e84669712a4ed36a45c585b2538ddbdf33b6225223266465ad60fc6\": rpc error: code = NotFound desc = could not find container \"3ee18cc11e84669712a4ed36a45c585b2538ddbdf33b6225223266465ad60fc6\": container with ID starting with 3ee18cc11e84669712a4ed36a45c585b2538ddbdf33b6225223266465ad60fc6 not found: ID does not exist"
Jul 07 11:55:48.345107 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:48.345050    2581 scope.go:117] "RemoveContainer" containerID="be5d3a03a2e96f5ef7c541b56184a93363f7ac9926d226125a9b2901e8f28446"
Jul 07 11:55:48.345296 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:48.345276    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be5d3a03a2e96f5ef7c541b56184a93363f7ac9926d226125a9b2901e8f28446"} err="failed to get container status \"be5d3a03a2e96f5ef7c541b56184a93363f7ac9926d226125a9b2901e8f28446\": rpc error: code = NotFound desc = could not find container \"be5d3a03a2e96f5ef7c541b56184a93363f7ac9926d226125a9b2901e8f28446\": container with ID starting with be5d3a03a2e96f5ef7c541b56184a93363f7ac9926d226125a9b2901e8f28446 not found: ID does not exist"
Jul 07 11:55:48.345366 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:48.345297    2581 scope.go:117] "RemoveContainer" containerID="3ee18cc11e84669712a4ed36a45c585b2538ddbdf33b6225223266465ad60fc6"
Jul 07 11:55:48.345574 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:48.345552    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ee18cc11e84669712a4ed36a45c585b2538ddbdf33b6225223266465ad60fc6"} err="failed to get container status \"3ee18cc11e84669712a4ed36a45c585b2538ddbdf33b6225223266465ad60fc6\": rpc error: code = NotFound desc = could not find container \"3ee18cc11e84669712a4ed36a45c585b2538ddbdf33b6225223266465ad60fc6\": container with ID starting with 3ee18cc11e84669712a4ed36a45c585b2538ddbdf33b6225223266465ad60fc6 not found: ID does not exist"
Jul 07 11:55:48.345885 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:48.345867    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9nrhc"]
Jul 07 11:55:48.351230 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:48.351211    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9nrhc"]
Jul 07 11:55:49.322189 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:49.322144    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk" event={"ID":"2a1155fd-96fe-41b1-813d-df25f6ee3a40","Type":"ContainerStarted","Data":"192734f2be42dbadb4eda481bcffcb93efaeafc297ab3e34ba353515c4a03ea8"}
Jul 07 11:55:49.904331 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:49.904289    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 07 11:55:50.176683 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:50.176591    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b929818-519c-47b4-a528-911d0e368001" path="/var/lib/kubelet/pods/4b929818-519c-47b4-a528-911d0e368001/volumes"
Jul 07 11:55:52.337186 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:52.337152    2581 generic.go:356] "Generic (PLEG): container finished" podID="83e4b486-4385-4e1f-854c-d4470a305428" containerID="51389dc5ccd0161d7a56e6c731d9e1c6798987a841683f397b2e48c204eaae17" exitCode=0
Jul 07 11:55:52.337186 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:52.337190    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" event={"ID":"83e4b486-4385-4e1f-854c-d4470a305428","Type":"ContainerDied","Data":"51389dc5ccd0161d7a56e6c731d9e1c6798987a841683f397b2e48c204eaae17"}
Jul 07 11:55:54.903801 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:54.903756    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 07 11:55:54.909169 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:54.909138    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 07 11:55:54.909520 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:54.909496    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:55:59.903905 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:59.903860    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 07 11:55:59.904312 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:55:59.904041    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc"
Jul 07 11:56:04.904195 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:04.904156    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 07 11:56:04.908571 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:04.908547    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 07 11:56:04.908903 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:04.908882    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:56:09.409775 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:09.409743    2581 generic.go:356] "Generic (PLEG): container finished" podID="2a1155fd-96fe-41b1-813d-df25f6ee3a40" containerID="192734f2be42dbadb4eda481bcffcb93efaeafc297ab3e34ba353515c4a03ea8" exitCode=0
Jul 07 11:56:09.410238 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:09.409790    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk" event={"ID":"2a1155fd-96fe-41b1-813d-df25f6ee3a40","Type":"ContainerDied","Data":"192734f2be42dbadb4eda481bcffcb93efaeafc297ab3e34ba353515c4a03ea8"}
Jul 07 11:56:09.411310 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:09.411293    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 11:56:09.904322 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:09.904280    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 07 11:56:14.904340 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:14.904295    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 07 11:56:14.909188 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:14.909155    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 07 11:56:14.909336 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:14.909274    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc"
Jul 07 11:56:14.909488 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:14.909467    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:56:14.909570 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:14.909558    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc"
Jul 07 11:56:17.380442 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:17.380420    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc"
Jul 07 11:56:17.453727 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:17.453686    2581 generic.go:356] "Generic (PLEG): container finished" podID="83e4b486-4385-4e1f-854c-d4470a305428" containerID="927774dcbffa40af1d281bf7451c7dc9ed3826a4fd0585d4bd1bdc2559d8f1c2" exitCode=0
Jul 07 11:56:17.453924 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:17.453759    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" event={"ID":"83e4b486-4385-4e1f-854c-d4470a305428","Type":"ContainerDied","Data":"927774dcbffa40af1d281bf7451c7dc9ed3826a4fd0585d4bd1bdc2559d8f1c2"}
Jul 07 11:56:17.453924 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:17.453775    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc"
Jul 07 11:56:17.453924 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:17.453823    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc" event={"ID":"83e4b486-4385-4e1f-854c-d4470a305428","Type":"ContainerDied","Data":"97f5931f1deca833697cff4533905ce53ca1f6d18aee60e3f420ea8009f3c923"}
Jul 07 11:56:17.453924 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:17.453859    2581 scope.go:117] "RemoveContainer" containerID="927774dcbffa40af1d281bf7451c7dc9ed3826a4fd0585d4bd1bdc2559d8f1c2"
Jul 07 11:56:17.464538 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:17.464512    2581 scope.go:117] "RemoveContainer" containerID="d97323a9a8dd25e133646dd259a14775ce687351e2d2e09c63cebc87040a835a"
Jul 07 11:56:17.472984 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:17.472949    2581 scope.go:117] "RemoveContainer" containerID="51389dc5ccd0161d7a56e6c731d9e1c6798987a841683f397b2e48c204eaae17"
Jul 07 11:56:17.477398 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:17.477373    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83e4b486-4385-4e1f-854c-d4470a305428-cabundle-cert\") pod \"83e4b486-4385-4e1f-854c-d4470a305428\" (UID: \"83e4b486-4385-4e1f-854c-d4470a305428\") "
Jul 07 11:56:17.477515 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:17.477409    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83e4b486-4385-4e1f-854c-d4470a305428-kube-rbac-proxy-sar-config\") pod \"83e4b486-4385-4e1f-854c-d4470a305428\" (UID: \"83e4b486-4385-4e1f-854c-d4470a305428\") "
Jul 07 11:56:17.477515 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:17.477437    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83e4b486-4385-4e1f-854c-d4470a305428-proxy-tls\") pod \"83e4b486-4385-4e1f-854c-d4470a305428\" (UID: \"83e4b486-4385-4e1f-854c-d4470a305428\") "
Jul 07 11:56:17.477515 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:17.477492    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83e4b486-4385-4e1f-854c-d4470a305428-kserve-provision-location\") pod \"83e4b486-4385-4e1f-854c-d4470a305428\" (UID: \"83e4b486-4385-4e1f-854c-d4470a305428\") "
Jul 07 11:56:17.477673 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:17.477571    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xb2zt\" (UniqueName: \"kubernetes.io/projected/83e4b486-4385-4e1f-854c-d4470a305428-kube-api-access-xb2zt\") pod \"83e4b486-4385-4e1f-854c-d4470a305428\" (UID: \"83e4b486-4385-4e1f-854c-d4470a305428\") "
Jul 07 11:56:17.477842 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:17.477813    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83e4b486-4385-4e1f-854c-d4470a305428-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "83e4b486-4385-4e1f-854c-d4470a305428" (UID: "83e4b486-4385-4e1f-854c-d4470a305428"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:56:17.478030 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:17.477991    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83e4b486-4385-4e1f-854c-d4470a305428-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:56:17.478125 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:17.478081    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83e4b486-4385-4e1f-854c-d4470a305428-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "83e4b486-4385-4e1f-854c-d4470a305428" (UID: "83e4b486-4385-4e1f-854c-d4470a305428"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 11:56:17.478401 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:17.478375    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83e4b486-4385-4e1f-854c-d4470a305428-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "83e4b486-4385-4e1f-854c-d4470a305428" (UID: "83e4b486-4385-4e1f-854c-d4470a305428"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:56:17.480214 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:17.480187    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83e4b486-4385-4e1f-854c-d4470a305428-kube-api-access-xb2zt" (OuterVolumeSpecName: "kube-api-access-xb2zt") pod "83e4b486-4385-4e1f-854c-d4470a305428" (UID: "83e4b486-4385-4e1f-854c-d4470a305428"). InnerVolumeSpecName "kube-api-access-xb2zt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 11:56:17.480583 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:17.480561    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83e4b486-4385-4e1f-854c-d4470a305428-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "83e4b486-4385-4e1f-854c-d4470a305428" (UID: "83e4b486-4385-4e1f-854c-d4470a305428"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:56:17.482547 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:17.482524    2581 scope.go:117] "RemoveContainer" containerID="56110ea244a091c53c4ec5ec556cd64fbf84bf1477b134eee990d04498f5d0ad"
Jul 07 11:56:17.497628 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:17.497609    2581 scope.go:117] "RemoveContainer" containerID="927774dcbffa40af1d281bf7451c7dc9ed3826a4fd0585d4bd1bdc2559d8f1c2"
Jul 07 11:56:17.497890 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:56:17.497868    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"927774dcbffa40af1d281bf7451c7dc9ed3826a4fd0585d4bd1bdc2559d8f1c2\": container with ID starting with 927774dcbffa40af1d281bf7451c7dc9ed3826a4fd0585d4bd1bdc2559d8f1c2 not found: ID does not exist" containerID="927774dcbffa40af1d281bf7451c7dc9ed3826a4fd0585d4bd1bdc2559d8f1c2"
Jul 07 11:56:17.497961 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:17.497899    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"927774dcbffa40af1d281bf7451c7dc9ed3826a4fd0585d4bd1bdc2559d8f1c2"} err="failed to get container status \"927774dcbffa40af1d281bf7451c7dc9ed3826a4fd0585d4bd1bdc2559d8f1c2\": rpc error: code = NotFound desc = could not find container \"927774dcbffa40af1d281bf7451c7dc9ed3826a4fd0585d4bd1bdc2559d8f1c2\": container with ID starting with 927774dcbffa40af1d281bf7451c7dc9ed3826a4fd0585d4bd1bdc2559d8f1c2 not found: ID does not exist"
Jul 07 11:56:17.497961 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:17.497917    2581 scope.go:117] "RemoveContainer" containerID="d97323a9a8dd25e133646dd259a14775ce687351e2d2e09c63cebc87040a835a"
Jul 07 11:56:17.498171 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:56:17.498148    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d97323a9a8dd25e133646dd259a14775ce687351e2d2e09c63cebc87040a835a\": container with ID starting with d97323a9a8dd25e133646dd259a14775ce687351e2d2e09c63cebc87040a835a not found: ID does not exist" containerID="d97323a9a8dd25e133646dd259a14775ce687351e2d2e09c63cebc87040a835a"
Jul 07 11:56:17.498286 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:17.498175    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d97323a9a8dd25e133646dd259a14775ce687351e2d2e09c63cebc87040a835a"} err="failed to get container status \"d97323a9a8dd25e133646dd259a14775ce687351e2d2e09c63cebc87040a835a\": rpc error: code = NotFound desc = could not find container \"d97323a9a8dd25e133646dd259a14775ce687351e2d2e09c63cebc87040a835a\": container with ID starting with d97323a9a8dd25e133646dd259a14775ce687351e2d2e09c63cebc87040a835a not found: ID does not exist"
Jul 07 11:56:17.498286 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:17.498198    2581 scope.go:117] "RemoveContainer" containerID="51389dc5ccd0161d7a56e6c731d9e1c6798987a841683f397b2e48c204eaae17"
Jul 07 11:56:17.498468 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:56:17.498442    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51389dc5ccd0161d7a56e6c731d9e1c6798987a841683f397b2e48c204eaae17\": container with ID starting with 51389dc5ccd0161d7a56e6c731d9e1c6798987a841683f397b2e48c204eaae17 not found: ID does not exist" containerID="51389dc5ccd0161d7a56e6c731d9e1c6798987a841683f397b2e48c204eaae17"
Jul 07 11:56:17.498537 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:17.498476    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51389dc5ccd0161d7a56e6c731d9e1c6798987a841683f397b2e48c204eaae17"} err="failed to get container status \"51389dc5ccd0161d7a56e6c731d9e1c6798987a841683f397b2e48c204eaae17\": rpc error: code = NotFound desc = could not find container \"51389dc5ccd0161d7a56e6c731d9e1c6798987a841683f397b2e48c204eaae17\": container with ID starting with 51389dc5ccd0161d7a56e6c731d9e1c6798987a841683f397b2e48c204eaae17 not found: ID does not exist"
Jul 07 11:56:17.498537 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:17.498498    2581 scope.go:117] "RemoveContainer" containerID="56110ea244a091c53c4ec5ec556cd64fbf84bf1477b134eee990d04498f5d0ad"
Jul 07 11:56:17.498772 ip-10-0-138-183 kubenswrapper[2581]: E0707 11:56:17.498739    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56110ea244a091c53c4ec5ec556cd64fbf84bf1477b134eee990d04498f5d0ad\": container with ID starting with 56110ea244a091c53c4ec5ec556cd64fbf84bf1477b134eee990d04498f5d0ad not found: ID does not exist" containerID="56110ea244a091c53c4ec5ec556cd64fbf84bf1477b134eee990d04498f5d0ad"
Jul 07 11:56:17.498847 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:17.498776    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56110ea244a091c53c4ec5ec556cd64fbf84bf1477b134eee990d04498f5d0ad"} err="failed to get container status \"56110ea244a091c53c4ec5ec556cd64fbf84bf1477b134eee990d04498f5d0ad\": rpc error: code = NotFound desc = could not find container \"56110ea244a091c53c4ec5ec556cd64fbf84bf1477b134eee990d04498f5d0ad\": container with ID starting with 56110ea244a091c53c4ec5ec556cd64fbf84bf1477b134eee990d04498f5d0ad not found: ID does not exist"
Jul 07 11:56:17.578913 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:17.578886    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83e4b486-4385-4e1f-854c-d4470a305428-kserve-provision-location\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:56:17.578913 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:17.578909    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb2zt\" (UniqueName: \"kubernetes.io/projected/83e4b486-4385-4e1f-854c-d4470a305428-kube-api-access-xb2zt\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:56:17.578913 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:17.578919    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83e4b486-4385-4e1f-854c-d4470a305428-cabundle-cert\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:56:17.579120 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:17.578927    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83e4b486-4385-4e1f-854c-d4470a305428-proxy-tls\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:56:17.781105 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:17.781076    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc"]
Jul 07 11:56:17.784445 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:17.784400    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-sflbc"]
Jul 07 11:56:18.173637 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:56:18.173605    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83e4b486-4385-4e1f-854c-d4470a305428" path="/var/lib/kubelet/pods/83e4b486-4385-4e1f-854c-d4470a305428/volumes"
Jul 07 11:58:13.012132 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:13.012094    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk" event={"ID":"2a1155fd-96fe-41b1-813d-df25f6ee3a40","Type":"ContainerStarted","Data":"261230b42c5694e6669fc47abf44216efc93dd2da7a3e50d77ad12ff0a6c4f5f"}
Jul 07 11:58:13.012574 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:13.012166    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk" event={"ID":"2a1155fd-96fe-41b1-813d-df25f6ee3a40","Type":"ContainerStarted","Data":"ee85513404a0a34b0c3fde6a862e25e9c0f254565440b0e7388555b8826ea993"}
Jul 07 11:58:13.012574 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:13.012329    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk"
Jul 07 11:58:13.041489 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:13.041446    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk" podStartSLOduration=2.562356222 podStartE2EDuration="2m26.041433721s" totalImagesPullingTime="2m2.941449013s" totalInitContainerRuntime="20.537628486s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:55:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:58:13.039612758 +0000 UTC m=+1083.386826383" watchObservedRunningTime="2026-07-07 11:58:13.041433721 +0000 UTC m=+1083.388647344"
Jul 07 11:58:14.016972 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:14.016935    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk"
Jul 07 11:58:14.018086 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:14.018058    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk" podUID="2a1155fd-96fe-41b1-813d-df25f6ee3a40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 07 11:58:15.020631 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:15.020592    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk" podUID="2a1155fd-96fe-41b1-813d-df25f6ee3a40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 07 11:58:20.024966 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:20.024927    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk"
Jul 07 11:58:20.025620 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:20.025588    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk" podUID="2a1155fd-96fe-41b1-813d-df25f6ee3a40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 07 11:58:30.026812 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:30.026777    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk"
Jul 07 11:58:37.329173 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:37.329141    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk"]
Jul 07 11:58:37.329561 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:37.329382    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk" podUID="2a1155fd-96fe-41b1-813d-df25f6ee3a40" containerName="kserve-container" containerID="cri-o://ee85513404a0a34b0c3fde6a862e25e9c0f254565440b0e7388555b8826ea993" gracePeriod=30
Jul 07 11:58:37.329561 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:37.329421    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk" podUID="2a1155fd-96fe-41b1-813d-df25f6ee3a40" containerName="kube-rbac-proxy" containerID="cri-o://261230b42c5694e6669fc47abf44216efc93dd2da7a3e50d77ad12ff0a6c4f5f" gracePeriod=30
Jul 07 11:58:38.123037 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:38.122987    2581 generic.go:356] "Generic (PLEG): container finished" podID="2a1155fd-96fe-41b1-813d-df25f6ee3a40" containerID="261230b42c5694e6669fc47abf44216efc93dd2da7a3e50d77ad12ff0a6c4f5f" exitCode=2
Jul 07 11:58:38.123037 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:38.123029    2581 generic.go:356] "Generic (PLEG): container finished" podID="2a1155fd-96fe-41b1-813d-df25f6ee3a40" containerID="ee85513404a0a34b0c3fde6a862e25e9c0f254565440b0e7388555b8826ea993" exitCode=0
Jul 07 11:58:38.123265 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:38.123054    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk" event={"ID":"2a1155fd-96fe-41b1-813d-df25f6ee3a40","Type":"ContainerDied","Data":"261230b42c5694e6669fc47abf44216efc93dd2da7a3e50d77ad12ff0a6c4f5f"}
Jul 07 11:58:38.123265 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:38.123128    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk" event={"ID":"2a1155fd-96fe-41b1-813d-df25f6ee3a40","Type":"ContainerDied","Data":"ee85513404a0a34b0c3fde6a862e25e9c0f254565440b0e7388555b8826ea993"}
Jul 07 11:58:38.198623 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:38.198600    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk"
Jul 07 11:58:38.283735 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:38.283670    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a1155fd-96fe-41b1-813d-df25f6ee3a40-kube-rbac-proxy-sar-config\") pod \"2a1155fd-96fe-41b1-813d-df25f6ee3a40\" (UID: \"2a1155fd-96fe-41b1-813d-df25f6ee3a40\") "
Jul 07 11:58:38.283735 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:38.283709    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a1155fd-96fe-41b1-813d-df25f6ee3a40-cabundle-cert\") pod \"2a1155fd-96fe-41b1-813d-df25f6ee3a40\" (UID: \"2a1155fd-96fe-41b1-813d-df25f6ee3a40\") "
Jul 07 11:58:38.283735 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:38.283731    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92m68\" (UniqueName: \"kubernetes.io/projected/2a1155fd-96fe-41b1-813d-df25f6ee3a40-kube-api-access-92m68\") pod \"2a1155fd-96fe-41b1-813d-df25f6ee3a40\" (UID: \"2a1155fd-96fe-41b1-813d-df25f6ee3a40\") "
Jul 07 11:58:38.283947 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:38.283772    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a1155fd-96fe-41b1-813d-df25f6ee3a40-kserve-provision-location\") pod \"2a1155fd-96fe-41b1-813d-df25f6ee3a40\" (UID: \"2a1155fd-96fe-41b1-813d-df25f6ee3a40\") "
Jul 07 11:58:38.283947 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:38.283850    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a1155fd-96fe-41b1-813d-df25f6ee3a40-proxy-tls\") pod \"2a1155fd-96fe-41b1-813d-df25f6ee3a40\" (UID: \"2a1155fd-96fe-41b1-813d-df25f6ee3a40\") "
Jul 07 11:58:38.284137 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:38.284109    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a1155fd-96fe-41b1-813d-df25f6ee3a40-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2a1155fd-96fe-41b1-813d-df25f6ee3a40" (UID: "2a1155fd-96fe-41b1-813d-df25f6ee3a40"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:58:38.284335 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:38.284299    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a1155fd-96fe-41b1-813d-df25f6ee3a40-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2a1155fd-96fe-41b1-813d-df25f6ee3a40" (UID: "2a1155fd-96fe-41b1-813d-df25f6ee3a40"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:58:38.285917 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:38.285892    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1155fd-96fe-41b1-813d-df25f6ee3a40-kube-api-access-92m68" (OuterVolumeSpecName: "kube-api-access-92m68") pod "2a1155fd-96fe-41b1-813d-df25f6ee3a40" (UID: "2a1155fd-96fe-41b1-813d-df25f6ee3a40"). InnerVolumeSpecName "kube-api-access-92m68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 11:58:38.285988 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:38.285936    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a1155fd-96fe-41b1-813d-df25f6ee3a40-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2a1155fd-96fe-41b1-813d-df25f6ee3a40" (UID: "2a1155fd-96fe-41b1-813d-df25f6ee3a40"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:58:38.289220 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:38.289198    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1155fd-96fe-41b1-813d-df25f6ee3a40-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2a1155fd-96fe-41b1-813d-df25f6ee3a40" (UID: "2a1155fd-96fe-41b1-813d-df25f6ee3a40"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 11:58:38.385135 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:38.385109    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a1155fd-96fe-41b1-813d-df25f6ee3a40-proxy-tls\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:58:38.385135 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:38.385132    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a1155fd-96fe-41b1-813d-df25f6ee3a40-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:58:38.385490 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:38.385142    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a1155fd-96fe-41b1-813d-df25f6ee3a40-cabundle-cert\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:58:38.385490 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:38.385152    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92m68\" (UniqueName: \"kubernetes.io/projected/2a1155fd-96fe-41b1-813d-df25f6ee3a40-kube-api-access-92m68\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:58:38.385490 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:38.385161    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a1155fd-96fe-41b1-813d-df25f6ee3a40-kserve-provision-location\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 11:58:39.129370 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:39.129334    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk" event={"ID":"2a1155fd-96fe-41b1-813d-df25f6ee3a40","Type":"ContainerDied","Data":"05c30af1b9b05b746a8dc5889f1ae4b065c4adb1f273b99af999e5327b183703"}
Jul 07 11:58:39.129539 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:39.129377    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk"
Jul 07 11:58:39.129539 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:39.129411    2581 scope.go:117] "RemoveContainer" containerID="261230b42c5694e6669fc47abf44216efc93dd2da7a3e50d77ad12ff0a6c4f5f"
Jul 07 11:58:39.138516 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:39.138497    2581 scope.go:117] "RemoveContainer" containerID="ee85513404a0a34b0c3fde6a862e25e9c0f254565440b0e7388555b8826ea993"
Jul 07 11:58:39.145679 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:39.145663    2581 scope.go:117] "RemoveContainer" containerID="192734f2be42dbadb4eda481bcffcb93efaeafc297ab3e34ba353515c4a03ea8"
Jul 07 11:58:39.151966 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:39.151946    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk"]
Jul 07 11:58:39.156741 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:39.156722    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-g4kxk"]
Jul 07 11:58:40.172225 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:40.172193    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a1155fd-96fe-41b1-813d-df25f6ee3a40" path="/var/lib/kubelet/pods/2a1155fd-96fe-41b1-813d-df25f6ee3a40/volumes"
Jul 07 11:58:42.784185 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.784155    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b"]
Jul 07 11:58:42.784595 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.784547    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="kube-rbac-proxy"
Jul 07 11:58:42.784595 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.784561    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="kube-rbac-proxy"
Jul 07 11:58:42.784595 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.784571    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b929818-519c-47b4-a528-911d0e368001" containerName="kserve-container"
Jul 07 11:58:42.784595 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.784577    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b929818-519c-47b4-a528-911d0e368001" containerName="kserve-container"
Jul 07 11:58:42.784595 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.784585    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a1155fd-96fe-41b1-813d-df25f6ee3a40" containerName="kserve-container"
Jul 07 11:58:42.784595 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.784590    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a1155fd-96fe-41b1-813d-df25f6ee3a40" containerName="kserve-container"
Jul 07 11:58:42.784811 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.784601    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a1155fd-96fe-41b1-813d-df25f6ee3a40" containerName="kube-rbac-proxy"
Jul 07 11:58:42.784811 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.784607    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a1155fd-96fe-41b1-813d-df25f6ee3a40" containerName="kube-rbac-proxy"
Jul 07 11:58:42.784811 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.784615    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b929818-519c-47b4-a528-911d0e368001" containerName="kube-rbac-proxy"
Jul 07 11:58:42.784811 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.784621    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b929818-519c-47b4-a528-911d0e368001" containerName="kube-rbac-proxy"
Jul 07 11:58:42.784811 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.784628    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="kserve-container"
Jul 07 11:58:42.784811 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.784633    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="kserve-container"
Jul 07 11:58:42.784811 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.784643    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="agent"
Jul 07 11:58:42.784811 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.784648    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="agent"
Jul 07 11:58:42.784811 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.784654    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a1155fd-96fe-41b1-813d-df25f6ee3a40" containerName="storage-initializer"
Jul 07 11:58:42.784811 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.784660    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a1155fd-96fe-41b1-813d-df25f6ee3a40" containerName="storage-initializer"
Jul 07 11:58:42.784811 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.784668    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="storage-initializer"
Jul 07 11:58:42.784811 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.784673    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="storage-initializer"
Jul 07 11:58:42.784811 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.784723    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a1155fd-96fe-41b1-813d-df25f6ee3a40" containerName="kserve-container"
Jul 07 11:58:42.784811 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.784731    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="kserve-container"
Jul 07 11:58:42.784811 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.784739    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="kube-rbac-proxy"
Jul 07 11:58:42.784811 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.784744    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b929818-519c-47b4-a528-911d0e368001" containerName="kserve-container"
Jul 07 11:58:42.784811 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.784749    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="83e4b486-4385-4e1f-854c-d4470a305428" containerName="agent"
Jul 07 11:58:42.784811 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.784756    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a1155fd-96fe-41b1-813d-df25f6ee3a40" containerName="kube-rbac-proxy"
Jul 07 11:58:42.784811 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.784761    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b929818-519c-47b4-a528-911d0e368001" containerName="kube-rbac-proxy"
Jul 07 11:58:42.806029 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.805987    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b"]
Jul 07 11:58:42.806267 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.806134    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b"
Jul 07 11:58:42.811179 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.811157    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 07 11:58:42.811302 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.811165    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 07 11:58:42.811374 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.811314    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 11:58:42.811374 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.811185    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 11:58:42.811484 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.811206    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 11:58:42.811484 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.811208    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-p2bwt\""
Jul 07 11:58:42.811728 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.811712    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 11:58:42.924341 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.924308    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58a77935-f061-423b-ab76-caf3a6c0d6d9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-ndm4b\" (UID: \"58a77935-f061-423b-ab76-caf3a6c0d6d9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b"
Jul 07 11:58:42.924341 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.924346    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58a77935-f061-423b-ab76-caf3a6c0d6d9-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-ndm4b\" (UID: \"58a77935-f061-423b-ab76-caf3a6c0d6d9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b"
Jul 07 11:58:42.924540 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.924438    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58a77935-f061-423b-ab76-caf3a6c0d6d9-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-ndm4b\" (UID: \"58a77935-f061-423b-ab76-caf3a6c0d6d9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b"
Jul 07 11:58:42.924540 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.924480    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvj7w\" (UniqueName: \"kubernetes.io/projected/58a77935-f061-423b-ab76-caf3a6c0d6d9-kube-api-access-vvj7w\") pod \"isvc-autogluon-v2-predictor-67b849d456-ndm4b\" (UID: \"58a77935-f061-423b-ab76-caf3a6c0d6d9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b"
Jul 07 11:58:42.924540 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:42.924506    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58a77935-f061-423b-ab76-caf3a6c0d6d9-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-ndm4b\" (UID: \"58a77935-f061-423b-ab76-caf3a6c0d6d9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b"
Jul 07 11:58:43.024885 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:43.024861    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58a77935-f061-423b-ab76-caf3a6c0d6d9-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-ndm4b\" (UID: \"58a77935-f061-423b-ab76-caf3a6c0d6d9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b"
Jul 07 11:58:43.025037 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:43.024907    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58a77935-f061-423b-ab76-caf3a6c0d6d9-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-ndm4b\" (UID: \"58a77935-f061-423b-ab76-caf3a6c0d6d9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b"
Jul 07 11:58:43.025091 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:43.025042    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvj7w\" (UniqueName: \"kubernetes.io/projected/58a77935-f061-423b-ab76-caf3a6c0d6d9-kube-api-access-vvj7w\") pod \"isvc-autogluon-v2-predictor-67b849d456-ndm4b\" (UID: \"58a77935-f061-423b-ab76-caf3a6c0d6d9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b"
Jul 07 11:58:43.025091 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:43.025073    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58a77935-f061-423b-ab76-caf3a6c0d6d9-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-ndm4b\" (UID: \"58a77935-f061-423b-ab76-caf3a6c0d6d9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b"
Jul 07 11:58:43.025158 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:43.025148    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58a77935-f061-423b-ab76-caf3a6c0d6d9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-ndm4b\" (UID: \"58a77935-f061-423b-ab76-caf3a6c0d6d9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b"
Jul 07 11:58:43.025262 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:43.025239    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58a77935-f061-423b-ab76-caf3a6c0d6d9-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-ndm4b\" (UID: \"58a77935-f061-423b-ab76-caf3a6c0d6d9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b"
Jul 07 11:58:43.025656 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:43.025634    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58a77935-f061-423b-ab76-caf3a6c0d6d9-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-ndm4b\" (UID: \"58a77935-f061-423b-ab76-caf3a6c0d6d9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b"
Jul 07 11:58:43.025747 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:43.025732    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58a77935-f061-423b-ab76-caf3a6c0d6d9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-ndm4b\" (UID: \"58a77935-f061-423b-ab76-caf3a6c0d6d9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b"
Jul 07 11:58:43.027337 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:43.027321    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58a77935-f061-423b-ab76-caf3a6c0d6d9-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-ndm4b\" (UID: \"58a77935-f061-423b-ab76-caf3a6c0d6d9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b"
Jul 07 11:58:43.034658 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:43.034604    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvj7w\" (UniqueName: \"kubernetes.io/projected/58a77935-f061-423b-ab76-caf3a6c0d6d9-kube-api-access-vvj7w\") pod \"isvc-autogluon-v2-predictor-67b849d456-ndm4b\" (UID: \"58a77935-f061-423b-ab76-caf3a6c0d6d9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b"
Jul 07 11:58:43.117465 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:43.117438    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b"
Jul 07 11:58:43.259223 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:43.259192    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b"]
Jul 07 11:58:43.263204 ip-10-0-138-183 kubenswrapper[2581]: W0707 11:58:43.263168    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a77935_f061_423b_ab76_caf3a6c0d6d9.slice/crio-b07ed38a3cc71cac566a2c35037b4aaa9161ac9c0821f85c5b14ae44a07b50d3 WatchSource:0}: Error finding container b07ed38a3cc71cac566a2c35037b4aaa9161ac9c0821f85c5b14ae44a07b50d3: Status 404 returned error can't find the container with id b07ed38a3cc71cac566a2c35037b4aaa9161ac9c0821f85c5b14ae44a07b50d3
Jul 07 11:58:44.154632 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:44.154598    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b" event={"ID":"58a77935-f061-423b-ab76-caf3a6c0d6d9","Type":"ContainerStarted","Data":"e1e9e2bb9494d11933ae566d17c05b1475f4b4b761aada0d50b0fd5a72b77a9e"}
Jul 07 11:58:44.155027 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:58:44.154654    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b" event={"ID":"58a77935-f061-423b-ab76-caf3a6c0d6d9","Type":"ContainerStarted","Data":"b07ed38a3cc71cac566a2c35037b4aaa9161ac9c0821f85c5b14ae44a07b50d3"}
Jul 07 11:59:04.242880 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:59:04.242795    2581 generic.go:356] "Generic (PLEG): container finished" podID="58a77935-f061-423b-ab76-caf3a6c0d6d9" containerID="e1e9e2bb9494d11933ae566d17c05b1475f4b4b761aada0d50b0fd5a72b77a9e" exitCode=0
Jul 07 11:59:04.242880 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:59:04.242867    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b" event={"ID":"58a77935-f061-423b-ab76-caf3a6c0d6d9","Type":"ContainerDied","Data":"e1e9e2bb9494d11933ae566d17c05b1475f4b4b761aada0d50b0fd5a72b77a9e"}
Jul 07 11:59:05.250067 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:59:05.250033    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b" event={"ID":"58a77935-f061-423b-ab76-caf3a6c0d6d9","Type":"ContainerStarted","Data":"e07bf6f0dd8c028194f48fed9bff718d228808b0c8a29ec09555c25ecec9d3f6"}
Jul 07 11:59:05.250546 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:59:05.250104    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b" event={"ID":"58a77935-f061-423b-ab76-caf3a6c0d6d9","Type":"ContainerStarted","Data":"9f00b6cb1d7d74f35651f5a6b4e5ba374b56f2bc5d3b1e6780a9c7f3a6ee6b52"}
Jul 07 11:59:05.250546 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:59:05.250326    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b"
Jul 07 11:59:05.273169 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:59:05.273121    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b" podStartSLOduration=2.642388248 podStartE2EDuration="23.273106634s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.630718386s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:59:05.271106792 +0000 UTC m=+1135.618320439" watchObservedRunningTime="2026-07-07 11:59:05.273106634 +0000 UTC m=+1135.620320258"
Jul 07 11:59:06.254542 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:59:06.254509    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b"
Jul 07 11:59:12.265721 ip-10-0-138-183 kubenswrapper[2581]: I0707 11:59:12.265697    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b"
Jul 07 12:00:12.103366 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:12.103338    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kf48j_4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801/ovn-acl-logging/0.log"
Jul 07 12:00:12.104353 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:12.104328    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kf48j_4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801/ovn-acl-logging/0.log"
Jul 07 12:00:35.171179 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:35.171147    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b"
Jul 07 12:00:42.788754 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:42.788718    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b"]
Jul 07 12:00:42.789286 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:42.788921    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b" podUID="58a77935-f061-423b-ab76-caf3a6c0d6d9" containerName="kserve-container" containerID="cri-o://9f00b6cb1d7d74f35651f5a6b4e5ba374b56f2bc5d3b1e6780a9c7f3a6ee6b52" gracePeriod=30
Jul 07 12:00:42.789286 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:42.788956    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b" podUID="58a77935-f061-423b-ab76-caf3a6c0d6d9" containerName="kube-rbac-proxy" containerID="cri-o://e07bf6f0dd8c028194f48fed9bff718d228808b0c8a29ec09555c25ecec9d3f6" gracePeriod=30
Jul 07 12:00:43.623092 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.623070    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b"
Jul 07 12:00:43.672452 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.672388    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58a77935-f061-423b-ab76-caf3a6c0d6d9-kube-rbac-proxy-sar-config\") pod \"58a77935-f061-423b-ab76-caf3a6c0d6d9\" (UID: \"58a77935-f061-423b-ab76-caf3a6c0d6d9\") "
Jul 07 12:00:43.672452 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.672425    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58a77935-f061-423b-ab76-caf3a6c0d6d9-cabundle-cert\") pod \"58a77935-f061-423b-ab76-caf3a6c0d6d9\" (UID: \"58a77935-f061-423b-ab76-caf3a6c0d6d9\") "
Jul 07 12:00:43.672611 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.672457    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvj7w\" (UniqueName: \"kubernetes.io/projected/58a77935-f061-423b-ab76-caf3a6c0d6d9-kube-api-access-vvj7w\") pod \"58a77935-f061-423b-ab76-caf3a6c0d6d9\" (UID: \"58a77935-f061-423b-ab76-caf3a6c0d6d9\") "
Jul 07 12:00:43.672611 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.672509    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58a77935-f061-423b-ab76-caf3a6c0d6d9-kserve-provision-location\") pod \"58a77935-f061-423b-ab76-caf3a6c0d6d9\" (UID: \"58a77935-f061-423b-ab76-caf3a6c0d6d9\") "
Jul 07 12:00:43.672723 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.672613    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58a77935-f061-423b-ab76-caf3a6c0d6d9-proxy-tls\") pod \"58a77935-f061-423b-ab76-caf3a6c0d6d9\" (UID: \"58a77935-f061-423b-ab76-caf3a6c0d6d9\") "
Jul 07 12:00:43.672853 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.672828    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58a77935-f061-423b-ab76-caf3a6c0d6d9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "58a77935-f061-423b-ab76-caf3a6c0d6d9" (UID: "58a77935-f061-423b-ab76-caf3a6c0d6d9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:00:43.672950 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.672915    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58a77935-f061-423b-ab76-caf3a6c0d6d9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "58a77935-f061-423b-ab76-caf3a6c0d6d9" (UID: "58a77935-f061-423b-ab76-caf3a6c0d6d9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:00:43.674645 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.674618    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58a77935-f061-423b-ab76-caf3a6c0d6d9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "58a77935-f061-423b-ab76-caf3a6c0d6d9" (UID: "58a77935-f061-423b-ab76-caf3a6c0d6d9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 12:00:43.674853 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.674823    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a77935-f061-423b-ab76-caf3a6c0d6d9-kube-api-access-vvj7w" (OuterVolumeSpecName: "kube-api-access-vvj7w") pod "58a77935-f061-423b-ab76-caf3a6c0d6d9" (UID: "58a77935-f061-423b-ab76-caf3a6c0d6d9"). InnerVolumeSpecName "kube-api-access-vvj7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 12:00:43.678242 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.678214    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a77935-f061-423b-ab76-caf3a6c0d6d9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "58a77935-f061-423b-ab76-caf3a6c0d6d9" (UID: "58a77935-f061-423b-ab76-caf3a6c0d6d9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 12:00:43.679370 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.679152    2581 generic.go:356] "Generic (PLEG): container finished" podID="58a77935-f061-423b-ab76-caf3a6c0d6d9" containerID="e07bf6f0dd8c028194f48fed9bff718d228808b0c8a29ec09555c25ecec9d3f6" exitCode=2
Jul 07 12:00:43.679370 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.679178    2581 generic.go:356] "Generic (PLEG): container finished" podID="58a77935-f061-423b-ab76-caf3a6c0d6d9" containerID="9f00b6cb1d7d74f35651f5a6b4e5ba374b56f2bc5d3b1e6780a9c7f3a6ee6b52" exitCode=0
Jul 07 12:00:43.679370 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.679199    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b"
Jul 07 12:00:43.679370 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.679228    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b" event={"ID":"58a77935-f061-423b-ab76-caf3a6c0d6d9","Type":"ContainerDied","Data":"e07bf6f0dd8c028194f48fed9bff718d228808b0c8a29ec09555c25ecec9d3f6"}
Jul 07 12:00:43.679370 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.679283    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b" event={"ID":"58a77935-f061-423b-ab76-caf3a6c0d6d9","Type":"ContainerDied","Data":"9f00b6cb1d7d74f35651f5a6b4e5ba374b56f2bc5d3b1e6780a9c7f3a6ee6b52"}
Jul 07 12:00:43.679370 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.679303    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b" event={"ID":"58a77935-f061-423b-ab76-caf3a6c0d6d9","Type":"ContainerDied","Data":"b07ed38a3cc71cac566a2c35037b4aaa9161ac9c0821f85c5b14ae44a07b50d3"}
Jul 07 12:00:43.679370 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.679331    2581 scope.go:117] "RemoveContainer" containerID="e07bf6f0dd8c028194f48fed9bff718d228808b0c8a29ec09555c25ecec9d3f6"
Jul 07 12:00:43.688226 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.688208    2581 scope.go:117] "RemoveContainer" containerID="9f00b6cb1d7d74f35651f5a6b4e5ba374b56f2bc5d3b1e6780a9c7f3a6ee6b52"
Jul 07 12:00:43.695385 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.695371    2581 scope.go:117] "RemoveContainer" containerID="e1e9e2bb9494d11933ae566d17c05b1475f4b4b761aada0d50b0fd5a72b77a9e"
Jul 07 12:00:43.702830 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.702816    2581 scope.go:117] "RemoveContainer" containerID="e07bf6f0dd8c028194f48fed9bff718d228808b0c8a29ec09555c25ecec9d3f6"
Jul 07 12:00:43.702974 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.702955    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b"]
Jul 07 12:00:43.703072 ip-10-0-138-183 kubenswrapper[2581]: E0707 12:00:43.703057    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e07bf6f0dd8c028194f48fed9bff718d228808b0c8a29ec09555c25ecec9d3f6\": container with ID starting with e07bf6f0dd8c028194f48fed9bff718d228808b0c8a29ec09555c25ecec9d3f6 not found: ID does not exist" containerID="e07bf6f0dd8c028194f48fed9bff718d228808b0c8a29ec09555c25ecec9d3f6"
Jul 07 12:00:43.703123 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.703079    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e07bf6f0dd8c028194f48fed9bff718d228808b0c8a29ec09555c25ecec9d3f6"} err="failed to get container status \"e07bf6f0dd8c028194f48fed9bff718d228808b0c8a29ec09555c25ecec9d3f6\": rpc error: code = NotFound desc = could not find container \"e07bf6f0dd8c028194f48fed9bff718d228808b0c8a29ec09555c25ecec9d3f6\": container with ID starting with e07bf6f0dd8c028194f48fed9bff718d228808b0c8a29ec09555c25ecec9d3f6 not found: ID does not exist"
Jul 07 12:00:43.703123 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.703094    2581 scope.go:117] "RemoveContainer" containerID="9f00b6cb1d7d74f35651f5a6b4e5ba374b56f2bc5d3b1e6780a9c7f3a6ee6b52"
Jul 07 12:00:43.703318 ip-10-0-138-183 kubenswrapper[2581]: E0707 12:00:43.703304    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f00b6cb1d7d74f35651f5a6b4e5ba374b56f2bc5d3b1e6780a9c7f3a6ee6b52\": container with ID starting with 9f00b6cb1d7d74f35651f5a6b4e5ba374b56f2bc5d3b1e6780a9c7f3a6ee6b52 not found: ID does not exist" containerID="9f00b6cb1d7d74f35651f5a6b4e5ba374b56f2bc5d3b1e6780a9c7f3a6ee6b52"
Jul 07 12:00:43.703358 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.703323    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f00b6cb1d7d74f35651f5a6b4e5ba374b56f2bc5d3b1e6780a9c7f3a6ee6b52"} err="failed to get container status \"9f00b6cb1d7d74f35651f5a6b4e5ba374b56f2bc5d3b1e6780a9c7f3a6ee6b52\": rpc error: code = NotFound desc = could not find container \"9f00b6cb1d7d74f35651f5a6b4e5ba374b56f2bc5d3b1e6780a9c7f3a6ee6b52\": container with ID starting with 9f00b6cb1d7d74f35651f5a6b4e5ba374b56f2bc5d3b1e6780a9c7f3a6ee6b52 not found: ID does not exist"
Jul 07 12:00:43.703358 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.703337    2581 scope.go:117] "RemoveContainer" containerID="e1e9e2bb9494d11933ae566d17c05b1475f4b4b761aada0d50b0fd5a72b77a9e"
Jul 07 12:00:43.703516 ip-10-0-138-183 kubenswrapper[2581]: E0707 12:00:43.703500    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1e9e2bb9494d11933ae566d17c05b1475f4b4b761aada0d50b0fd5a72b77a9e\": container with ID starting with e1e9e2bb9494d11933ae566d17c05b1475f4b4b761aada0d50b0fd5a72b77a9e not found: ID does not exist" containerID="e1e9e2bb9494d11933ae566d17c05b1475f4b4b761aada0d50b0fd5a72b77a9e"
Jul 07 12:00:43.703557 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.703517    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1e9e2bb9494d11933ae566d17c05b1475f4b4b761aada0d50b0fd5a72b77a9e"} err="failed to get container status \"e1e9e2bb9494d11933ae566d17c05b1475f4b4b761aada0d50b0fd5a72b77a9e\": rpc error: code = NotFound desc = could not find container \"e1e9e2bb9494d11933ae566d17c05b1475f4b4b761aada0d50b0fd5a72b77a9e\": container with ID starting with e1e9e2bb9494d11933ae566d17c05b1475f4b4b761aada0d50b0fd5a72b77a9e not found: ID does not exist"
Jul 07 12:00:43.703557 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.703528    2581 scope.go:117] "RemoveContainer" containerID="e07bf6f0dd8c028194f48fed9bff718d228808b0c8a29ec09555c25ecec9d3f6"
Jul 07 12:00:43.703697 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.703681    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e07bf6f0dd8c028194f48fed9bff718d228808b0c8a29ec09555c25ecec9d3f6"} err="failed to get container status \"e07bf6f0dd8c028194f48fed9bff718d228808b0c8a29ec09555c25ecec9d3f6\": rpc error: code = NotFound desc = could not find container \"e07bf6f0dd8c028194f48fed9bff718d228808b0c8a29ec09555c25ecec9d3f6\": container with ID starting with e07bf6f0dd8c028194f48fed9bff718d228808b0c8a29ec09555c25ecec9d3f6 not found: ID does not exist"
Jul 07 12:00:43.703736 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.703697    2581 scope.go:117] "RemoveContainer" containerID="9f00b6cb1d7d74f35651f5a6b4e5ba374b56f2bc5d3b1e6780a9c7f3a6ee6b52"
Jul 07 12:00:43.703861 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.703847    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f00b6cb1d7d74f35651f5a6b4e5ba374b56f2bc5d3b1e6780a9c7f3a6ee6b52"} err="failed to get container status \"9f00b6cb1d7d74f35651f5a6b4e5ba374b56f2bc5d3b1e6780a9c7f3a6ee6b52\": rpc error: code = NotFound desc = could not find container \"9f00b6cb1d7d74f35651f5a6b4e5ba374b56f2bc5d3b1e6780a9c7f3a6ee6b52\": container with ID starting with 9f00b6cb1d7d74f35651f5a6b4e5ba374b56f2bc5d3b1e6780a9c7f3a6ee6b52 not found: ID does not exist"
Jul 07 12:00:43.703904 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.703862    2581 scope.go:117] "RemoveContainer" containerID="e1e9e2bb9494d11933ae566d17c05b1475f4b4b761aada0d50b0fd5a72b77a9e"
Jul 07 12:00:43.704140 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.704093    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1e9e2bb9494d11933ae566d17c05b1475f4b4b761aada0d50b0fd5a72b77a9e"} err="failed to get container status \"e1e9e2bb9494d11933ae566d17c05b1475f4b4b761aada0d50b0fd5a72b77a9e\": rpc error: code = NotFound desc = could not find container \"e1e9e2bb9494d11933ae566d17c05b1475f4b4b761aada0d50b0fd5a72b77a9e\": container with ID starting with e1e9e2bb9494d11933ae566d17c05b1475f4b4b761aada0d50b0fd5a72b77a9e not found: ID does not exist"
Jul 07 12:00:43.706607 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.706587    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ndm4b"]
Jul 07 12:00:43.774181 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.774160    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58a77935-f061-423b-ab76-caf3a6c0d6d9-proxy-tls\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 12:00:43.774273 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.774180    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58a77935-f061-423b-ab76-caf3a6c0d6d9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 12:00:43.774273 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.774201    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58a77935-f061-423b-ab76-caf3a6c0d6d9-cabundle-cert\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 12:00:43.774273 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.774210    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvj7w\" (UniqueName: \"kubernetes.io/projected/58a77935-f061-423b-ab76-caf3a6c0d6d9-kube-api-access-vvj7w\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 12:00:43.774273 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:43.774218    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58a77935-f061-423b-ab76-caf3a6c0d6d9-kserve-provision-location\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 12:00:44.171862 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:44.171828    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58a77935-f061-423b-ab76-caf3a6c0d6d9" path="/var/lib/kubelet/pods/58a77935-f061-423b-ab76-caf3a6c0d6d9/volumes"
Jul 07 12:00:47.949375 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:47.949343    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc"]
Jul 07 12:00:47.950801 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:47.949660    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58a77935-f061-423b-ab76-caf3a6c0d6d9" containerName="kserve-container"
Jul 07 12:00:47.950801 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:47.949670    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="58a77935-f061-423b-ab76-caf3a6c0d6d9" containerName="kserve-container"
Jul 07 12:00:47.950801 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:47.949682    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58a77935-f061-423b-ab76-caf3a6c0d6d9" containerName="storage-initializer"
Jul 07 12:00:47.950801 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:47.949688    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="58a77935-f061-423b-ab76-caf3a6c0d6d9" containerName="storage-initializer"
Jul 07 12:00:47.950801 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:47.949695    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58a77935-f061-423b-ab76-caf3a6c0d6d9" containerName="kube-rbac-proxy"
Jul 07 12:00:47.950801 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:47.949701    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="58a77935-f061-423b-ab76-caf3a6c0d6d9" containerName="kube-rbac-proxy"
Jul 07 12:00:47.950801 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:47.949750    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="58a77935-f061-423b-ab76-caf3a6c0d6d9" containerName="kserve-container"
Jul 07 12:00:47.950801 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:47.949760    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="58a77935-f061-423b-ab76-caf3a6c0d6d9" containerName="kube-rbac-proxy"
Jul 07 12:00:47.951751 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:47.951732    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc"
Jul 07 12:00:47.954713 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:47.954687    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 07 12:00:47.954955 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:47.954936    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 12:00:47.955026 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:47.954939    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 12:00:47.955026 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:47.954942    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 07 12:00:47.956293 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:47.956273    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 12:00:47.956424 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:47.956357    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 12:00:47.956424 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:47.956359    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-p2bwt\""
Jul 07 12:00:47.963634 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:47.963616    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc"]
Jul 07 12:00:48.006765 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:48.006742    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9128e597-a77d-474c-8571-968179b1e8f4-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc\" (UID: \"9128e597-a77d-474c-8571-968179b1e8f4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc"
Jul 07 12:00:48.006890 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:48.006776    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9128e597-a77d-474c-8571-968179b1e8f4-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc\" (UID: \"9128e597-a77d-474c-8571-968179b1e8f4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc"
Jul 07 12:00:48.006890 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:48.006820    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9128e597-a77d-474c-8571-968179b1e8f4-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc\" (UID: \"9128e597-a77d-474c-8571-968179b1e8f4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc"
Jul 07 12:00:48.006890 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:48.006860    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9128e597-a77d-474c-8571-968179b1e8f4-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc\" (UID: \"9128e597-a77d-474c-8571-968179b1e8f4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc"
Jul 07 12:00:48.006890 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:48.006880    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxrq7\" (UniqueName: \"kubernetes.io/projected/9128e597-a77d-474c-8571-968179b1e8f4-kube-api-access-zxrq7\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc\" (UID: \"9128e597-a77d-474c-8571-968179b1e8f4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc"
Jul 07 12:00:48.107339 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:48.107303    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9128e597-a77d-474c-8571-968179b1e8f4-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc\" (UID: \"9128e597-a77d-474c-8571-968179b1e8f4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc"
Jul 07 12:00:48.107492 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:48.107343    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9128e597-a77d-474c-8571-968179b1e8f4-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc\" (UID: \"9128e597-a77d-474c-8571-968179b1e8f4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc"
Jul 07 12:00:48.107492 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:48.107364    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9128e597-a77d-474c-8571-968179b1e8f4-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc\" (UID: \"9128e597-a77d-474c-8571-968179b1e8f4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc"
Jul 07 12:00:48.107492 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:48.107397    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9128e597-a77d-474c-8571-968179b1e8f4-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc\" (UID: \"9128e597-a77d-474c-8571-968179b1e8f4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc"
Jul 07 12:00:48.107492 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:48.107414    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxrq7\" (UniqueName: \"kubernetes.io/projected/9128e597-a77d-474c-8571-968179b1e8f4-kube-api-access-zxrq7\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc\" (UID: \"9128e597-a77d-474c-8571-968179b1e8f4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc"
Jul 07 12:00:48.107771 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:48.107747    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9128e597-a77d-474c-8571-968179b1e8f4-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc\" (UID: \"9128e597-a77d-474c-8571-968179b1e8f4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc"
Jul 07 12:00:48.108175 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:48.108146    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9128e597-a77d-474c-8571-968179b1e8f4-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc\" (UID: \"9128e597-a77d-474c-8571-968179b1e8f4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc"
Jul 07 12:00:48.108288 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:48.108146    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9128e597-a77d-474c-8571-968179b1e8f4-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc\" (UID: \"9128e597-a77d-474c-8571-968179b1e8f4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc"
Jul 07 12:00:48.109910 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:48.109890    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9128e597-a77d-474c-8571-968179b1e8f4-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc\" (UID: \"9128e597-a77d-474c-8571-968179b1e8f4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc"
Jul 07 12:00:48.116073 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:48.116053    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxrq7\" (UniqueName: \"kubernetes.io/projected/9128e597-a77d-474c-8571-968179b1e8f4-kube-api-access-zxrq7\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc\" (UID: \"9128e597-a77d-474c-8571-968179b1e8f4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc"
Jul 07 12:00:48.263116 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:48.263028    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc"
Jul 07 12:00:48.387761 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:48.387728    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc"]
Jul 07 12:00:48.391056 ip-10-0-138-183 kubenswrapper[2581]: W0707 12:00:48.391027    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9128e597_a77d_474c_8571_968179b1e8f4.slice/crio-f0c3d4c0364547c4f26075dd2e2e4d9b587f7cd739ecd090d6095e45e8b7b320 WatchSource:0}: Error finding container f0c3d4c0364547c4f26075dd2e2e4d9b587f7cd739ecd090d6095e45e8b7b320: Status 404 returned error can't find the container with id f0c3d4c0364547c4f26075dd2e2e4d9b587f7cd739ecd090d6095e45e8b7b320
Jul 07 12:00:48.703956 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:48.703915    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc" event={"ID":"9128e597-a77d-474c-8571-968179b1e8f4","Type":"ContainerStarted","Data":"8c7685d3a8834c03a03299ce8f3a9afdf9abdea56a26006e9e5fdab79f703d76"}
Jul 07 12:00:48.704141 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:00:48.703992    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc" event={"ID":"9128e597-a77d-474c-8571-968179b1e8f4","Type":"ContainerStarted","Data":"f0c3d4c0364547c4f26075dd2e2e4d9b587f7cd739ecd090d6095e45e8b7b320"}
Jul 07 12:01:18.839508 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:01:18.839471    2581 generic.go:356] "Generic (PLEG): container finished" podID="9128e597-a77d-474c-8571-968179b1e8f4" containerID="8c7685d3a8834c03a03299ce8f3a9afdf9abdea56a26006e9e5fdab79f703d76" exitCode=0
Jul 07 12:01:18.839921 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:01:18.839545    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc" event={"ID":"9128e597-a77d-474c-8571-968179b1e8f4","Type":"ContainerDied","Data":"8c7685d3a8834c03a03299ce8f3a9afdf9abdea56a26006e9e5fdab79f703d76"}
Jul 07 12:01:19.846134 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:01:19.846096    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc" event={"ID":"9128e597-a77d-474c-8571-968179b1e8f4","Type":"ContainerStarted","Data":"9c62fc2bf88bff4c87d76e938f4ec189c5c247ac85d4231d2cdb2f4ef1acf64e"}
Jul 07 12:01:19.846515 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:01:19.846163    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc" event={"ID":"9128e597-a77d-474c-8571-968179b1e8f4","Type":"ContainerStarted","Data":"5224f4a4e6f6024fd409923e02d351b58c008aa8f3fc11003ab066ffb5cd6f5e"}
Jul 07 12:01:19.846515 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:01:19.846364    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc"
Jul 07 12:01:19.869430 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:01:19.869379    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc" podStartSLOduration=3.030087743 podStartE2EDuration="32.869364623s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.83927688s" isStatefulPod=false podCreationTimestamp="2026-07-07 12:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 12:01:19.868149233 +0000 UTC m=+1270.215362857" watchObservedRunningTime="2026-07-07 12:01:19.869364623 +0000 UTC m=+1270.216578246"
Jul 07 12:01:20.850154 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:01:20.850125    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc"
Jul 07 12:01:26.861249 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:01:26.861222    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc"
Jul 07 12:02:53.170180 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:53.170149    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc"
Jul 07 12:02:58.092094 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:58.092063    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc"]
Jul 07 12:02:58.092516 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:58.092286    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc" podUID="9128e597-a77d-474c-8571-968179b1e8f4" containerName="kserve-container" containerID="cri-o://5224f4a4e6f6024fd409923e02d351b58c008aa8f3fc11003ab066ffb5cd6f5e" gracePeriod=30
Jul 07 12:02:58.092516 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:58.092337    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc" podUID="9128e597-a77d-474c-8571-968179b1e8f4" containerName="kube-rbac-proxy" containerID="cri-o://9c62fc2bf88bff4c87d76e938f4ec189c5c247ac85d4231d2cdb2f4ef1acf64e" gracePeriod=30
Jul 07 12:02:58.258994 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:58.258958    2581 generic.go:356] "Generic (PLEG): container finished" podID="9128e597-a77d-474c-8571-968179b1e8f4" containerID="9c62fc2bf88bff4c87d76e938f4ec189c5c247ac85d4231d2cdb2f4ef1acf64e" exitCode=2
Jul 07 12:02:58.259165 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:58.259039    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc" event={"ID":"9128e597-a77d-474c-8571-968179b1e8f4","Type":"ContainerDied","Data":"9c62fc2bf88bff4c87d76e938f4ec189c5c247ac85d4231d2cdb2f4ef1acf64e"}
Jul 07 12:02:58.835288 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:58.835266    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc"
Jul 07 12:02:58.927617 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:58.927588    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxrq7\" (UniqueName: \"kubernetes.io/projected/9128e597-a77d-474c-8571-968179b1e8f4-kube-api-access-zxrq7\") pod \"9128e597-a77d-474c-8571-968179b1e8f4\" (UID: \"9128e597-a77d-474c-8571-968179b1e8f4\") "
Jul 07 12:02:58.927774 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:58.927626    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9128e597-a77d-474c-8571-968179b1e8f4-kube-rbac-proxy-sar-config\") pod \"9128e597-a77d-474c-8571-968179b1e8f4\" (UID: \"9128e597-a77d-474c-8571-968179b1e8f4\") "
Jul 07 12:02:58.927774 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:58.927667    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9128e597-a77d-474c-8571-968179b1e8f4-cabundle-cert\") pod \"9128e597-a77d-474c-8571-968179b1e8f4\" (UID: \"9128e597-a77d-474c-8571-968179b1e8f4\") "
Jul 07 12:02:58.927774 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:58.927714    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9128e597-a77d-474c-8571-968179b1e8f4-proxy-tls\") pod \"9128e597-a77d-474c-8571-968179b1e8f4\" (UID: \"9128e597-a77d-474c-8571-968179b1e8f4\") "
Jul 07 12:02:58.927952 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:58.927862    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9128e597-a77d-474c-8571-968179b1e8f4-kserve-provision-location\") pod \"9128e597-a77d-474c-8571-968179b1e8f4\" (UID: \"9128e597-a77d-474c-8571-968179b1e8f4\") "
Jul 07 12:02:58.928129 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:58.928098    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9128e597-a77d-474c-8571-968179b1e8f4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9128e597-a77d-474c-8571-968179b1e8f4" (UID: "9128e597-a77d-474c-8571-968179b1e8f4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:02:58.928225 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:58.928147    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9128e597-a77d-474c-8571-968179b1e8f4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9128e597-a77d-474c-8571-968179b1e8f4" (UID: "9128e597-a77d-474c-8571-968179b1e8f4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:02:58.928225 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:58.928198    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9128e597-a77d-474c-8571-968179b1e8f4-cabundle-cert\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 12:02:58.929707 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:58.929679    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9128e597-a77d-474c-8571-968179b1e8f4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9128e597-a77d-474c-8571-968179b1e8f4" (UID: "9128e597-a77d-474c-8571-968179b1e8f4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 12:02:58.929707 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:58.929686    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9128e597-a77d-474c-8571-968179b1e8f4-kube-api-access-zxrq7" (OuterVolumeSpecName: "kube-api-access-zxrq7") pod "9128e597-a77d-474c-8571-968179b1e8f4" (UID: "9128e597-a77d-474c-8571-968179b1e8f4"). InnerVolumeSpecName "kube-api-access-zxrq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 12:02:58.932688 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:58.932666    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9128e597-a77d-474c-8571-968179b1e8f4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9128e597-a77d-474c-8571-968179b1e8f4" (UID: "9128e597-a77d-474c-8571-968179b1e8f4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 12:02:59.029365 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:59.029334    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9128e597-a77d-474c-8571-968179b1e8f4-kserve-provision-location\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 12:02:59.029365 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:59.029360    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxrq7\" (UniqueName: \"kubernetes.io/projected/9128e597-a77d-474c-8571-968179b1e8f4-kube-api-access-zxrq7\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 12:02:59.029523 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:59.029374    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9128e597-a77d-474c-8571-968179b1e8f4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 12:02:59.029523 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:59.029387    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9128e597-a77d-474c-8571-968179b1e8f4-proxy-tls\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 12:02:59.265251 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:59.265177    2581 generic.go:356] "Generic (PLEG): container finished" podID="9128e597-a77d-474c-8571-968179b1e8f4" containerID="5224f4a4e6f6024fd409923e02d351b58c008aa8f3fc11003ab066ffb5cd6f5e" exitCode=0
Jul 07 12:02:59.265251 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:59.265236    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc"
Jul 07 12:02:59.265681 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:59.265260    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc" event={"ID":"9128e597-a77d-474c-8571-968179b1e8f4","Type":"ContainerDied","Data":"5224f4a4e6f6024fd409923e02d351b58c008aa8f3fc11003ab066ffb5cd6f5e"}
Jul 07 12:02:59.265681 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:59.265316    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc" event={"ID":"9128e597-a77d-474c-8571-968179b1e8f4","Type":"ContainerDied","Data":"f0c3d4c0364547c4f26075dd2e2e4d9b587f7cd739ecd090d6095e45e8b7b320"}
Jul 07 12:02:59.265681 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:59.265346    2581 scope.go:117] "RemoveContainer" containerID="9c62fc2bf88bff4c87d76e938f4ec189c5c247ac85d4231d2cdb2f4ef1acf64e"
Jul 07 12:02:59.274704 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:59.274688    2581 scope.go:117] "RemoveContainer" containerID="5224f4a4e6f6024fd409923e02d351b58c008aa8f3fc11003ab066ffb5cd6f5e"
Jul 07 12:02:59.281555 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:59.281541    2581 scope.go:117] "RemoveContainer" containerID="8c7685d3a8834c03a03299ce8f3a9afdf9abdea56a26006e9e5fdab79f703d76"
Jul 07 12:02:59.288180 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:59.288160    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc"]
Jul 07 12:02:59.289184 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:59.289169    2581 scope.go:117] "RemoveContainer" containerID="9c62fc2bf88bff4c87d76e938f4ec189c5c247ac85d4231d2cdb2f4ef1acf64e"
Jul 07 12:02:59.289495 ip-10-0-138-183 kubenswrapper[2581]: E0707 12:02:59.289471    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c62fc2bf88bff4c87d76e938f4ec189c5c247ac85d4231d2cdb2f4ef1acf64e\": container with ID starting with 9c62fc2bf88bff4c87d76e938f4ec189c5c247ac85d4231d2cdb2f4ef1acf64e not found: ID does not exist" containerID="9c62fc2bf88bff4c87d76e938f4ec189c5c247ac85d4231d2cdb2f4ef1acf64e"
Jul 07 12:02:59.289578 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:59.289505    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c62fc2bf88bff4c87d76e938f4ec189c5c247ac85d4231d2cdb2f4ef1acf64e"} err="failed to get container status \"9c62fc2bf88bff4c87d76e938f4ec189c5c247ac85d4231d2cdb2f4ef1acf64e\": rpc error: code = NotFound desc = could not find container \"9c62fc2bf88bff4c87d76e938f4ec189c5c247ac85d4231d2cdb2f4ef1acf64e\": container with ID starting with 9c62fc2bf88bff4c87d76e938f4ec189c5c247ac85d4231d2cdb2f4ef1acf64e not found: ID does not exist"
Jul 07 12:02:59.289578 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:59.289529    2581 scope.go:117] "RemoveContainer" containerID="5224f4a4e6f6024fd409923e02d351b58c008aa8f3fc11003ab066ffb5cd6f5e"
Jul 07 12:02:59.289910 ip-10-0-138-183 kubenswrapper[2581]: E0707 12:02:59.289853    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5224f4a4e6f6024fd409923e02d351b58c008aa8f3fc11003ab066ffb5cd6f5e\": container with ID starting with 5224f4a4e6f6024fd409923e02d351b58c008aa8f3fc11003ab066ffb5cd6f5e not found: ID does not exist" containerID="5224f4a4e6f6024fd409923e02d351b58c008aa8f3fc11003ab066ffb5cd6f5e"
Jul 07 12:02:59.289910 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:59.289889    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5224f4a4e6f6024fd409923e02d351b58c008aa8f3fc11003ab066ffb5cd6f5e"} err="failed to get container status \"5224f4a4e6f6024fd409923e02d351b58c008aa8f3fc11003ab066ffb5cd6f5e\": rpc error: code = NotFound desc = could not find container \"5224f4a4e6f6024fd409923e02d351b58c008aa8f3fc11003ab066ffb5cd6f5e\": container with ID starting with 5224f4a4e6f6024fd409923e02d351b58c008aa8f3fc11003ab066ffb5cd6f5e not found: ID does not exist"
Jul 07 12:02:59.290133 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:59.289912    2581 scope.go:117] "RemoveContainer" containerID="8c7685d3a8834c03a03299ce8f3a9afdf9abdea56a26006e9e5fdab79f703d76"
Jul 07 12:02:59.290279 ip-10-0-138-183 kubenswrapper[2581]: E0707 12:02:59.290254    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c7685d3a8834c03a03299ce8f3a9afdf9abdea56a26006e9e5fdab79f703d76\": container with ID starting with 8c7685d3a8834c03a03299ce8f3a9afdf9abdea56a26006e9e5fdab79f703d76 not found: ID does not exist" containerID="8c7685d3a8834c03a03299ce8f3a9afdf9abdea56a26006e9e5fdab79f703d76"
Jul 07 12:02:59.290375 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:59.290289    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c7685d3a8834c03a03299ce8f3a9afdf9abdea56a26006e9e5fdab79f703d76"} err="failed to get container status \"8c7685d3a8834c03a03299ce8f3a9afdf9abdea56a26006e9e5fdab79f703d76\": rpc error: code = NotFound desc = could not find container \"8c7685d3a8834c03a03299ce8f3a9afdf9abdea56a26006e9e5fdab79f703d76\": container with ID starting with 8c7685d3a8834c03a03299ce8f3a9afdf9abdea56a26006e9e5fdab79f703d76 not found: ID does not exist"
Jul 07 12:02:59.292004 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:02:59.291984    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rt9rc"]
Jul 07 12:03:00.172232 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:00.172191    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9128e597-a77d-474c-8571-968179b1e8f4" path="/var/lib/kubelet/pods/9128e597-a77d-474c-8571-968179b1e8f4/volumes"
Jul 07 12:03:03.235263 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:03.235230    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5"]
Jul 07 12:03:03.235641 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:03.235567    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9128e597-a77d-474c-8571-968179b1e8f4" containerName="kube-rbac-proxy"
Jul 07 12:03:03.235641 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:03.235579    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9128e597-a77d-474c-8571-968179b1e8f4" containerName="kube-rbac-proxy"
Jul 07 12:03:03.235641 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:03.235590    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9128e597-a77d-474c-8571-968179b1e8f4" containerName="storage-initializer"
Jul 07 12:03:03.235641 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:03.235595    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9128e597-a77d-474c-8571-968179b1e8f4" containerName="storage-initializer"
Jul 07 12:03:03.235641 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:03.235607    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9128e597-a77d-474c-8571-968179b1e8f4" containerName="kserve-container"
Jul 07 12:03:03.235641 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:03.235613    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9128e597-a77d-474c-8571-968179b1e8f4" containerName="kserve-container"
Jul 07 12:03:03.235873 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:03.235667    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9128e597-a77d-474c-8571-968179b1e8f4" containerName="kube-rbac-proxy"
Jul 07 12:03:03.235873 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:03.235679    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9128e597-a77d-474c-8571-968179b1e8f4" containerName="kserve-container"
Jul 07 12:03:03.240246 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:03.240227    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5"
Jul 07 12:03:03.243156 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:03.243134    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 12:03:03.243254 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:03.243152    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-p2bwt\""
Jul 07 12:03:03.243254 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:03.243168    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 07 12:03:03.243353 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:03.243281    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 12:03:03.244210 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:03.244190    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 12:03:03.244286 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:03.244248    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 07 12:03:03.244348 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:03.244315    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 12:03:03.248423 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:03.248405    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5"]
Jul 07 12:03:03.363209 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:03.363173    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d5f8b8c-2aca-43f1-804a-07ea0b0203b5-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5\" (UID: \"4d5f8b8c-2aca-43f1-804a-07ea0b0203b5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5"
Jul 07 12:03:03.363209 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:03.363208    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d5f8b8c-2aca-43f1-804a-07ea0b0203b5-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5\" (UID: \"4d5f8b8c-2aca-43f1-804a-07ea0b0203b5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5"
Jul 07 12:03:03.363427 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:03.363251    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6xnz\" (UniqueName: \"kubernetes.io/projected/4d5f8b8c-2aca-43f1-804a-07ea0b0203b5-kube-api-access-x6xnz\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5\" (UID: \"4d5f8b8c-2aca-43f1-804a-07ea0b0203b5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5"
Jul 07 12:03:03.363427 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:03.363286    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d5f8b8c-2aca-43f1-804a-07ea0b0203b5-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5\" (UID: \"4d5f8b8c-2aca-43f1-804a-07ea0b0203b5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5"
Jul 07 12:03:03.363427 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:03.363305    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d5f8b8c-2aca-43f1-804a-07ea0b0203b5-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5\" (UID: \"4d5f8b8c-2aca-43f1-804a-07ea0b0203b5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5"
Jul 07 12:03:03.464252 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:03.464180    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d5f8b8c-2aca-43f1-804a-07ea0b0203b5-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5\" (UID: \"4d5f8b8c-2aca-43f1-804a-07ea0b0203b5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5"
Jul 07 12:03:03.464252 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:03.464210    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d5f8b8c-2aca-43f1-804a-07ea0b0203b5-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5\" (UID: \"4d5f8b8c-2aca-43f1-804a-07ea0b0203b5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5"
Jul 07 12:03:03.464456 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:03.464273    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d5f8b8c-2aca-43f1-804a-07ea0b0203b5-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5\" (UID: \"4d5f8b8c-2aca-43f1-804a-07ea0b0203b5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5"
Jul 07 12:03:03.464456 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:03.464289    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d5f8b8c-2aca-43f1-804a-07ea0b0203b5-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5\" (UID: \"4d5f8b8c-2aca-43f1-804a-07ea0b0203b5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5"
Jul 07 12:03:03.464456 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:03.464320    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6xnz\" (UniqueName: \"kubernetes.io/projected/4d5f8b8c-2aca-43f1-804a-07ea0b0203b5-kube-api-access-x6xnz\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5\" (UID: \"4d5f8b8c-2aca-43f1-804a-07ea0b0203b5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5"
Jul 07 12:03:03.464838 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:03.464720    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d5f8b8c-2aca-43f1-804a-07ea0b0203b5-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5\" (UID: \"4d5f8b8c-2aca-43f1-804a-07ea0b0203b5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5"
Jul 07 12:03:03.465021 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:03.464973    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d5f8b8c-2aca-43f1-804a-07ea0b0203b5-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5\" (UID: \"4d5f8b8c-2aca-43f1-804a-07ea0b0203b5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5"
Jul 07 12:03:03.465113 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:03.465067    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d5f8b8c-2aca-43f1-804a-07ea0b0203b5-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5\" (UID: \"4d5f8b8c-2aca-43f1-804a-07ea0b0203b5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5"
Jul 07 12:03:03.466975 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:03.466952    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d5f8b8c-2aca-43f1-804a-07ea0b0203b5-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5\" (UID: \"4d5f8b8c-2aca-43f1-804a-07ea0b0203b5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5"
Jul 07 12:03:03.472311 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:03.472279    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6xnz\" (UniqueName: \"kubernetes.io/projected/4d5f8b8c-2aca-43f1-804a-07ea0b0203b5-kube-api-access-x6xnz\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5\" (UID: \"4d5f8b8c-2aca-43f1-804a-07ea0b0203b5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5"
Jul 07 12:03:03.551282 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:03.551250    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5"
Jul 07 12:03:03.678223 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:03.678192    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5"]
Jul 07 12:03:03.679807 ip-10-0-138-183 kubenswrapper[2581]: W0707 12:03:03.679778    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d5f8b8c_2aca_43f1_804a_07ea0b0203b5.slice/crio-7c8b85a210178e72e76a12afc2757de57b6dc7fcf1e6a1622e11d85fc6bc2f7b WatchSource:0}: Error finding container 7c8b85a210178e72e76a12afc2757de57b6dc7fcf1e6a1622e11d85fc6bc2f7b: Status 404 returned error can't find the container with id 7c8b85a210178e72e76a12afc2757de57b6dc7fcf1e6a1622e11d85fc6bc2f7b
Jul 07 12:03:04.288467 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:04.288430    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5" event={"ID":"4d5f8b8c-2aca-43f1-804a-07ea0b0203b5","Type":"ContainerStarted","Data":"8907cd3d0bb20421afb92a19be9f01911ed8dbc48a2aa851bfa6ad9efcad87d3"}
Jul 07 12:03:04.288838 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:04.288486    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5" event={"ID":"4d5f8b8c-2aca-43f1-804a-07ea0b0203b5","Type":"ContainerStarted","Data":"7c8b85a210178e72e76a12afc2757de57b6dc7fcf1e6a1622e11d85fc6bc2f7b"}
Jul 07 12:03:34.412200 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:34.412166    2581 generic.go:356] "Generic (PLEG): container finished" podID="4d5f8b8c-2aca-43f1-804a-07ea0b0203b5" containerID="8907cd3d0bb20421afb92a19be9f01911ed8dbc48a2aa851bfa6ad9efcad87d3" exitCode=0
Jul 07 12:03:34.412637 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:34.412243    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5" event={"ID":"4d5f8b8c-2aca-43f1-804a-07ea0b0203b5","Type":"ContainerDied","Data":"8907cd3d0bb20421afb92a19be9f01911ed8dbc48a2aa851bfa6ad9efcad87d3"}
Jul 07 12:03:35.418146 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:35.418108    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5" event={"ID":"4d5f8b8c-2aca-43f1-804a-07ea0b0203b5","Type":"ContainerStarted","Data":"b46c152f6a74ac1e4443cc04b681293d032ad8b19764ee2528f0e67df5b49ef3"}
Jul 07 12:03:35.418589 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:35.418171    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5" event={"ID":"4d5f8b8c-2aca-43f1-804a-07ea0b0203b5","Type":"ContainerStarted","Data":"6be8d6009bb6e8faef831d07b03592eb1d5ce121722f755984c360e36f31292a"}
Jul 07 12:03:35.418589 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:35.418376    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5"
Jul 07 12:03:35.437643 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:35.437583    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5" podStartSLOduration=2.722762554 podStartE2EDuration="32.437570224s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.71480767s" isStatefulPod=false podCreationTimestamp="2026-07-07 12:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 12:03:35.436514393 +0000 UTC m=+1405.783728016" watchObservedRunningTime="2026-07-07 12:03:35.437570224 +0000 UTC m=+1405.784783847"
Jul 07 12:03:36.422670 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:36.422627    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5"
Jul 07 12:03:42.432590 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:03:42.432558    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5"
Jul 07 12:05:08.171065 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:08.171037    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5"
Jul 07 12:05:12.131633 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:12.131605    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kf48j_4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801/ovn-acl-logging/0.log"
Jul 07 12:05:12.133803 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:12.133782    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kf48j_4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801/ovn-acl-logging/0.log"
Jul 07 12:05:13.394175 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:13.394138    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5"]
Jul 07 12:05:13.394597 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:13.394435    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5" podUID="4d5f8b8c-2aca-43f1-804a-07ea0b0203b5" containerName="kserve-container" containerID="cri-o://6be8d6009bb6e8faef831d07b03592eb1d5ce121722f755984c360e36f31292a" gracePeriod=30
Jul 07 12:05:13.394597 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:13.394479    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5" podUID="4d5f8b8c-2aca-43f1-804a-07ea0b0203b5" containerName="kube-rbac-proxy" containerID="cri-o://b46c152f6a74ac1e4443cc04b681293d032ad8b19764ee2528f0e67df5b49ef3" gracePeriod=30
Jul 07 12:05:13.829445 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:13.829346    2581 generic.go:356] "Generic (PLEG): container finished" podID="4d5f8b8c-2aca-43f1-804a-07ea0b0203b5" containerID="b46c152f6a74ac1e4443cc04b681293d032ad8b19764ee2528f0e67df5b49ef3" exitCode=2
Jul 07 12:05:13.829445 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:13.829420    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5" event={"ID":"4d5f8b8c-2aca-43f1-804a-07ea0b0203b5","Type":"ContainerDied","Data":"b46c152f6a74ac1e4443cc04b681293d032ad8b19764ee2528f0e67df5b49ef3"}
Jul 07 12:05:14.142878 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:14.142855    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5"
Jul 07 12:05:14.195782 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:14.195752    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d5f8b8c-2aca-43f1-804a-07ea0b0203b5-kube-rbac-proxy-sar-config\") pod \"4d5f8b8c-2aca-43f1-804a-07ea0b0203b5\" (UID: \"4d5f8b8c-2aca-43f1-804a-07ea0b0203b5\") "
Jul 07 12:05:14.195940 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:14.195804    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6xnz\" (UniqueName: \"kubernetes.io/projected/4d5f8b8c-2aca-43f1-804a-07ea0b0203b5-kube-api-access-x6xnz\") pod \"4d5f8b8c-2aca-43f1-804a-07ea0b0203b5\" (UID: \"4d5f8b8c-2aca-43f1-804a-07ea0b0203b5\") "
Jul 07 12:05:14.195940 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:14.195831    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d5f8b8c-2aca-43f1-804a-07ea0b0203b5-cabundle-cert\") pod \"4d5f8b8c-2aca-43f1-804a-07ea0b0203b5\" (UID: \"4d5f8b8c-2aca-43f1-804a-07ea0b0203b5\") "
Jul 07 12:05:14.195940 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:14.195858    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d5f8b8c-2aca-43f1-804a-07ea0b0203b5-kserve-provision-location\") pod \"4d5f8b8c-2aca-43f1-804a-07ea0b0203b5\" (UID: \"4d5f8b8c-2aca-43f1-804a-07ea0b0203b5\") "
Jul 07 12:05:14.196212 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:14.195979    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d5f8b8c-2aca-43f1-804a-07ea0b0203b5-proxy-tls\") pod \"4d5f8b8c-2aca-43f1-804a-07ea0b0203b5\" (UID: \"4d5f8b8c-2aca-43f1-804a-07ea0b0203b5\") "
Jul 07 12:05:14.196212 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:14.196172    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d5f8b8c-2aca-43f1-804a-07ea0b0203b5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4d5f8b8c-2aca-43f1-804a-07ea0b0203b5" (UID: "4d5f8b8c-2aca-43f1-804a-07ea0b0203b5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:05:14.196212 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:14.196196    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d5f8b8c-2aca-43f1-804a-07ea0b0203b5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4d5f8b8c-2aca-43f1-804a-07ea0b0203b5" (UID: "4d5f8b8c-2aca-43f1-804a-07ea0b0203b5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:05:14.196377 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:14.196338    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d5f8b8c-2aca-43f1-804a-07ea0b0203b5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 12:05:14.196377 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:14.196353    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d5f8b8c-2aca-43f1-804a-07ea0b0203b5-cabundle-cert\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 12:05:14.197919 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:14.197893    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d5f8b8c-2aca-43f1-804a-07ea0b0203b5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4d5f8b8c-2aca-43f1-804a-07ea0b0203b5" (UID: "4d5f8b8c-2aca-43f1-804a-07ea0b0203b5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 12:05:14.198105 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:14.197961    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d5f8b8c-2aca-43f1-804a-07ea0b0203b5-kube-api-access-x6xnz" (OuterVolumeSpecName: "kube-api-access-x6xnz") pod "4d5f8b8c-2aca-43f1-804a-07ea0b0203b5" (UID: "4d5f8b8c-2aca-43f1-804a-07ea0b0203b5"). InnerVolumeSpecName "kube-api-access-x6xnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 12:05:14.202056 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:14.202032    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d5f8b8c-2aca-43f1-804a-07ea0b0203b5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4d5f8b8c-2aca-43f1-804a-07ea0b0203b5" (UID: "4d5f8b8c-2aca-43f1-804a-07ea0b0203b5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 12:05:14.297542 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:14.297517    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d5f8b8c-2aca-43f1-804a-07ea0b0203b5-kserve-provision-location\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 12:05:14.297542 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:14.297538    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d5f8b8c-2aca-43f1-804a-07ea0b0203b5-proxy-tls\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 12:05:14.297689 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:14.297549    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6xnz\" (UniqueName: \"kubernetes.io/projected/4d5f8b8c-2aca-43f1-804a-07ea0b0203b5-kube-api-access-x6xnz\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 12:05:14.836180 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:14.836143    2581 generic.go:356] "Generic (PLEG): container finished" podID="4d5f8b8c-2aca-43f1-804a-07ea0b0203b5" containerID="6be8d6009bb6e8faef831d07b03592eb1d5ce121722f755984c360e36f31292a" exitCode=0
Jul 07 12:05:14.836659 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:14.836207    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5"
Jul 07 12:05:14.836659 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:14.836209    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5" event={"ID":"4d5f8b8c-2aca-43f1-804a-07ea0b0203b5","Type":"ContainerDied","Data":"6be8d6009bb6e8faef831d07b03592eb1d5ce121722f755984c360e36f31292a"}
Jul 07 12:05:14.836659 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:14.836331    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5" event={"ID":"4d5f8b8c-2aca-43f1-804a-07ea0b0203b5","Type":"ContainerDied","Data":"7c8b85a210178e72e76a12afc2757de57b6dc7fcf1e6a1622e11d85fc6bc2f7b"}
Jul 07 12:05:14.836659 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:14.836358    2581 scope.go:117] "RemoveContainer" containerID="b46c152f6a74ac1e4443cc04b681293d032ad8b19764ee2528f0e67df5b49ef3"
Jul 07 12:05:14.844540 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:14.844523    2581 scope.go:117] "RemoveContainer" containerID="6be8d6009bb6e8faef831d07b03592eb1d5ce121722f755984c360e36f31292a"
Jul 07 12:05:14.851527 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:14.851512    2581 scope.go:117] "RemoveContainer" containerID="8907cd3d0bb20421afb92a19be9f01911ed8dbc48a2aa851bfa6ad9efcad87d3"
Jul 07 12:05:14.858427 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:14.858404    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5"]
Jul 07 12:05:14.858863 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:14.858843    2581 scope.go:117] "RemoveContainer" containerID="b46c152f6a74ac1e4443cc04b681293d032ad8b19764ee2528f0e67df5b49ef3"
Jul 07 12:05:14.859128 ip-10-0-138-183 kubenswrapper[2581]: E0707 12:05:14.859107    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b46c152f6a74ac1e4443cc04b681293d032ad8b19764ee2528f0e67df5b49ef3\": container with ID starting with b46c152f6a74ac1e4443cc04b681293d032ad8b19764ee2528f0e67df5b49ef3 not found: ID does not exist" containerID="b46c152f6a74ac1e4443cc04b681293d032ad8b19764ee2528f0e67df5b49ef3"
Jul 07 12:05:14.859233 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:14.859136    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b46c152f6a74ac1e4443cc04b681293d032ad8b19764ee2528f0e67df5b49ef3"} err="failed to get container status \"b46c152f6a74ac1e4443cc04b681293d032ad8b19764ee2528f0e67df5b49ef3\": rpc error: code = NotFound desc = could not find container \"b46c152f6a74ac1e4443cc04b681293d032ad8b19764ee2528f0e67df5b49ef3\": container with ID starting with b46c152f6a74ac1e4443cc04b681293d032ad8b19764ee2528f0e67df5b49ef3 not found: ID does not exist"
Jul 07 12:05:14.859233 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:14.859152    2581 scope.go:117] "RemoveContainer" containerID="6be8d6009bb6e8faef831d07b03592eb1d5ce121722f755984c360e36f31292a"
Jul 07 12:05:14.859398 ip-10-0-138-183 kubenswrapper[2581]: E0707 12:05:14.859380    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6be8d6009bb6e8faef831d07b03592eb1d5ce121722f755984c360e36f31292a\": container with ID starting with 6be8d6009bb6e8faef831d07b03592eb1d5ce121722f755984c360e36f31292a not found: ID does not exist" containerID="6be8d6009bb6e8faef831d07b03592eb1d5ce121722f755984c360e36f31292a"
Jul 07 12:05:14.859441 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:14.859402    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6be8d6009bb6e8faef831d07b03592eb1d5ce121722f755984c360e36f31292a"} err="failed to get container status \"6be8d6009bb6e8faef831d07b03592eb1d5ce121722f755984c360e36f31292a\": rpc error: code = NotFound desc = could not find container \"6be8d6009bb6e8faef831d07b03592eb1d5ce121722f755984c360e36f31292a\": container with ID starting with 6be8d6009bb6e8faef831d07b03592eb1d5ce121722f755984c360e36f31292a not found: ID does not exist"
Jul 07 12:05:14.859441 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:14.859416    2581 scope.go:117] "RemoveContainer" containerID="8907cd3d0bb20421afb92a19be9f01911ed8dbc48a2aa851bfa6ad9efcad87d3"
Jul 07 12:05:14.859619 ip-10-0-138-183 kubenswrapper[2581]: E0707 12:05:14.859605    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8907cd3d0bb20421afb92a19be9f01911ed8dbc48a2aa851bfa6ad9efcad87d3\": container with ID starting with 8907cd3d0bb20421afb92a19be9f01911ed8dbc48a2aa851bfa6ad9efcad87d3 not found: ID does not exist" containerID="8907cd3d0bb20421afb92a19be9f01911ed8dbc48a2aa851bfa6ad9efcad87d3"
Jul 07 12:05:14.859660 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:14.859623    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8907cd3d0bb20421afb92a19be9f01911ed8dbc48a2aa851bfa6ad9efcad87d3"} err="failed to get container status \"8907cd3d0bb20421afb92a19be9f01911ed8dbc48a2aa851bfa6ad9efcad87d3\": rpc error: code = NotFound desc = could not find container \"8907cd3d0bb20421afb92a19be9f01911ed8dbc48a2aa851bfa6ad9efcad87d3\": container with ID starting with 8907cd3d0bb20421afb92a19be9f01911ed8dbc48a2aa851bfa6ad9efcad87d3 not found: ID does not exist"
Jul 07 12:05:14.864799 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:14.864776    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-z8sh5"]
Jul 07 12:05:16.170802 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:16.170759    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d5f8b8c-2aca-43f1-804a-07ea0b0203b5" path="/var/lib/kubelet/pods/4d5f8b8c-2aca-43f1-804a-07ea0b0203b5/volumes"
Jul 07 12:05:18.544158 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:18.544082    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv"]
Jul 07 12:05:18.544489 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:18.544388    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d5f8b8c-2aca-43f1-804a-07ea0b0203b5" containerName="kube-rbac-proxy"
Jul 07 12:05:18.544489 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:18.544399    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d5f8b8c-2aca-43f1-804a-07ea0b0203b5" containerName="kube-rbac-proxy"
Jul 07 12:05:18.544489 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:18.544415    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d5f8b8c-2aca-43f1-804a-07ea0b0203b5" containerName="kserve-container"
Jul 07 12:05:18.544489 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:18.544420    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d5f8b8c-2aca-43f1-804a-07ea0b0203b5" containerName="kserve-container"
Jul 07 12:05:18.544489 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:18.544434    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d5f8b8c-2aca-43f1-804a-07ea0b0203b5" containerName="storage-initializer"
Jul 07 12:05:18.544489 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:18.544440    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d5f8b8c-2aca-43f1-804a-07ea0b0203b5" containerName="storage-initializer"
Jul 07 12:05:18.544489 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:18.544487    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d5f8b8c-2aca-43f1-804a-07ea0b0203b5" containerName="kserve-container"
Jul 07 12:05:18.544711 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:18.544499    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d5f8b8c-2aca-43f1-804a-07ea0b0203b5" containerName="kube-rbac-proxy"
Jul 07 12:05:18.549456 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:18.549434    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv"
Jul 07 12:05:18.552160 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:18.552141    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 07 12:05:18.552254 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:18.552211    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 12:05:18.552349 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:18.552331    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 12:05:18.552453 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:18.552415    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 12:05:18.553451 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:18.553433    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 07 12:05:18.553567 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:18.553463    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 12:05:18.553567 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:18.553556    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-p2bwt\""
Jul 07 12:05:18.559609 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:18.559586    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv"]
Jul 07 12:05:18.632336 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:18.632308    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e98c573-65dd-4572-b6f1-915191d4da3c-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv\" (UID: \"7e98c573-65dd-4572-b6f1-915191d4da3c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv"
Jul 07 12:05:18.632455 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:18.632371    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75dsp\" (UniqueName: \"kubernetes.io/projected/7e98c573-65dd-4572-b6f1-915191d4da3c-kube-api-access-75dsp\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv\" (UID: \"7e98c573-65dd-4572-b6f1-915191d4da3c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv"
Jul 07 12:05:18.632455 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:18.632435    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e98c573-65dd-4572-b6f1-915191d4da3c-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv\" (UID: \"7e98c573-65dd-4572-b6f1-915191d4da3c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv"
Jul 07 12:05:18.632547 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:18.632456    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e98c573-65dd-4572-b6f1-915191d4da3c-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv\" (UID: \"7e98c573-65dd-4572-b6f1-915191d4da3c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv"
Jul 07 12:05:18.632547 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:18.632481    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e98c573-65dd-4572-b6f1-915191d4da3c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv\" (UID: \"7e98c573-65dd-4572-b6f1-915191d4da3c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv"
Jul 07 12:05:18.733532 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:18.733509    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e98c573-65dd-4572-b6f1-915191d4da3c-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv\" (UID: \"7e98c573-65dd-4572-b6f1-915191d4da3c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv"
Jul 07 12:05:18.733645 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:18.733536    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e98c573-65dd-4572-b6f1-915191d4da3c-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv\" (UID: \"7e98c573-65dd-4572-b6f1-915191d4da3c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv"
Jul 07 12:05:18.733645 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:18.733559    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e98c573-65dd-4572-b6f1-915191d4da3c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv\" (UID: \"7e98c573-65dd-4572-b6f1-915191d4da3c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv"
Jul 07 12:05:18.733645 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:18.733583    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e98c573-65dd-4572-b6f1-915191d4da3c-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv\" (UID: \"7e98c573-65dd-4572-b6f1-915191d4da3c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv"
Jul 07 12:05:18.733645 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:18.733638    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75dsp\" (UniqueName: \"kubernetes.io/projected/7e98c573-65dd-4572-b6f1-915191d4da3c-kube-api-access-75dsp\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv\" (UID: \"7e98c573-65dd-4572-b6f1-915191d4da3c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv"
Jul 07 12:05:18.734088 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:18.734064    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e98c573-65dd-4572-b6f1-915191d4da3c-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv\" (UID: \"7e98c573-65dd-4572-b6f1-915191d4da3c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv"
Jul 07 12:05:18.734386 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:18.734366    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e98c573-65dd-4572-b6f1-915191d4da3c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv\" (UID: \"7e98c573-65dd-4572-b6f1-915191d4da3c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv"
Jul 07 12:05:18.734434 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:18.734366    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e98c573-65dd-4572-b6f1-915191d4da3c-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv\" (UID: \"7e98c573-65dd-4572-b6f1-915191d4da3c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv"
Jul 07 12:05:18.736130 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:18.736111    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e98c573-65dd-4572-b6f1-915191d4da3c-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv\" (UID: \"7e98c573-65dd-4572-b6f1-915191d4da3c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv"
Jul 07 12:05:18.741820 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:18.741792    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75dsp\" (UniqueName: \"kubernetes.io/projected/7e98c573-65dd-4572-b6f1-915191d4da3c-kube-api-access-75dsp\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv\" (UID: \"7e98c573-65dd-4572-b6f1-915191d4da3c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv"
Jul 07 12:05:18.861320 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:18.861296    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv"
Jul 07 12:05:18.987099 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:18.986983    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv"]
Jul 07 12:05:18.989330 ip-10-0-138-183 kubenswrapper[2581]: W0707 12:05:18.989302    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e98c573_65dd_4572_b6f1_915191d4da3c.slice/crio-6c639794bb6951728d0e7fb3fcf33339351336766f901e5368cb6581f4dde48e WatchSource:0}: Error finding container 6c639794bb6951728d0e7fb3fcf33339351336766f901e5368cb6581f4dde48e: Status 404 returned error can't find the container with id 6c639794bb6951728d0e7fb3fcf33339351336766f901e5368cb6581f4dde48e
Jul 07 12:05:19.859582 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:19.859550    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv" event={"ID":"7e98c573-65dd-4572-b6f1-915191d4da3c","Type":"ContainerStarted","Data":"c72b2623c08936402fbf3cb49b7c990f65b15a712f0015baadd36517a66b21fc"}
Jul 07 12:05:19.859959 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:19.859606    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv" event={"ID":"7e98c573-65dd-4572-b6f1-915191d4da3c","Type":"ContainerStarted","Data":"6c639794bb6951728d0e7fb3fcf33339351336766f901e5368cb6581f4dde48e"}
Jul 07 12:05:28.897790 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:28.897753    2581 generic.go:356] "Generic (PLEG): container finished" podID="7e98c573-65dd-4572-b6f1-915191d4da3c" containerID="c72b2623c08936402fbf3cb49b7c990f65b15a712f0015baadd36517a66b21fc" exitCode=0
Jul 07 12:05:28.898189 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:28.897814    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv" event={"ID":"7e98c573-65dd-4572-b6f1-915191d4da3c","Type":"ContainerDied","Data":"c72b2623c08936402fbf3cb49b7c990f65b15a712f0015baadd36517a66b21fc"}
Jul 07 12:05:29.904570 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:29.904530    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv" event={"ID":"7e98c573-65dd-4572-b6f1-915191d4da3c","Type":"ContainerStarted","Data":"a0b0c78283633cc8ab412398545b32f227036190cf7a542b3ea2548bfd7820e6"}
Jul 07 12:05:29.904987 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:29.904604    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv" event={"ID":"7e98c573-65dd-4572-b6f1-915191d4da3c","Type":"ContainerStarted","Data":"76aa0c6026b6241427bebeed30e2fc8b12997986e9d777d60ef09e660657c540"}
Jul 07 12:05:29.904987 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:29.904795    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv"
Jul 07 12:05:29.926261 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:29.926216    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv" podStartSLOduration=2.947966805 podStartE2EDuration="11.926196027s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.978229222s" isStatefulPod=false podCreationTimestamp="2026-07-07 12:05:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 12:05:29.925253079 +0000 UTC m=+1520.272466704" watchObservedRunningTime="2026-07-07 12:05:29.926196027 +0000 UTC m=+1520.273409650"
Jul 07 12:05:30.909951 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:30.909910    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv"
Jul 07 12:05:30.911306 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:30.911280    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv" podUID="7e98c573-65dd-4572-b6f1-915191d4da3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 07 12:05:31.915624 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:31.915517    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv" podUID="7e98c573-65dd-4572-b6f1-915191d4da3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 07 12:05:36.920456 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:36.920426    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv"
Jul 07 12:05:36.921070 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:36.921039    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv" podUID="7e98c573-65dd-4572-b6f1-915191d4da3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 07 12:05:46.922206 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:46.922171    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv"
Jul 07 12:05:48.599555 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:48.599518    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv"]
Jul 07 12:05:48.600870 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:48.599803    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv" podUID="7e98c573-65dd-4572-b6f1-915191d4da3c" containerName="kserve-container" containerID="cri-o://76aa0c6026b6241427bebeed30e2fc8b12997986e9d777d60ef09e660657c540" gracePeriod=30
Jul 07 12:05:48.600870 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:48.599850    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv" podUID="7e98c573-65dd-4572-b6f1-915191d4da3c" containerName="kube-rbac-proxy" containerID="cri-o://a0b0c78283633cc8ab412398545b32f227036190cf7a542b3ea2548bfd7820e6" gracePeriod=30
Jul 07 12:05:48.992301 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:48.992214    2581 generic.go:356] "Generic (PLEG): container finished" podID="7e98c573-65dd-4572-b6f1-915191d4da3c" containerID="a0b0c78283633cc8ab412398545b32f227036190cf7a542b3ea2548bfd7820e6" exitCode=2
Jul 07 12:05:48.992464 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:48.992294    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv" event={"ID":"7e98c573-65dd-4572-b6f1-915191d4da3c","Type":"ContainerDied","Data":"a0b0c78283633cc8ab412398545b32f227036190cf7a542b3ea2548bfd7820e6"}
Jul 07 12:05:49.448184 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:49.448160    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv"
Jul 07 12:05:49.489739 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:49.489712    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e98c573-65dd-4572-b6f1-915191d4da3c-kserve-provision-location\") pod \"7e98c573-65dd-4572-b6f1-915191d4da3c\" (UID: \"7e98c573-65dd-4572-b6f1-915191d4da3c\") "
Jul 07 12:05:49.489862 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:49.489753    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e98c573-65dd-4572-b6f1-915191d4da3c-cabundle-cert\") pod \"7e98c573-65dd-4572-b6f1-915191d4da3c\" (UID: \"7e98c573-65dd-4572-b6f1-915191d4da3c\") "
Jul 07 12:05:49.489862 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:49.489778    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e98c573-65dd-4572-b6f1-915191d4da3c-proxy-tls\") pod \"7e98c573-65dd-4572-b6f1-915191d4da3c\" (UID: \"7e98c573-65dd-4572-b6f1-915191d4da3c\") "
Jul 07 12:05:49.489988 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:49.489890    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75dsp\" (UniqueName: \"kubernetes.io/projected/7e98c573-65dd-4572-b6f1-915191d4da3c-kube-api-access-75dsp\") pod \"7e98c573-65dd-4572-b6f1-915191d4da3c\" (UID: \"7e98c573-65dd-4572-b6f1-915191d4da3c\") "
Jul 07 12:05:49.489988 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:49.489944    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e98c573-65dd-4572-b6f1-915191d4da3c-kube-rbac-proxy-sar-config\") pod \"7e98c573-65dd-4572-b6f1-915191d4da3c\" (UID: \"7e98c573-65dd-4572-b6f1-915191d4da3c\") "
Jul 07 12:05:49.490148 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:49.490121    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e98c573-65dd-4572-b6f1-915191d4da3c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7e98c573-65dd-4572-b6f1-915191d4da3c" (UID: "7e98c573-65dd-4572-b6f1-915191d4da3c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:05:49.490365 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:49.490334    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e98c573-65dd-4572-b6f1-915191d4da3c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7e98c573-65dd-4572-b6f1-915191d4da3c" (UID: "7e98c573-65dd-4572-b6f1-915191d4da3c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:05:49.490365 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:49.490353    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e98c573-65dd-4572-b6f1-915191d4da3c-cabundle-cert\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 12:05:49.490937 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:49.490909    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e98c573-65dd-4572-b6f1-915191d4da3c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7e98c573-65dd-4572-b6f1-915191d4da3c" (UID: "7e98c573-65dd-4572-b6f1-915191d4da3c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 12:05:49.492374 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:49.492353    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e98c573-65dd-4572-b6f1-915191d4da3c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7e98c573-65dd-4572-b6f1-915191d4da3c" (UID: "7e98c573-65dd-4572-b6f1-915191d4da3c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 12:05:49.492457 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:49.492437    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e98c573-65dd-4572-b6f1-915191d4da3c-kube-api-access-75dsp" (OuterVolumeSpecName: "kube-api-access-75dsp") pod "7e98c573-65dd-4572-b6f1-915191d4da3c" (UID: "7e98c573-65dd-4572-b6f1-915191d4da3c"). InnerVolumeSpecName "kube-api-access-75dsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 12:05:49.591636 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:49.591575    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e98c573-65dd-4572-b6f1-915191d4da3c-kserve-provision-location\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 12:05:49.591636 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:49.591597    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e98c573-65dd-4572-b6f1-915191d4da3c-proxy-tls\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 12:05:49.591636 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:49.591607    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75dsp\" (UniqueName: \"kubernetes.io/projected/7e98c573-65dd-4572-b6f1-915191d4da3c-kube-api-access-75dsp\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 12:05:49.591636 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:49.591616    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e98c573-65dd-4572-b6f1-915191d4da3c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 12:05:49.998362 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:49.998330    2581 generic.go:356] "Generic (PLEG): container finished" podID="7e98c573-65dd-4572-b6f1-915191d4da3c" containerID="76aa0c6026b6241427bebeed30e2fc8b12997986e9d777d60ef09e660657c540" exitCode=0
Jul 07 12:05:49.998804 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:49.998368    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv" event={"ID":"7e98c573-65dd-4572-b6f1-915191d4da3c","Type":"ContainerDied","Data":"76aa0c6026b6241427bebeed30e2fc8b12997986e9d777d60ef09e660657c540"}
Jul 07 12:05:49.998804 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:49.998381    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv"
Jul 07 12:05:49.998804 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:49.998467    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv" event={"ID":"7e98c573-65dd-4572-b6f1-915191d4da3c","Type":"ContainerDied","Data":"6c639794bb6951728d0e7fb3fcf33339351336766f901e5368cb6581f4dde48e"}
Jul 07 12:05:49.998804 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:49.998511    2581 scope.go:117] "RemoveContainer" containerID="a0b0c78283633cc8ab412398545b32f227036190cf7a542b3ea2548bfd7820e6"
Jul 07 12:05:50.007181 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:50.007165    2581 scope.go:117] "RemoveContainer" containerID="76aa0c6026b6241427bebeed30e2fc8b12997986e9d777d60ef09e660657c540"
Jul 07 12:05:50.014452 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:50.014435    2581 scope.go:117] "RemoveContainer" containerID="c72b2623c08936402fbf3cb49b7c990f65b15a712f0015baadd36517a66b21fc"
Jul 07 12:05:50.020127 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:50.020108    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv"]
Jul 07 12:05:50.021985 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:50.021966    2581 scope.go:117] "RemoveContainer" containerID="a0b0c78283633cc8ab412398545b32f227036190cf7a542b3ea2548bfd7820e6"
Jul 07 12:05:50.022358 ip-10-0-138-183 kubenswrapper[2581]: E0707 12:05:50.022333    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0b0c78283633cc8ab412398545b32f227036190cf7a542b3ea2548bfd7820e6\": container with ID starting with a0b0c78283633cc8ab412398545b32f227036190cf7a542b3ea2548bfd7820e6 not found: ID does not exist" containerID="a0b0c78283633cc8ab412398545b32f227036190cf7a542b3ea2548bfd7820e6"
Jul 07 12:05:50.022450 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:50.022363    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0b0c78283633cc8ab412398545b32f227036190cf7a542b3ea2548bfd7820e6"} err="failed to get container status \"a0b0c78283633cc8ab412398545b32f227036190cf7a542b3ea2548bfd7820e6\": rpc error: code = NotFound desc = could not find container \"a0b0c78283633cc8ab412398545b32f227036190cf7a542b3ea2548bfd7820e6\": container with ID starting with a0b0c78283633cc8ab412398545b32f227036190cf7a542b3ea2548bfd7820e6 not found: ID does not exist"
Jul 07 12:05:50.022450 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:50.022381    2581 scope.go:117] "RemoveContainer" containerID="76aa0c6026b6241427bebeed30e2fc8b12997986e9d777d60ef09e660657c540"
Jul 07 12:05:50.022678 ip-10-0-138-183 kubenswrapper[2581]: E0707 12:05:50.022648    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76aa0c6026b6241427bebeed30e2fc8b12997986e9d777d60ef09e660657c540\": container with ID starting with 76aa0c6026b6241427bebeed30e2fc8b12997986e9d777d60ef09e660657c540 not found: ID does not exist" containerID="76aa0c6026b6241427bebeed30e2fc8b12997986e9d777d60ef09e660657c540"
Jul 07 12:05:50.022776 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:50.022686    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76aa0c6026b6241427bebeed30e2fc8b12997986e9d777d60ef09e660657c540"} err="failed to get container status \"76aa0c6026b6241427bebeed30e2fc8b12997986e9d777d60ef09e660657c540\": rpc error: code = NotFound desc = could not find container \"76aa0c6026b6241427bebeed30e2fc8b12997986e9d777d60ef09e660657c540\": container with ID starting with 76aa0c6026b6241427bebeed30e2fc8b12997986e9d777d60ef09e660657c540 not found: ID does not exist"
Jul 07 12:05:50.022776 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:50.022708    2581 scope.go:117] "RemoveContainer" containerID="c72b2623c08936402fbf3cb49b7c990f65b15a712f0015baadd36517a66b21fc"
Jul 07 12:05:50.022958 ip-10-0-138-183 kubenswrapper[2581]: E0707 12:05:50.022943    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c72b2623c08936402fbf3cb49b7c990f65b15a712f0015baadd36517a66b21fc\": container with ID starting with c72b2623c08936402fbf3cb49b7c990f65b15a712f0015baadd36517a66b21fc not found: ID does not exist" containerID="c72b2623c08936402fbf3cb49b7c990f65b15a712f0015baadd36517a66b21fc"
Jul 07 12:05:50.023027 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:50.022960    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c72b2623c08936402fbf3cb49b7c990f65b15a712f0015baadd36517a66b21fc"} err="failed to get container status \"c72b2623c08936402fbf3cb49b7c990f65b15a712f0015baadd36517a66b21fc\": rpc error: code = NotFound desc = could not find container \"c72b2623c08936402fbf3cb49b7c990f65b15a712f0015baadd36517a66b21fc\": container with ID starting with c72b2623c08936402fbf3cb49b7c990f65b15a712f0015baadd36517a66b21fc not found: ID does not exist"
Jul 07 12:05:50.023990 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:50.023973    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-h4xwv"]
Jul 07 12:05:50.172167 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:50.172138    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e98c573-65dd-4572-b6f1-915191d4da3c" path="/var/lib/kubelet/pods/7e98c573-65dd-4572-b6f1-915191d4da3c/volumes"
Jul 07 12:05:54.900511 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:54.900476    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-78phf/must-gather-g9v9k"]
Jul 07 12:05:54.900883 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:54.900861    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e98c573-65dd-4572-b6f1-915191d4da3c" containerName="kube-rbac-proxy"
Jul 07 12:05:54.900883 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:54.900873    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e98c573-65dd-4572-b6f1-915191d4da3c" containerName="kube-rbac-proxy"
Jul 07 12:05:54.900954 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:54.900889    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e98c573-65dd-4572-b6f1-915191d4da3c" containerName="storage-initializer"
Jul 07 12:05:54.900954 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:54.900895    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e98c573-65dd-4572-b6f1-915191d4da3c" containerName="storage-initializer"
Jul 07 12:05:54.900954 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:54.900905    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e98c573-65dd-4572-b6f1-915191d4da3c" containerName="kserve-container"
Jul 07 12:05:54.900954 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:54.900911    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e98c573-65dd-4572-b6f1-915191d4da3c" containerName="kserve-container"
Jul 07 12:05:54.901099 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:54.900959    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e98c573-65dd-4572-b6f1-915191d4da3c" containerName="kserve-container"
Jul 07 12:05:54.901099 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:54.900970    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e98c573-65dd-4572-b6f1-915191d4da3c" containerName="kube-rbac-proxy"
Jul 07 12:05:54.902955 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:54.902939    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-78phf/must-gather-g9v9k"
Jul 07 12:05:54.905556 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:54.905532    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-78phf\"/\"openshift-service-ca.crt\""
Jul 07 12:05:54.905707 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:54.905672    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-78phf\"/\"default-dockercfg-pw9fm\""
Jul 07 12:05:54.906787 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:54.906772    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-78phf\"/\"kube-root-ca.crt\""
Jul 07 12:05:54.912160 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:54.912139    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-78phf/must-gather-g9v9k"]
Jul 07 12:05:54.933893 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:54.933873    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2abd573b-6867-434c-a3d5-b2da832d8b29-must-gather-output\") pod \"must-gather-g9v9k\" (UID: \"2abd573b-6867-434c-a3d5-b2da832d8b29\") " pod="openshift-must-gather-78phf/must-gather-g9v9k"
Jul 07 12:05:54.933996 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:54.933915    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqk2x\" (UniqueName: \"kubernetes.io/projected/2abd573b-6867-434c-a3d5-b2da832d8b29-kube-api-access-zqk2x\") pod \"must-gather-g9v9k\" (UID: \"2abd573b-6867-434c-a3d5-b2da832d8b29\") " pod="openshift-must-gather-78phf/must-gather-g9v9k"
Jul 07 12:05:55.034899 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:55.034878    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2abd573b-6867-434c-a3d5-b2da832d8b29-must-gather-output\") pod \"must-gather-g9v9k\" (UID: \"2abd573b-6867-434c-a3d5-b2da832d8b29\") " pod="openshift-must-gather-78phf/must-gather-g9v9k"
Jul 07 12:05:55.035037 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:55.034912    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqk2x\" (UniqueName: \"kubernetes.io/projected/2abd573b-6867-434c-a3d5-b2da832d8b29-kube-api-access-zqk2x\") pod \"must-gather-g9v9k\" (UID: \"2abd573b-6867-434c-a3d5-b2da832d8b29\") " pod="openshift-must-gather-78phf/must-gather-g9v9k"
Jul 07 12:05:55.035256 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:55.035239    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2abd573b-6867-434c-a3d5-b2da832d8b29-must-gather-output\") pod \"must-gather-g9v9k\" (UID: \"2abd573b-6867-434c-a3d5-b2da832d8b29\") " pod="openshift-must-gather-78phf/must-gather-g9v9k"
Jul 07 12:05:55.044985 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:55.044964    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqk2x\" (UniqueName: \"kubernetes.io/projected/2abd573b-6867-434c-a3d5-b2da832d8b29-kube-api-access-zqk2x\") pod \"must-gather-g9v9k\" (UID: \"2abd573b-6867-434c-a3d5-b2da832d8b29\") " pod="openshift-must-gather-78phf/must-gather-g9v9k"
Jul 07 12:05:55.229781 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:55.229722    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-78phf/must-gather-g9v9k"
Jul 07 12:05:55.357435 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:55.357374    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-78phf/must-gather-g9v9k"]
Jul 07 12:05:55.359733 ip-10-0-138-183 kubenswrapper[2581]: W0707 12:05:55.359702    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2abd573b_6867_434c_a3d5_b2da832d8b29.slice/crio-52229d5df7a66249acfa4b49c1c4a779f22830847470c78a7d5d0e8e72611977 WatchSource:0}: Error finding container 52229d5df7a66249acfa4b49c1c4a779f22830847470c78a7d5d0e8e72611977: Status 404 returned error can't find the container with id 52229d5df7a66249acfa4b49c1c4a779f22830847470c78a7d5d0e8e72611977
Jul 07 12:05:55.361600 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:55.361580    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 12:05:56.029217 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:05:56.029151    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-78phf/must-gather-g9v9k" event={"ID":"2abd573b-6867-434c-a3d5-b2da832d8b29","Type":"ContainerStarted","Data":"52229d5df7a66249acfa4b49c1c4a779f22830847470c78a7d5d0e8e72611977"}
Jul 07 12:06:02.065143 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:02.065110    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-78phf/must-gather-g9v9k" event={"ID":"2abd573b-6867-434c-a3d5-b2da832d8b29","Type":"ContainerStarted","Data":"0a6f7b3660b21df2348b765b90eedff4271d63c29ed5cc4d990faeb663166d55"}
Jul 07 12:06:02.065513 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:02.065150    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-78phf/must-gather-g9v9k" event={"ID":"2abd573b-6867-434c-a3d5-b2da832d8b29","Type":"ContainerStarted","Data":"d0306e5f987816501f5a340cdda1fff928fcab8014d5a9e5ce805fc3f8e18083"}
Jul 07 12:06:02.082259 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:02.082194    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-78phf/must-gather-g9v9k" podStartSLOduration=2.167973551 podStartE2EDuration="8.082178363s" totalImagesPullingTime="5.914204812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 12:05:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 12:06:02.081719975 +0000 UTC m=+1552.428933599" watchObservedRunningTime="2026-07-07 12:06:02.082178363 +0000 UTC m=+1552.429391986"
Jul 07 12:06:21.154918 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:21.154883    2581 generic.go:356] "Generic (PLEG): container finished" podID="2abd573b-6867-434c-a3d5-b2da832d8b29" containerID="d0306e5f987816501f5a340cdda1fff928fcab8014d5a9e5ce805fc3f8e18083" exitCode=0
Jul 07 12:06:21.155339 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:21.154955    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-78phf/must-gather-g9v9k" event={"ID":"2abd573b-6867-434c-a3d5-b2da832d8b29","Type":"ContainerDied","Data":"d0306e5f987816501f5a340cdda1fff928fcab8014d5a9e5ce805fc3f8e18083"}
Jul 07 12:06:21.155339 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:21.155252    2581 scope.go:117] "RemoveContainer" containerID="d0306e5f987816501f5a340cdda1fff928fcab8014d5a9e5ce805fc3f8e18083"
Jul 07 12:06:21.645344 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:21.645314    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-78phf_must-gather-g9v9k_2abd573b-6867-434c-a3d5-b2da832d8b29/gather/0.log"
Jul 07 12:06:24.867528 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:24.867498    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-v289k_140115cd-5270-4373-b7eb-3db479036e81/global-pull-secret-syncer/0.log"
Jul 07 12:06:24.945718 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:24.945692    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8c7wn_f1745e00-65cf-4d21-9fd1-4ff342a9bf55/konnectivity-agent/0.log"
Jul 07 12:06:25.062081 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:25.062059    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-183.ec2.internal_f35e0bd4cbf1174e802ed0ff417ef039/haproxy/0.log"
Jul 07 12:06:27.071721 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:27.071691    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-78phf/must-gather-g9v9k"]
Jul 07 12:06:27.072104 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:27.071903    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-78phf/must-gather-g9v9k" podUID="2abd573b-6867-434c-a3d5-b2da832d8b29" containerName="copy" containerID="cri-o://0a6f7b3660b21df2348b765b90eedff4271d63c29ed5cc4d990faeb663166d55" gracePeriod=2
Jul 07 12:06:27.074762 ip-10-0-138-183 kubenswrapper[2581]: E0707 12:06:27.074727    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-g9v9k\" is forbidden: User \"system:node:ip-10-0-138-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-78phf\": no relationship found between node 'ip-10-0-138-183.ec2.internal' and this object" podUID="2abd573b-6867-434c-a3d5-b2da832d8b29" pod="openshift-must-gather-78phf/must-gather-g9v9k"
Jul 07 12:06:27.076205 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:27.075515    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-78phf/must-gather-g9v9k"]
Jul 07 12:06:27.306207 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:27.306185    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-78phf_must-gather-g9v9k_2abd573b-6867-434c-a3d5-b2da832d8b29/copy/0.log"
Jul 07 12:06:27.307615 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:27.307595    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-78phf/must-gather-g9v9k"
Jul 07 12:06:27.309937 ip-10-0-138-183 kubenswrapper[2581]: E0707 12:06:27.309913    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-g9v9k\" is forbidden: User \"system:node:ip-10-0-138-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-78phf\": no relationship found between node 'ip-10-0-138-183.ec2.internal' and this object" podUID="2abd573b-6867-434c-a3d5-b2da832d8b29" pod="openshift-must-gather-78phf/must-gather-g9v9k"
Jul 07 12:06:27.327308 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:27.327257    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2abd573b-6867-434c-a3d5-b2da832d8b29-must-gather-output\") pod \"2abd573b-6867-434c-a3d5-b2da832d8b29\" (UID: \"2abd573b-6867-434c-a3d5-b2da832d8b29\") "
Jul 07 12:06:27.327390 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:27.327373    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqk2x\" (UniqueName: \"kubernetes.io/projected/2abd573b-6867-434c-a3d5-b2da832d8b29-kube-api-access-zqk2x\") pod \"2abd573b-6867-434c-a3d5-b2da832d8b29\" (UID: \"2abd573b-6867-434c-a3d5-b2da832d8b29\") "
Jul 07 12:06:27.328622 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:27.328597    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abd573b-6867-434c-a3d5-b2da832d8b29-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "2abd573b-6867-434c-a3d5-b2da832d8b29" (UID: "2abd573b-6867-434c-a3d5-b2da832d8b29"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 12:06:27.329400 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:27.329379    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2abd573b-6867-434c-a3d5-b2da832d8b29-kube-api-access-zqk2x" (OuterVolumeSpecName: "kube-api-access-zqk2x") pod "2abd573b-6867-434c-a3d5-b2da832d8b29" (UID: "2abd573b-6867-434c-a3d5-b2da832d8b29"). InnerVolumeSpecName "kube-api-access-zqk2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 12:06:27.428530 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:27.428495    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqk2x\" (UniqueName: \"kubernetes.io/projected/2abd573b-6867-434c-a3d5-b2da832d8b29-kube-api-access-zqk2x\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 12:06:27.428530 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:27.428527    2581 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2abd573b-6867-434c-a3d5-b2da832d8b29-must-gather-output\") on node \"ip-10-0-138-183.ec2.internal\" DevicePath \"\""
Jul 07 12:06:28.171329 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:28.171296    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2abd573b-6867-434c-a3d5-b2da832d8b29" path="/var/lib/kubelet/pods/2abd573b-6867-434c-a3d5-b2da832d8b29/volumes"
Jul 07 12:06:28.183591 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:28.183572    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-78phf_must-gather-g9v9k_2abd573b-6867-434c-a3d5-b2da832d8b29/copy/0.log"
Jul 07 12:06:28.183904 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:28.183883    2581 generic.go:356] "Generic (PLEG): container finished" podID="2abd573b-6867-434c-a3d5-b2da832d8b29" containerID="0a6f7b3660b21df2348b765b90eedff4271d63c29ed5cc4d990faeb663166d55" exitCode=143
Jul 07 12:06:28.183979 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:28.183928    2581 scope.go:117] "RemoveContainer" containerID="0a6f7b3660b21df2348b765b90eedff4271d63c29ed5cc4d990faeb663166d55"
Jul 07 12:06:28.183979 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:28.183947    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-78phf/must-gather-g9v9k"
Jul 07 12:06:28.191888 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:28.191873    2581 scope.go:117] "RemoveContainer" containerID="d0306e5f987816501f5a340cdda1fff928fcab8014d5a9e5ce805fc3f8e18083"
Jul 07 12:06:28.202695 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:28.202680    2581 scope.go:117] "RemoveContainer" containerID="0a6f7b3660b21df2348b765b90eedff4271d63c29ed5cc4d990faeb663166d55"
Jul 07 12:06:28.202955 ip-10-0-138-183 kubenswrapper[2581]: E0707 12:06:28.202936    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a6f7b3660b21df2348b765b90eedff4271d63c29ed5cc4d990faeb663166d55\": container with ID starting with 0a6f7b3660b21df2348b765b90eedff4271d63c29ed5cc4d990faeb663166d55 not found: ID does not exist" containerID="0a6f7b3660b21df2348b765b90eedff4271d63c29ed5cc4d990faeb663166d55"
Jul 07 12:06:28.203034 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:28.202965    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a6f7b3660b21df2348b765b90eedff4271d63c29ed5cc4d990faeb663166d55"} err="failed to get container status \"0a6f7b3660b21df2348b765b90eedff4271d63c29ed5cc4d990faeb663166d55\": rpc error: code = NotFound desc = could not find container \"0a6f7b3660b21df2348b765b90eedff4271d63c29ed5cc4d990faeb663166d55\": container with ID starting with 0a6f7b3660b21df2348b765b90eedff4271d63c29ed5cc4d990faeb663166d55 not found: ID does not exist"
Jul 07 12:06:28.203034 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:28.202982    2581 scope.go:117] "RemoveContainer" containerID="d0306e5f987816501f5a340cdda1fff928fcab8014d5a9e5ce805fc3f8e18083"
Jul 07 12:06:28.203270 ip-10-0-138-183 kubenswrapper[2581]: E0707 12:06:28.203252    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0306e5f987816501f5a340cdda1fff928fcab8014d5a9e5ce805fc3f8e18083\": container with ID starting with d0306e5f987816501f5a340cdda1fff928fcab8014d5a9e5ce805fc3f8e18083 not found: ID does not exist" containerID="d0306e5f987816501f5a340cdda1fff928fcab8014d5a9e5ce805fc3f8e18083"
Jul 07 12:06:28.203312 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:28.203277    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0306e5f987816501f5a340cdda1fff928fcab8014d5a9e5ce805fc3f8e18083"} err="failed to get container status \"d0306e5f987816501f5a340cdda1fff928fcab8014d5a9e5ce805fc3f8e18083\": rpc error: code = NotFound desc = could not find container \"d0306e5f987816501f5a340cdda1fff928fcab8014d5a9e5ce805fc3f8e18083\": container with ID starting with d0306e5f987816501f5a340cdda1fff928fcab8014d5a9e5ce805fc3f8e18083 not found: ID does not exist"
Jul 07 12:06:28.977383 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:28.977310    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ff9decad-737f-4260-bc1a-a55386e2e26d/alertmanager/0.log"
Jul 07 12:06:29.004596 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:29.004577    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ff9decad-737f-4260-bc1a-a55386e2e26d/config-reloader/0.log"
Jul 07 12:06:29.032779 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:29.032758    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ff9decad-737f-4260-bc1a-a55386e2e26d/kube-rbac-proxy-web/0.log"
Jul 07 12:06:29.061826 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:29.061809    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ff9decad-737f-4260-bc1a-a55386e2e26d/kube-rbac-proxy/0.log"
Jul 07 12:06:29.091518 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:29.091499    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ff9decad-737f-4260-bc1a-a55386e2e26d/kube-rbac-proxy-metric/0.log"
Jul 07 12:06:29.122731 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:29.122714    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ff9decad-737f-4260-bc1a-a55386e2e26d/prom-label-proxy/0.log"
Jul 07 12:06:29.148119 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:29.148100    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ff9decad-737f-4260-bc1a-a55386e2e26d/init-config-reloader/0.log"
Jul 07 12:06:29.373348 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:29.373309    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6877c5899b-4jnwg_62e155d9-5bf1-441f-a9a6-720f883b176f/metrics-server/0.log"
Jul 07 12:06:29.441472 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:29.441452    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fl86s_148c135d-0b0f-45d0-af99-c4de21b03097/node-exporter/0.log"
Jul 07 12:06:29.469160 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:29.469141    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fl86s_148c135d-0b0f-45d0-af99-c4de21b03097/kube-rbac-proxy/0.log"
Jul 07 12:06:29.493166 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:29.493144    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fl86s_148c135d-0b0f-45d0-af99-c4de21b03097/init-textfile/0.log"
Jul 07 12:06:29.781825 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:29.781778    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3ffe3c1b-fea5-4bfd-8a62-11590159a96a/prometheus/0.log"
Jul 07 12:06:29.801060 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:29.801042    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3ffe3c1b-fea5-4bfd-8a62-11590159a96a/config-reloader/0.log"
Jul 07 12:06:29.822972 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:29.822912    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3ffe3c1b-fea5-4bfd-8a62-11590159a96a/thanos-sidecar/0.log"
Jul 07 12:06:29.842829 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:29.842806    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3ffe3c1b-fea5-4bfd-8a62-11590159a96a/kube-rbac-proxy-web/0.log"
Jul 07 12:06:29.864977 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:29.864954    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3ffe3c1b-fea5-4bfd-8a62-11590159a96a/kube-rbac-proxy/0.log"
Jul 07 12:06:29.893287 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:29.893259    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3ffe3c1b-fea5-4bfd-8a62-11590159a96a/kube-rbac-proxy-thanos/0.log"
Jul 07 12:06:29.919234 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:29.919217    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3ffe3c1b-fea5-4bfd-8a62-11590159a96a/init-config-reloader/0.log"
Jul 07 12:06:29.945644 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:29.945623    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-82xzg_2da87046-353b-4951-aa42-8d095745d164/prometheus-operator/0.log"
Jul 07 12:06:29.966872 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:29.966854    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-82xzg_2da87046-353b-4951-aa42-8d095745d164/kube-rbac-proxy/0.log"
Jul 07 12:06:30.032147 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:30.032081    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-666b56dbc-kp4zb_2af33142-052d-442a-b5bc-e2767c7c6a14/telemeter-client/0.log"
Jul 07 12:06:30.054884 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:30.054867    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-666b56dbc-kp4zb_2af33142-052d-442a-b5bc-e2767c7c6a14/reload/0.log"
Jul 07 12:06:30.078175 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:30.078157    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-666b56dbc-kp4zb_2af33142-052d-442a-b5bc-e2767c7c6a14/kube-rbac-proxy/0.log"
Jul 07 12:06:30.107949 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:30.107920    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67b754fb45-58684_5273bb01-e902-4ae5-a8ea-9f2cedfcf29c/thanos-query/0.log"
Jul 07 12:06:30.134739 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:30.134716    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67b754fb45-58684_5273bb01-e902-4ae5-a8ea-9f2cedfcf29c/kube-rbac-proxy-web/0.log"
Jul 07 12:06:30.160464 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:30.160444    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67b754fb45-58684_5273bb01-e902-4ae5-a8ea-9f2cedfcf29c/kube-rbac-proxy/0.log"
Jul 07 12:06:30.179617 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:30.179601    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67b754fb45-58684_5273bb01-e902-4ae5-a8ea-9f2cedfcf29c/prom-label-proxy/0.log"
Jul 07 12:06:30.202889 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:30.202873    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67b754fb45-58684_5273bb01-e902-4ae5-a8ea-9f2cedfcf29c/kube-rbac-proxy-rules/0.log"
Jul 07 12:06:30.227562 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:30.227546    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67b754fb45-58684_5273bb01-e902-4ae5-a8ea-9f2cedfcf29c/kube-rbac-proxy-metrics/0.log"
Jul 07 12:06:32.419367 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:32.419341    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-nhq5h_33fb0109-522c-4a2d-ab29-985199d918ef/download-server/0.log"
Jul 07 12:06:33.033439 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:33.033409    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vk8v2/perf-node-gather-daemonset-tnjqt"]
Jul 07 12:06:33.033763 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:33.033751    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2abd573b-6867-434c-a3d5-b2da832d8b29" containerName="gather"
Jul 07 12:06:33.033807 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:33.033764    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2abd573b-6867-434c-a3d5-b2da832d8b29" containerName="gather"
Jul 07 12:06:33.033807 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:33.033777    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2abd573b-6867-434c-a3d5-b2da832d8b29" containerName="copy"
Jul 07 12:06:33.033807 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:33.033782    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2abd573b-6867-434c-a3d5-b2da832d8b29" containerName="copy"
Jul 07 12:06:33.033913 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:33.033830    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2abd573b-6867-434c-a3d5-b2da832d8b29" containerName="gather"
Jul 07 12:06:33.033913 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:33.033837    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2abd573b-6867-434c-a3d5-b2da832d8b29" containerName="copy"
Jul 07 12:06:33.037285 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:33.037265    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vk8v2/perf-node-gather-daemonset-tnjqt"
Jul 07 12:06:33.039936 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:33.039908    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vk8v2\"/\"kube-root-ca.crt\""
Jul 07 12:06:33.040109 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:33.040090    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vk8v2\"/\"default-dockercfg-7sbrz\""
Jul 07 12:06:33.041249 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:33.041231    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vk8v2\"/\"openshift-service-ca.crt\""
Jul 07 12:06:33.043990 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:33.043967    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vk8v2/perf-node-gather-daemonset-tnjqt"]
Jul 07 12:06:33.077182 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:33.077152    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fcca69fb-e6e7-480c-ae61-cf87f7e8c838-proc\") pod \"perf-node-gather-daemonset-tnjqt\" (UID: \"fcca69fb-e6e7-480c-ae61-cf87f7e8c838\") " pod="openshift-must-gather-vk8v2/perf-node-gather-daemonset-tnjqt"
Jul 07 12:06:33.077323 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:33.077201    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fcca69fb-e6e7-480c-ae61-cf87f7e8c838-lib-modules\") pod \"perf-node-gather-daemonset-tnjqt\" (UID: \"fcca69fb-e6e7-480c-ae61-cf87f7e8c838\") " pod="openshift-must-gather-vk8v2/perf-node-gather-daemonset-tnjqt"
Jul 07 12:06:33.077323 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:33.077264    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fcca69fb-e6e7-480c-ae61-cf87f7e8c838-sys\") pod \"perf-node-gather-daemonset-tnjqt\" (UID: \"fcca69fb-e6e7-480c-ae61-cf87f7e8c838\") " pod="openshift-must-gather-vk8v2/perf-node-gather-daemonset-tnjqt"
Jul 07 12:06:33.077323 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:33.077304    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fcca69fb-e6e7-480c-ae61-cf87f7e8c838-podres\") pod \"perf-node-gather-daemonset-tnjqt\" (UID: \"fcca69fb-e6e7-480c-ae61-cf87f7e8c838\") " pod="openshift-must-gather-vk8v2/perf-node-gather-daemonset-tnjqt"
Jul 07 12:06:33.077479 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:33.077362    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9tfq\" (UniqueName: \"kubernetes.io/projected/fcca69fb-e6e7-480c-ae61-cf87f7e8c838-kube-api-access-m9tfq\") pod \"perf-node-gather-daemonset-tnjqt\" (UID: \"fcca69fb-e6e7-480c-ae61-cf87f7e8c838\") " pod="openshift-must-gather-vk8v2/perf-node-gather-daemonset-tnjqt"
Jul 07 12:06:33.178084 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:33.178055    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fcca69fb-e6e7-480c-ae61-cf87f7e8c838-proc\") pod \"perf-node-gather-daemonset-tnjqt\" (UID: \"fcca69fb-e6e7-480c-ae61-cf87f7e8c838\") " pod="openshift-must-gather-vk8v2/perf-node-gather-daemonset-tnjqt"
Jul 07 12:06:33.178229 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:33.178115    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fcca69fb-e6e7-480c-ae61-cf87f7e8c838-lib-modules\") pod \"perf-node-gather-daemonset-tnjqt\" (UID: \"fcca69fb-e6e7-480c-ae61-cf87f7e8c838\") " pod="openshift-must-gather-vk8v2/perf-node-gather-daemonset-tnjqt"
Jul 07 12:06:33.178229 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:33.178147    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fcca69fb-e6e7-480c-ae61-cf87f7e8c838-sys\") pod \"perf-node-gather-daemonset-tnjqt\" (UID: \"fcca69fb-e6e7-480c-ae61-cf87f7e8c838\") " pod="openshift-must-gather-vk8v2/perf-node-gather-daemonset-tnjqt"
Jul 07 12:06:33.178229 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:33.178170    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fcca69fb-e6e7-480c-ae61-cf87f7e8c838-podres\") pod \"perf-node-gather-daemonset-tnjqt\" (UID: \"fcca69fb-e6e7-480c-ae61-cf87f7e8c838\") " pod="openshift-must-gather-vk8v2/perf-node-gather-daemonset-tnjqt"
Jul 07 12:06:33.178229 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:33.178177    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fcca69fb-e6e7-480c-ae61-cf87f7e8c838-proc\") pod \"perf-node-gather-daemonset-tnjqt\" (UID: \"fcca69fb-e6e7-480c-ae61-cf87f7e8c838\") " pod="openshift-must-gather-vk8v2/perf-node-gather-daemonset-tnjqt"
Jul 07 12:06:33.178419 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:33.178272    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9tfq\" (UniqueName: \"kubernetes.io/projected/fcca69fb-e6e7-480c-ae61-cf87f7e8c838-kube-api-access-m9tfq\") pod \"perf-node-gather-daemonset-tnjqt\" (UID: \"fcca69fb-e6e7-480c-ae61-cf87f7e8c838\") " pod="openshift-must-gather-vk8v2/perf-node-gather-daemonset-tnjqt"
Jul 07 12:06:33.178419 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:33.178295    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fcca69fb-e6e7-480c-ae61-cf87f7e8c838-podres\") pod \"perf-node-gather-daemonset-tnjqt\" (UID: \"fcca69fb-e6e7-480c-ae61-cf87f7e8c838\") " pod="openshift-must-gather-vk8v2/perf-node-gather-daemonset-tnjqt"
Jul 07 12:06:33.178419 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:33.178275    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fcca69fb-e6e7-480c-ae61-cf87f7e8c838-sys\") pod \"perf-node-gather-daemonset-tnjqt\" (UID: \"fcca69fb-e6e7-480c-ae61-cf87f7e8c838\") " pod="openshift-must-gather-vk8v2/perf-node-gather-daemonset-tnjqt"
Jul 07 12:06:33.178419 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:33.178328    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fcca69fb-e6e7-480c-ae61-cf87f7e8c838-lib-modules\") pod \"perf-node-gather-daemonset-tnjqt\" (UID: \"fcca69fb-e6e7-480c-ae61-cf87f7e8c838\") " pod="openshift-must-gather-vk8v2/perf-node-gather-daemonset-tnjqt"
Jul 07 12:06:33.186318 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:33.186295    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9tfq\" (UniqueName: \"kubernetes.io/projected/fcca69fb-e6e7-480c-ae61-cf87f7e8c838-kube-api-access-m9tfq\") pod \"perf-node-gather-daemonset-tnjqt\" (UID: \"fcca69fb-e6e7-480c-ae61-cf87f7e8c838\") " pod="openshift-must-gather-vk8v2/perf-node-gather-daemonset-tnjqt"
Jul 07 12:06:33.347914 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:33.347891    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vk8v2/perf-node-gather-daemonset-tnjqt"
Jul 07 12:06:33.474827 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:33.474642    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vk8v2/perf-node-gather-daemonset-tnjqt"]
Jul 07 12:06:33.477681 ip-10-0-138-183 kubenswrapper[2581]: W0707 12:06:33.477650    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfcca69fb_e6e7_480c_ae61_cf87f7e8c838.slice/crio-e43cbbb58b39796c450872e4658d82e1caeb29add00b10056e316616f149cfd8 WatchSource:0}: Error finding container e43cbbb58b39796c450872e4658d82e1caeb29add00b10056e316616f149cfd8: Status 404 returned error can't find the container with id e43cbbb58b39796c450872e4658d82e1caeb29add00b10056e316616f149cfd8
Jul 07 12:06:33.831988 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:33.831907    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9q5bf_8bcb4b6a-f2fd-4e77-8c2a-d5e2d7e39e3e/dns/0.log"
Jul 07 12:06:33.850441 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:33.850414    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9q5bf_8bcb4b6a-f2fd-4e77-8c2a-d5e2d7e39e3e/kube-rbac-proxy/0.log"
Jul 07 12:06:33.991411 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:33.991385    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r47kl_04b0713d-434f-4a0a-af55-d93a2a9e9804/dns-node-resolver/0.log"
Jul 07 12:06:34.210859 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:34.210828    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vk8v2/perf-node-gather-daemonset-tnjqt" event={"ID":"fcca69fb-e6e7-480c-ae61-cf87f7e8c838","Type":"ContainerStarted","Data":"9dd14add5ea71d4d4b0762bd3da349dca7bcfdced47b5ce5bb98accfd3768ae3"}
Jul 07 12:06:34.211035 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:34.210878    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vk8v2/perf-node-gather-daemonset-tnjqt" event={"ID":"fcca69fb-e6e7-480c-ae61-cf87f7e8c838","Type":"ContainerStarted","Data":"e43cbbb58b39796c450872e4658d82e1caeb29add00b10056e316616f149cfd8"}
Jul 07 12:06:34.211035 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:34.210945    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vk8v2/perf-node-gather-daemonset-tnjqt"
Jul 07 12:06:34.227237 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:34.227198    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vk8v2/perf-node-gather-daemonset-tnjqt" podStartSLOduration=1.22718652 podStartE2EDuration="1.22718652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 12:06:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 12:06:34.226718981 +0000 UTC m=+1584.573932625" watchObservedRunningTime="2026-07-07 12:06:34.22718652 +0000 UTC m=+1584.574400155"
Jul 07 12:06:34.524604 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:34.524503    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gpk8v_7b7d6bea-562a-423c-bd1c-4b12eb5103ef/node-ca/0.log"
Jul 07 12:06:35.690682 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:35.690653    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hb9w7_23cf426f-2bff-437d-93a2-c004d20e6eb6/serve-healthcheck-canary/0.log"
Jul 07 12:06:36.227685 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:36.227659    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rf5j2_3f5839e5-63b3-422a-a575-ec979c30e759/kube-rbac-proxy/0.log"
Jul 07 12:06:36.245776 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:36.245757    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rf5j2_3f5839e5-63b3-422a-a575-ec979c30e759/exporter/0.log"
Jul 07 12:06:36.264136 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:36.264117    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rf5j2_3f5839e5-63b3-422a-a575-ec979c30e759/extractor/0.log"
Jul 07 12:06:38.282600 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:38.282567    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-57676b4bd8-c4mpb_f63dc8ee-c323-4fe4-931f-627df2115c9f/manager/0.log"
Jul 07 12:06:38.334270 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:38.334241    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-kbsc8_47645c7d-9e9e-4682-a7ab-36dc6a0f0c9b/server/0.log"
Jul 07 12:06:38.403329 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:38.403302    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-bqh87_f02ca4e5-50d1-4b05-9a9b-41e28e5db5eb/manager/0.log"
Jul 07 12:06:38.435262 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:38.435234    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-pnrfw_ccebb7d4-9e60-421d-9c09-b43a19b0f2c0/s3-init/0.log"
Jul 07 12:06:38.450393 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:38.450374    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-pnrfw_ccebb7d4-9e60-421d-9c09-b43a19b0f2c0/download-hf-model/0.log"
Jul 07 12:06:38.477489 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:38.477468    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-rmcgf_f3b659ee-e419-41cc-9958-c651dd1273bd/s3-tls-init-custom/0.log"
Jul 07 12:06:38.492089 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:38.492060    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-rmcgf_f3b659ee-e419-41cc-9958-c651dd1273bd/download-hf-model/0.log"
Jul 07 12:06:38.517590 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:38.517567    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-4p794_50714eb9-46b4-4893-9c55-c2167edc6505/s3-tls-init-serving/0.log"
Jul 07 12:06:38.533555 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:38.533498    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-4p794_50714eb9-46b4-4893-9c55-c2167edc6505/download-hf-model/0.log"
Jul 07 12:06:40.225067 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:40.225040    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vk8v2/perf-node-gather-daemonset-tnjqt"
Jul 07 12:06:44.243863 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:44.243836    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9j6l9_70d0c362-29de-4978-b557-ce4181aabb9a/kube-multus/0.log"
Jul 07 12:06:44.464515 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:44.464441    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p5l2w_5d7ce38f-f030-42d2-a486-84f6a8b6ebc4/kube-multus-additional-cni-plugins/0.log"
Jul 07 12:06:44.486815 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:44.486792    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p5l2w_5d7ce38f-f030-42d2-a486-84f6a8b6ebc4/egress-router-binary-copy/0.log"
Jul 07 12:06:44.508938 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:44.508913    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p5l2w_5d7ce38f-f030-42d2-a486-84f6a8b6ebc4/cni-plugins/0.log"
Jul 07 12:06:44.534114 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:44.534097    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p5l2w_5d7ce38f-f030-42d2-a486-84f6a8b6ebc4/bond-cni-plugin/0.log"
Jul 07 12:06:44.554344 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:44.554320    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p5l2w_5d7ce38f-f030-42d2-a486-84f6a8b6ebc4/routeoverride-cni/0.log"
Jul 07 12:06:44.575850 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:44.575830    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p5l2w_5d7ce38f-f030-42d2-a486-84f6a8b6ebc4/whereabouts-cni-bincopy/0.log"
Jul 07 12:06:44.594726 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:44.594709    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p5l2w_5d7ce38f-f030-42d2-a486-84f6a8b6ebc4/whereabouts-cni/0.log"
Jul 07 12:06:44.898584 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:44.898554    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mt9fg_fd7d46ac-c068-44b2-adca-77ddc1da152e/network-metrics-daemon/0.log"
Jul 07 12:06:44.916751 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:44.916706    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mt9fg_fd7d46ac-c068-44b2-adca-77ddc1da152e/kube-rbac-proxy/0.log"
Jul 07 12:06:46.116686 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:46.116638    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kf48j_4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801/ovn-controller/0.log"
Jul 07 12:06:46.137700 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:46.137675    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kf48j_4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801/ovn-acl-logging/0.log"
Jul 07 12:06:46.137837 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:46.137705    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kf48j_4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801/ovn-acl-logging/1.log"
Jul 07 12:06:46.154767 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:46.154749    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kf48j_4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801/kube-rbac-proxy-node/0.log"
Jul 07 12:06:46.173463 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:46.173446    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kf48j_4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801/kube-rbac-proxy-ovn-metrics/0.log"
Jul 07 12:06:46.192433 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:46.192415    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kf48j_4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801/northd/0.log"
Jul 07 12:06:46.214054 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:46.214032    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kf48j_4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801/nbdb/0.log"
Jul 07 12:06:46.235569 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:46.235528    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kf48j_4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801/sbdb/0.log"
Jul 07 12:06:46.282254 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:46.282223    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kf48j_4d0cbd46-ebd5-4b15-9588-4c6ae7ddd801/ovnkube-controller/0.log"
Jul 07 12:06:47.376879 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:47.376847    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-86xkz_9f060945-568c-484f-affd-2bbaa59faa07/network-check-target-container/0.log"
Jul 07 12:06:48.338498 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:48.338466    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-rmjgp_f1937f0c-df84-4f64-b4be-678b7e871e8c/iptables-alerter/0.log"
Jul 07 12:06:49.003160 ip-10-0-138-183 kubenswrapper[2581]: I0707 12:06:49.003135    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-n49lw_17e097fb-7012-4f55-a0c1-8cc356fc093d/tuned/0.log"