Jul 07 16:45:51.293339 ip-10-0-133-184 systemd[1]: Starting Kubernetes Kubelet...
Jul 07 16:45:51.695847 ip-10-0-133-184 kubenswrapper[2584]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 16:45:51.695847 ip-10-0-133-184 kubenswrapper[2584]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 07 16:45:51.695847 ip-10-0-133-184 kubenswrapper[2584]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 16:45:51.695847 ip-10-0-133-184 kubenswrapper[2584]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 07 16:45:51.695847 ip-10-0-133-184 kubenswrapper[2584]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 16:45:51.697005 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.696862    2584 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 07 16:45:51.701790 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701773    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 16:45:51.701790 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701790    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 16:45:51.701858 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701794    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 16:45:51.701858 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701797    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 16:45:51.701858 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701800    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 16:45:51.701858 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701803    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 16:45:51.701858 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701806    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 16:45:51.701858 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701808    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 16:45:51.701858 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701811    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 16:45:51.701858 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701813    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 16:45:51.701858 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701816    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 16:45:51.701858 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701819    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 16:45:51.701858 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701822    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 16:45:51.701858 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701824    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 16:45:51.701858 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701827    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 16:45:51.701858 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701829    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 16:45:51.701858 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701832    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 16:45:51.701858 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701835    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 16:45:51.701858 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701837    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 16:45:51.701858 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701840    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 16:45:51.701858 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701842    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 16:45:51.701858 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701845    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 16:45:51.702354 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701849    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 16:45:51.702354 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701851    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 16:45:51.702354 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701853    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 16:45:51.702354 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701858    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 16:45:51.702354 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701861    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 16:45:51.702354 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701863    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 16:45:51.702354 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701866    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 16:45:51.702354 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701869    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 16:45:51.702354 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701872    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 16:45:51.702354 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701874    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 16:45:51.702354 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701877    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 16:45:51.702354 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701879    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 16:45:51.702354 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701882    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 16:45:51.702354 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701884    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 16:45:51.702354 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701887    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 16:45:51.702354 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701889    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 16:45:51.702354 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701892    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 16:45:51.702354 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701894    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 16:45:51.702354 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701896    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 16:45:51.702354 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701899    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 16:45:51.702839 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701901    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 16:45:51.702839 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701903    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 16:45:51.702839 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701906    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 16:45:51.702839 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701909    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 16:45:51.702839 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701911    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 16:45:51.702839 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701914    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 16:45:51.702839 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701919    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 16:45:51.702839 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701922    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 16:45:51.702839 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701929    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 16:45:51.702839 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701936    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 16:45:51.702839 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701940    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 16:45:51.702839 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701945    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 16:45:51.702839 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701950    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 16:45:51.702839 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701954    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 16:45:51.702839 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701958    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 16:45:51.702839 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701962    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 16:45:51.702839 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701965    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 16:45:51.702839 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701968    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 16:45:51.702839 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701972    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 16:45:51.703312 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701975    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 16:45:51.703312 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701977    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 16:45:51.703312 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701980    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 16:45:51.703312 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701982    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 16:45:51.703312 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701985    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 16:45:51.703312 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701988    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 16:45:51.703312 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701991    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 16:45:51.703312 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701993    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 16:45:51.703312 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701996    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 16:45:51.703312 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.701998    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 16:45:51.703312 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702001    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 16:45:51.703312 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702003    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 16:45:51.703312 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702005    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 16:45:51.703312 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702008    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 16:45:51.703312 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702011    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 16:45:51.703312 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702013    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 16:45:51.703312 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702016    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 16:45:51.703312 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702019    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 16:45:51.703312 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702022    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 16:45:51.703312 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702025    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 16:45:51.703901 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702028    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 16:45:51.703901 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702031    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 16:45:51.703901 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702033    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 16:45:51.703901 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702036    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 16:45:51.703901 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702038    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 16:45:51.703901 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702041    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 16:45:51.703901 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702059    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 16:45:51.703901 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702063    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 16:45:51.703901 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702068    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 16:45:51.703901 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702072    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 16:45:51.703901 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702074    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 16:45:51.703901 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702077    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 16:45:51.703901 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702079    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 16:45:51.703901 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702082    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 16:45:51.703901 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702085    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 16:45:51.703901 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702087    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 16:45:51.703901 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702090    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 16:45:51.703901 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702094    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 16:45:51.703901 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702796    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 16:45:51.703901 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702803    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 16:45:51.704401 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702806    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 16:45:51.704401 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702809    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 16:45:51.704401 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702812    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 16:45:51.704401 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702815    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 16:45:51.704401 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702819    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 16:45:51.704401 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702822    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 16:45:51.704401 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702824    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 16:45:51.704401 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702827    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 16:45:51.704401 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702829    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 16:45:51.704401 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702833    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 16:45:51.704401 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702836    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 16:45:51.704401 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702839    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 16:45:51.704401 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702842    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 16:45:51.704401 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702845    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 16:45:51.704401 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702848    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 16:45:51.704401 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702850    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 16:45:51.704401 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702853    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 16:45:51.704401 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702856    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 16:45:51.704401 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702858    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 16:45:51.704401 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702861    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 16:45:51.704896 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702863    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 16:45:51.704896 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702866    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 16:45:51.704896 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702868    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 16:45:51.704896 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702871    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 16:45:51.704896 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702873    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 16:45:51.704896 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702876    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 16:45:51.704896 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702878    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 16:45:51.704896 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702881    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 16:45:51.704896 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702883    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 16:45:51.704896 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702885    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 16:45:51.704896 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702888    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 16:45:51.704896 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702891    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 16:45:51.704896 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702893    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 16:45:51.704896 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702898    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 16:45:51.704896 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702901    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 16:45:51.704896 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702904    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 16:45:51.704896 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702906    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 16:45:51.704896 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702909    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 16:45:51.704896 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702912    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 16:45:51.705390 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702914    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 16:45:51.705390 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702917    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 16:45:51.705390 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702920    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 16:45:51.705390 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702923    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 16:45:51.705390 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702926    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 16:45:51.705390 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702928    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 16:45:51.705390 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702931    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 16:45:51.705390 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702934    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 16:45:51.705390 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702936    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 16:45:51.705390 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702939    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 16:45:51.705390 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702941    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 16:45:51.705390 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702945    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 16:45:51.705390 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702948    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 16:45:51.705390 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702952    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 16:45:51.705390 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702954    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 16:45:51.705390 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702957    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 16:45:51.705390 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702960    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 16:45:51.705390 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702962    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 16:45:51.705390 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702966    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 16:45:51.705390 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702968    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 16:45:51.705880 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702971    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 16:45:51.705880 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702974    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 16:45:51.705880 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702976    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 16:45:51.705880 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702978    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 16:45:51.705880 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702981    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 16:45:51.705880 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702983    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 16:45:51.705880 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702986    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 16:45:51.705880 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702988    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 16:45:51.705880 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702992    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 16:45:51.705880 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702995    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 16:45:51.705880 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.702998    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 16:45:51.705880 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703001    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 16:45:51.705880 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703003    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 16:45:51.705880 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703006    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 16:45:51.705880 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703010    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 16:45:51.705880 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703013    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 16:45:51.705880 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703016    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 16:45:51.705880 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703019    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 16:45:51.705880 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703021    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 16:45:51.706358 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703024    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 16:45:51.706358 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703027    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 16:45:51.706358 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703029    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 16:45:51.706358 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703032    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 16:45:51.706358 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703034    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 16:45:51.706358 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703036    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 16:45:51.706358 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703039    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 16:45:51.706358 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703058    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 16:45:51.706358 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703061    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 16:45:51.706358 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703064    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 16:45:51.706358 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703066    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 16:45:51.706358 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703069    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 16:45:51.706358 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703072    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 16:45:51.706358 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703074    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 16:45:51.706358 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703077    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 16:45:51.706358 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703079    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 16:45:51.706358 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703082    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 16:45:51.706358 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703084    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 16:45:51.706358 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703087    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 16:45:51.706358 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703474    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 16:45:51.706867 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703478    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 16:45:51.706867 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703481    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 16:45:51.706867 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703484    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 16:45:51.706867 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703487    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 16:45:51.706867 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703489    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 16:45:51.706867 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703492    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 16:45:51.706867 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703494    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 16:45:51.706867 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703499    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 16:45:51.706867 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703501    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 16:45:51.706867 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703504    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 16:45:51.706867 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703507    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 16:45:51.706867 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703509    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 16:45:51.706867 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703512    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 16:45:51.706867 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703515    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 16:45:51.706867 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703517    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 16:45:51.706867 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703520    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 16:45:51.706867 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703523    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 16:45:51.706867 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703525    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 16:45:51.706867 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703528    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 16:45:51.706867 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703531    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 16:45:51.707387 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703533    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 16:45:51.707387 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703535    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 16:45:51.707387 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703538    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 16:45:51.707387 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703540    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 16:45:51.707387 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703543    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 16:45:51.707387 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703546    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 16:45:51.707387 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703548    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 16:45:51.707387 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703552    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 16:45:51.707387 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703556    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 16:45:51.707387 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703559    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 16:45:51.707387 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703562    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 16:45:51.707387 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703565    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 16:45:51.707387 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703568    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 16:45:51.707387 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703571    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 16:45:51.707387 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703573    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 16:45:51.707387 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703576    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 16:45:51.707387 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703578    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 16:45:51.707387 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703581    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 16:45:51.707387 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703583    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 16:45:51.707846 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703587    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 16:45:51.707846 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703590    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 16:45:51.707846 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703592    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 16:45:51.707846 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703598    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 16:45:51.707846 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703601    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 16:45:51.707846 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703605    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 16:45:51.707846 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703609    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 16:45:51.707846 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703611    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 16:45:51.707846 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703614    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 16:45:51.707846 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703617    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 16:45:51.707846 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703619    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 16:45:51.707846 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703622    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 16:45:51.707846 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703625    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 16:45:51.707846 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703627    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 16:45:51.707846 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703630    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 16:45:51.707846 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703633    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 16:45:51.707846 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703635    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 16:45:51.707846 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703638    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 16:45:51.707846 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703640    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 16:45:51.707846 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703643    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 16:45:51.708398 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703646    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 16:45:51.708398 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703648    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 16:45:51.708398 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703651    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 16:45:51.708398 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703653    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 16:45:51.708398 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703656    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 16:45:51.708398 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703658    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 16:45:51.708398 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703661    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 16:45:51.708398 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703663    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 16:45:51.708398 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703666    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 16:45:51.708398 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703668    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 16:45:51.708398 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703670    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 16:45:51.708398 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703673    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 16:45:51.708398 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703679    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 16:45:51.708398 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703682    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 16:45:51.708398 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703685    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 16:45:51.708398 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703688    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 16:45:51.708398 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703691    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 16:45:51.708398 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703693    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 16:45:51.708398 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703696    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 16:45:51.708856 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703698    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 16:45:51.708856 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703701    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 16:45:51.708856 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703703    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 16:45:51.708856 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703706    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 16:45:51.708856 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703708    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 16:45:51.708856 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703710    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 16:45:51.708856 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703713    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 16:45:51.708856 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703715    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 16:45:51.708856 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703718    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 16:45:51.708856 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703720    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 16:45:51.708856 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703722    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 16:45:51.708856 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703724    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 16:45:51.708856 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703727    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 16:45:51.708856 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703730    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 16:45:51.708856 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703732    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 16:45:51.708856 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703734    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 16:45:51.708856 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703737    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 16:45:51.708856 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703739    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 16:45:51.708856 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703741    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 16:45:51.708856 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.703744    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 16:45:51.709399 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703820    2584 flags.go:64] FLAG: --address="0.0.0.0"
Jul 07 16:45:51.709399 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703827    2584 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 07 16:45:51.709399 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703833    2584 flags.go:64] FLAG: --anonymous-auth="true"
Jul 07 16:45:51.709399 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703837    2584 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 07 16:45:51.709399 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703841    2584 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 07 16:45:51.709399 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703846    2584 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 07 16:45:51.709399 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703850    2584 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 07 16:45:51.709399 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703855    2584 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 07 16:45:51.709399 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703858    2584 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 07 16:45:51.709399 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703861    2584 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 07 16:45:51.709399 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703865    2584 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 07 16:45:51.709399 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703868    2584 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 07 16:45:51.709399 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703871    2584 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 07 16:45:51.709399 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703874    2584 flags.go:64] FLAG: --cgroup-root=""
Jul 07 16:45:51.709399 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703877    2584 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 07 16:45:51.709399 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703880    2584 flags.go:64] FLAG: --client-ca-file=""
Jul 07 16:45:51.709399 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703882    2584 flags.go:64] FLAG: --cloud-provider="external"
Jul 07 16:45:51.709399 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703885    2584 flags.go:64] FLAG: --cluster-dns="[]"
Jul 07 16:45:51.709399 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703890    2584 flags.go:64] FLAG: --cluster-domain=""
Jul 07 16:45:51.709399 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703892    2584 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 07 16:45:51.709399 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703895    2584 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 07 16:45:51.709399 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703898    2584 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 07 16:45:51.709399 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703902    2584 flags.go:64] FLAG: --container-log-max-files="5"
Jul 07 16:45:51.709399 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703905    2584 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 07 16:45:51.709973 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703909    2584 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 07 16:45:51.709973 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703912    2584 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 07 16:45:51.709973 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703916    2584 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 07 16:45:51.709973 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703919    2584 flags.go:64] FLAG: --contention-profiling="false"
Jul 07 16:45:51.709973 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703922    2584 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 07 16:45:51.709973 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703924    2584 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 07 16:45:51.709973 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703929    2584 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 07 16:45:51.709973 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703932    2584 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 07 16:45:51.709973 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703937    2584 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 07 16:45:51.709973 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703940    2584 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 07 16:45:51.709973 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703942    2584 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 07 16:45:51.709973 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703945    2584 flags.go:64] FLAG: --enable-load-reader="false"
Jul 07 16:45:51.709973 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703948    2584 flags.go:64] FLAG: --enable-server="true"
Jul 07 16:45:51.709973 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703952    2584 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 07 16:45:51.709973 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703956    2584 flags.go:64] FLAG: --event-burst="100"
Jul 07 16:45:51.709973 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703959    2584 flags.go:64] FLAG: --event-qps="50"
Jul 07 16:45:51.709973 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703961    2584 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 07 16:45:51.709973 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703964    2584 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 07 16:45:51.709973 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703967    2584 flags.go:64] FLAG: --eviction-hard=""
Jul 07 16:45:51.709973 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703972    2584 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 07 16:45:51.709973 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703975    2584 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 07 16:45:51.709973 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703977    2584 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 07 16:45:51.709973 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703980    2584 flags.go:64] FLAG: --eviction-soft=""
Jul 07 16:45:51.709973 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703983    2584 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 07 16:45:51.709973 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703986    2584 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 07 16:45:51.710610 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703988    2584 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 07 16:45:51.710610 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703991    2584 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 07 16:45:51.710610 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703994    2584 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 07 16:45:51.710610 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703997    2584 flags.go:64] FLAG: --fail-swap-on="true"
Jul 07 16:45:51.710610 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.703999    2584 flags.go:64] FLAG: --feature-gates=""
Jul 07 16:45:51.710610 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704003    2584 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 07 16:45:51.710610 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704006    2584 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 07 16:45:51.710610 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704008    2584 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 07 16:45:51.710610 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704011    2584 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 07 16:45:51.710610 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704014    2584 flags.go:64] FLAG: --healthz-port="10248"
Jul 07 16:45:51.710610 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704017    2584 flags.go:64] FLAG: --help="false"
Jul 07 16:45:51.710610 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704020    2584 flags.go:64] FLAG: --hostname-override="ip-10-0-133-184.ec2.internal"
Jul 07 16:45:51.710610 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704023    2584 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 07 16:45:51.710610 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704028    2584 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 07 16:45:51.710610 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704031    2584 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 07 16:45:51.710610 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704034    2584 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 07 16:45:51.710610 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704037    2584 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 07 16:45:51.710610 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704040    2584 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 07 16:45:51.710610 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704056    2584 flags.go:64] FLAG: --image-service-endpoint=""
Jul 07 16:45:51.710610 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704059    2584 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 07 16:45:51.710610 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704063    2584 flags.go:64] FLAG: --kube-api-burst="100"
Jul 07 16:45:51.710610 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704066    2584 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 07 16:45:51.710610 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704069    2584 flags.go:64] FLAG: --kube-api-qps="50"
Jul 07 16:45:51.710610 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704072    2584 flags.go:64] FLAG: --kube-reserved=""
Jul 07 16:45:51.711229 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704075    2584 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 07 16:45:51.711229 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704077    2584 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 07 16:45:51.711229 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704080    2584 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 07 16:45:51.711229 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704085    2584 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 07 16:45:51.711229 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704087    2584 flags.go:64] FLAG: --lock-file=""
Jul 07 16:45:51.711229 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704090    2584 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 07 16:45:51.711229 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704093    2584 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 07 16:45:51.711229 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704096    2584 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 07 16:45:51.711229 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704101    2584 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 07 16:45:51.711229 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704104    2584 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 07 16:45:51.711229 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704107    2584 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 07 16:45:51.711229 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704109    2584 flags.go:64] FLAG: --logging-format="text"
Jul 07 16:45:51.711229 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704112    2584 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 07 16:45:51.711229 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704116    2584 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 07 16:45:51.711229 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704118    2584 flags.go:64] FLAG: --manifest-url=""
Jul 07 16:45:51.711229 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704121    2584 flags.go:64] FLAG: --manifest-url-header=""
Jul 07 16:45:51.711229 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704126    2584 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 07 16:45:51.711229 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704128    2584 flags.go:64] FLAG: --max-open-files="1000000"
Jul 07 16:45:51.711229 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704132    2584 flags.go:64] FLAG: --max-pods="110"
Jul 07 16:45:51.711229 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704135    2584 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 07 16:45:51.711229 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704138    2584 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 07 16:45:51.711229 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704143    2584 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 07 16:45:51.711229 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704145    2584 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 07 16:45:51.711229 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704148    2584 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 07 16:45:51.711229 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704151    2584 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 07 16:45:51.711840 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704154    2584 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 07 16:45:51.711840 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704161    2584 flags.go:64] FLAG: --node-status-max-images="50"
Jul 07 16:45:51.711840 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704164    2584 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 07 16:45:51.711840 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704168    2584 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 07 16:45:51.711840 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704171    2584 flags.go:64] FLAG: --pod-cidr=""
Jul 07 16:45:51.711840 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704174    2584 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 07 16:45:51.711840 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704178    2584 flags.go:64] FLAG: --pod-manifest-path=""
Jul 07 16:45:51.711840 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704181    2584 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 07 16:45:51.711840 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704184    2584 flags.go:64] FLAG: --pods-per-core="0"
Jul 07 16:45:51.711840 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704187    2584 flags.go:64] FLAG: --port="10250"
Jul 07 16:45:51.711840 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704190    2584 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 07 16:45:51.711840 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704193    2584 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ac84d090b975752b"
Jul 07 16:45:51.711840 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704196    2584 flags.go:64] FLAG: --qos-reserved=""
Jul 07 16:45:51.711840 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704199    2584 flags.go:64] FLAG: --read-only-port="10255"
Jul 07 16:45:51.711840 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704202    2584 flags.go:64] FLAG: --register-node="true"
Jul 07 16:45:51.711840 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704205    2584 flags.go:64] FLAG: --register-with-taints=""
Jul 07 16:45:51.711840 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704209    2584 flags.go:64] FLAG: --registry-burst="10"
Jul 07 16:45:51.711840 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704211    2584 flags.go:64] FLAG: --registry-qps="5"
Jul 07 16:45:51.711840 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704214    2584 flags.go:64] FLAG: --reserved-cpus=""
Jul 07 16:45:51.711840 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704217    2584 flags.go:64] FLAG: --reserved-memory=""
Jul 07 16:45:51.711840 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704220    2584 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 07 16:45:51.711840 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704223    2584 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 07 16:45:51.711840 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704226    2584 flags.go:64] FLAG: --rotate-certificates="false"
Jul 07 16:45:51.711840 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704229    2584 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 07 16:45:51.711840 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704232    2584 flags.go:64] FLAG: --runonce="false"
Jul 07 16:45:51.712462 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704234    2584 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 07 16:45:51.712462 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704238    2584 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 07 16:45:51.712462 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704241    2584 flags.go:64] FLAG: --seccomp-default="false"
Jul 07 16:45:51.712462 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704243    2584 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 07 16:45:51.712462 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704247    2584 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 07 16:45:51.712462 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704251    2584 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 07 16:45:51.712462 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704253    2584 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 07 16:45:51.712462 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704257    2584 flags.go:64] FLAG: --storage-driver-password="root"
Jul 07 16:45:51.712462 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704259    2584 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 07 16:45:51.712462 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704262    2584 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 07 16:45:51.712462 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704266    2584 flags.go:64] FLAG: --storage-driver-user="root"
Jul 07 16:45:51.712462 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704269    2584 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 07 16:45:51.712462 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704272    2584 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 07 16:45:51.712462 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704275    2584 flags.go:64] FLAG: --system-cgroups=""
Jul 07 16:45:51.712462 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704278    2584 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 07 16:45:51.712462 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704282    2584 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 07 16:45:51.712462 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704285    2584 flags.go:64] FLAG: --tls-cert-file=""
Jul 07 16:45:51.712462 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704288    2584 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 07 16:45:51.712462 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704292    2584 flags.go:64] FLAG: --tls-min-version=""
Jul 07 16:45:51.712462 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704296    2584 flags.go:64] FLAG: --tls-private-key-file=""
Jul 07 16:45:51.712462 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704298    2584 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 07 16:45:51.712462 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704301    2584 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 07 16:45:51.712462 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704303    2584 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 07 16:45:51.712462 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704306    2584 flags.go:64] FLAG: --v="2"
Jul 07 16:45:51.712462 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704310    2584 flags.go:64] FLAG: --version="false"
Jul 07 16:45:51.713128 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704314    2584 flags.go:64] FLAG: --vmodule=""
Jul 07 16:45:51.713128 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704318    2584 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 07 16:45:51.713128 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704321    2584 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 07 16:45:51.713128 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704456    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 16:45:51.713128 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704460    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 16:45:51.713128 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704463    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 16:45:51.713128 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704466    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 16:45:51.713128 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704468    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 16:45:51.713128 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704471    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 16:45:51.713128 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704474    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 16:45:51.713128 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704476    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 16:45:51.713128 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704480    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 16:45:51.713128 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704482    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 16:45:51.713128 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704486    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 16:45:51.713128 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704489    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 16:45:51.713128 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704492    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 16:45:51.713128 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704495    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 16:45:51.713128 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704499    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 16:45:51.713128 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704501    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 16:45:51.713598 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704504    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 16:45:51.713598 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704506    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 16:45:51.713598 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704509    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 16:45:51.713598 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704511    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 16:45:51.713598 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704514    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 16:45:51.713598 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704517    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 16:45:51.713598 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704519    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 16:45:51.713598 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704522    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 16:45:51.713598 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704525    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 16:45:51.713598 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704527    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 16:45:51.713598 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704529    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 16:45:51.713598 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704532    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 16:45:51.713598 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704534    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 16:45:51.713598 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704537    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 16:45:51.713598 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704539    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 16:45:51.713598 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704542    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 16:45:51.713598 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704544    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 16:45:51.713598 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704547    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 16:45:51.713598 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704549    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 16:45:51.713598 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704552    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 16:45:51.713598 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704554    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 16:45:51.714138 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704556    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 16:45:51.714138 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704559    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 16:45:51.714138 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704561    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 16:45:51.714138 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704565    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 16:45:51.714138 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704567    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 16:45:51.714138 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704570    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 16:45:51.714138 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704572    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 16:45:51.714138 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704575    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 16:45:51.714138 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704577    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 16:45:51.714138 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704580    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 16:45:51.714138 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704583    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 16:45:51.714138 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704586    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 16:45:51.714138 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704588    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 16:45:51.714138 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704590    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 16:45:51.714138 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704593    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 16:45:51.714138 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704595    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 16:45:51.714138 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704598    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 16:45:51.714138 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704600    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 16:45:51.714138 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704602    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 16:45:51.714605 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704606    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 16:45:51.714605 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704610    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 16:45:51.714605 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704614    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 16:45:51.714605 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704617    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 16:45:51.714605 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704619    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 16:45:51.714605 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704622    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 16:45:51.714605 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704625    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 16:45:51.714605 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704627    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 16:45:51.714605 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704630    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 16:45:51.714605 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704632    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 16:45:51.714605 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704635    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 16:45:51.714605 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704637    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 16:45:51.714605 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704639    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 16:45:51.714605 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704642    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 16:45:51.714605 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704644    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 16:45:51.714605 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704647    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 16:45:51.714605 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704650    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 16:45:51.714605 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704653    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 16:45:51.714605 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704655    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 16:45:51.715092 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704658    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 16:45:51.715092 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704660    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 16:45:51.715092 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704663    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 16:45:51.715092 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704667    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 16:45:51.715092 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704669    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 16:45:51.715092 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704672    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 16:45:51.715092 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704674    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 16:45:51.715092 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704676    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 16:45:51.715092 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704679    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 16:45:51.715092 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704682    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 16:45:51.715092 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704685    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 16:45:51.715092 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704687    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 16:45:51.715092 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704689    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 16:45:51.715092 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704692    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 16:45:51.715092 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704695    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 16:45:51.715092 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704697    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 16:45:51.715092 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704699    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 16:45:51.715092 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704702    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 16:45:51.715092 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704704    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 16:45:51.715092 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704707    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 16:45:51.715604 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704709    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 16:45:51.715604 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704712    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 16:45:51.715604 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704714    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 16:45:51.715604 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.704716    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 16:45:51.715604 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.704721    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 16:45:51.715604 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.712036    2584 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 07 16:45:51.715604 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.712074    2584 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 07 16:45:51.715604 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.712096    2584 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 07 16:45:51.715604 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.712102    2584 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 07 16:45:51.715604 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712159    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 16:45:51.715604 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712164    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 16:45:51.715604 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712168    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 16:45:51.715604 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712171    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 16:45:51.715604 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712173    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 16:45:51.715604 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712176    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 16:45:51.715985 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712178    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 16:45:51.715985 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712181    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 16:45:51.715985 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712183    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 16:45:51.715985 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712186    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 16:45:51.715985 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712188    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 16:45:51.715985 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712191    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 16:45:51.715985 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712193    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 16:45:51.715985 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712196    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 16:45:51.715985 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712198    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 16:45:51.715985 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712201    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 16:45:51.715985 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712204    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 16:45:51.715985 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712206    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 16:45:51.715985 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712209    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 16:45:51.715985 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712213    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 16:45:51.715985 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712216    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 16:45:51.715985 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712219    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 16:45:51.715985 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712222    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 16:45:51.715985 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712224    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 16:45:51.715985 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712227    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 16:45:51.716465 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712230    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 16:45:51.716465 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712233    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 16:45:51.716465 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712235    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 16:45:51.716465 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712238    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 16:45:51.716465 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712242    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 16:45:51.716465 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712245    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 16:45:51.716465 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712247    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 16:45:51.716465 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712250    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 16:45:51.716465 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712252    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 16:45:51.716465 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712255    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 16:45:51.716465 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712257    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 16:45:51.716465 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712260    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 16:45:51.716465 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712262    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 16:45:51.716465 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712265    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 16:45:51.716465 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712267    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 16:45:51.716465 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712270    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 16:45:51.716465 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712272    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 16:45:51.716465 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712274    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 16:45:51.716465 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712276    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 16:45:51.716465 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712279    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 16:45:51.716948 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712281    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 16:45:51.716948 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712284    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 16:45:51.716948 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712286    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 16:45:51.716948 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712288    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 16:45:51.716948 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712291    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 16:45:51.716948 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712293    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 16:45:51.716948 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712296    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 16:45:51.716948 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712298    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 16:45:51.716948 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712301    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 16:45:51.716948 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712303    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 16:45:51.716948 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712306    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 16:45:51.716948 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712308    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 16:45:51.716948 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712310    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 16:45:51.716948 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712313    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 16:45:51.716948 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712315    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 16:45:51.716948 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712317    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 16:45:51.716948 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712321    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 16:45:51.716948 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712324    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 16:45:51.716948 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712327    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 16:45:51.717435 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712329    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 16:45:51.717435 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712332    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 16:45:51.717435 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712334    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 16:45:51.717435 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712337    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 16:45:51.717435 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712339    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 16:45:51.717435 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712343    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 16:45:51.717435 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712346    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 16:45:51.717435 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712349    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 16:45:51.717435 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712352    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 16:45:51.717435 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712370    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 16:45:51.717435 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712374    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 16:45:51.717435 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712377    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 16:45:51.717435 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712380    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 16:45:51.717435 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712383    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 16:45:51.717435 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712386    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 16:45:51.717435 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712388    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 16:45:51.717435 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712392    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 16:45:51.717435 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712396    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 16:45:51.717435 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712399    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 16:45:51.717435 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712401    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 16:45:51.717912 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712404    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 16:45:51.717912 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712406    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 16:45:51.717912 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712409    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 16:45:51.717912 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712411    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 16:45:51.717912 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712414    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 16:45:51.717912 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712416    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 16:45:51.717912 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712418    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 16:45:51.717912 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712421    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 16:45:51.717912 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712423    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 16:45:51.717912 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712426    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 16:45:51.717912 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712428    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 16:45:51.717912 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712431    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 16:45:51.717912 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712434    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 16:45:51.717912 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712436    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 16:45:51.717912 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712439    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 16:45:51.718300 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.712444    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 16:45:51.718300 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712555    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 16:45:51.718300 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712560    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 16:45:51.718300 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712564    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 16:45:51.718300 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712566    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 16:45:51.718300 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712569    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 16:45:51.718300 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712572    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 16:45:51.718300 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712575    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 16:45:51.718300 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712577    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 16:45:51.718300 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712580    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 16:45:51.718300 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712582    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 16:45:51.718300 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712585    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 16:45:51.718300 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712587    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 16:45:51.718300 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712590    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 16:45:51.718300 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712593    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 16:45:51.718665 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712595    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 16:45:51.718665 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712598    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 16:45:51.718665 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712600    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 16:45:51.718665 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712603    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 16:45:51.718665 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712605    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 16:45:51.718665 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712607    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 16:45:51.718665 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712610    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 16:45:51.718665 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712612    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 16:45:51.718665 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712615    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 16:45:51.718665 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712617    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 16:45:51.718665 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712620    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 16:45:51.718665 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712623    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 16:45:51.718665 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712625    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 16:45:51.718665 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712628    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 16:45:51.718665 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712631    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 16:45:51.718665 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712634    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 16:45:51.718665 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712636    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 16:45:51.718665 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712638    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 16:45:51.718665 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712641    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 16:45:51.719280 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712643    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 16:45:51.719280 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712646    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 16:45:51.719280 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712648    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 16:45:51.719280 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712651    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 16:45:51.719280 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712653    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 16:45:51.719280 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712655    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 16:45:51.719280 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712659    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 16:45:51.719280 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712661    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 16:45:51.719280 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712663    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 16:45:51.719280 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712666    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 16:45:51.719280 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712668    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 16:45:51.719280 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712670    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 16:45:51.719280 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712674    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 16:45:51.719280 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712677    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 16:45:51.719280 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712680    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 16:45:51.719280 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712682    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 16:45:51.719280 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712685    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 16:45:51.719280 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712688    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 16:45:51.719280 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712690    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 16:45:51.719280 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712693    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 16:45:51.719766 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712695    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 16:45:51.719766 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712698    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 16:45:51.719766 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712700    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 16:45:51.719766 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712703    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 16:45:51.719766 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712705    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 16:45:51.719766 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712707    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 16:45:51.719766 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712710    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 16:45:51.719766 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712712    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 16:45:51.719766 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712715    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 16:45:51.719766 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712717    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 16:45:51.719766 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712720    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 16:45:51.719766 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712722    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 16:45:51.719766 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712725    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 16:45:51.719766 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712727    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 16:45:51.719766 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712729    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 16:45:51.719766 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712731    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 16:45:51.719766 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712735    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 16:45:51.719766 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712738    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 16:45:51.719766 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712741    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 16:45:51.719766 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712744    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 16:45:51.720264 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712746    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 16:45:51.720264 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712749    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 16:45:51.720264 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712751    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 16:45:51.720264 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712754    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 16:45:51.720264 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712756    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 16:45:51.720264 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712759    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 16:45:51.720264 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712761    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 16:45:51.720264 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712764    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 16:45:51.720264 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712766    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 16:45:51.720264 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712769    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 16:45:51.720264 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712771    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 16:45:51.720264 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712774    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 16:45:51.720264 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712776    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 16:45:51.720264 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712779    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 16:45:51.720264 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712781    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 16:45:51.720264 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712784    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 16:45:51.720264 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712786    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 16:45:51.720264 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712788    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 16:45:51.720264 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712791    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 16:45:51.720264 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712793    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 16:45:51.720756 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712795    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 16:45:51.720756 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712798    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 16:45:51.720756 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712802    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 16:45:51.720756 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712805    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 16:45:51.720756 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712808    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 16:45:51.720756 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:51.712811    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 16:45:51.720756 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.712815    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 16:45:51.720756 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.712932    2584 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 07 16:45:51.720756 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.715361    2584 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 07 16:45:51.720756 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.717148    2584 server.go:1019] "Starting client certificate rotation"
Jul 07 16:45:51.720756 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.717248    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 16:45:51.720756 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.717289    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 16:45:51.721145 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.720965    2584 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 16:45:51.728036 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.728013    2584 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 16:45:51.736149 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.736133    2584 log.go:25] "Validated CRI v1 runtime API"
Jul 07 16:45:51.738311 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.738292    2584 log.go:25] "Validated CRI v1 image API"
Jul 07 16:45:51.739031 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.739013    2584 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 07 16:45:51.741684 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.741660    2584 fs.go:135] Filesystem UUIDs: map[38960e46-3ce0-47a8-85e3-1d5308950a18:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d828e981-a210-4b77-8025-9ec2b3af3307:/dev/nvme0n1p3]
Jul 07 16:45:51.741734 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.741685    2584 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 07 16:45:51.746824 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.746709    2584 manager.go:216] Machine: {Timestamp:2026-07-07 16:45:51.74529943 +0000 UTC m=+0.346508125 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104135 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ac8abf6eeb32030ed80772706a39f SystemUUID:ec2ac8ab-f6ee-b320-30ed-80772706a39f BootID:61fb147e-2de7-4ea2-9f28-eec5c3243595 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:db:a9:4b:14:93 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:db:a9:4b:14:93 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:3c:d5:10:1e:3c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 07 16:45:51.746824 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.746817    2584 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 07 16:45:51.746961 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.746897    2584 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 07 16:45:51.750424 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.750396    2584 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 07 16:45:51.750560 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.750424    2584 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-184.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 07 16:45:51.750649 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.750634    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 16:45:51.751233 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.751223    2584 topology_manager.go:138] "Creating topology manager with none policy"
Jul 07 16:45:51.751267 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.751236    2584 container_manager_linux.go:309] "Creating device plugin manager"
Jul 07 16:45:51.751267 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.751248    2584 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 16:45:51.751977 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.751967    2584 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 16:45:51.753685 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.753674    2584 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 07 16:45:51.757535 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.757524    2584 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 16:45:51.757645 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.757636    2584 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 07 16:45:51.757681 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.757664    2584 kubelet.go:486] "Attempting to sync node with API server"
Jul 07 16:45:51.757681 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.757674    2584 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 07 16:45:51.757736 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.757688    2584 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 07 16:45:51.757736 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.757696    2584 kubelet.go:398] "Adding apiserver pod source"
Jul 07 16:45:51.757736 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.757705    2584 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 07 16:45:51.758026 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.758014    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 16:45:51.758102 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.758033    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 16:45:51.760603 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.760586    2584 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 07 16:45:51.763007 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.762994    2584 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 07 16:45:51.763080 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.763024    2584 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 07 16:45:51.764413 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.764401    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 07 16:45:51.764449 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.764419    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 07 16:45:51.764449 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.764427    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 07 16:45:51.764449 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.764436    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 07 16:45:51.764449 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.764448    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 07 16:45:51.764554 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.764453    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 07 16:45:51.764554 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.764462    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 07 16:45:51.764554 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.764467    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 07 16:45:51.764554 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.764474    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 07 16:45:51.764554 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.764479    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 07 16:45:51.764554 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.764493    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 07 16:45:51.764554 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.764499    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 07 16:45:51.765420 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.765408    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 07 16:45:51.765462 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.765422    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 07 16:45:51.768220 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.768200    2584 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-184.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 07 16:45:51.768220 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:51.768207    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-184.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 07 16:45:51.768339 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:51.768230    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 07 16:45:51.768786 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.768773    2584 server.go:1268] "Started kubelet"
Jul 07 16:45:51.768869 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.768846    2584 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 07 16:45:51.768922 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.768858    2584 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 07 16:45:51.768966 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.768937    2584 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 07 16:45:51.773419 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.773370    2584 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 07 16:45:51.773563 ip-10-0-133-184 systemd[1]: Started Kubernetes Kubelet.
Jul 07 16:45:51.776308 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.776288    2584 server.go:310] "Adding debug handlers to kubelet server"
Jul 07 16:45:51.777230 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:51.777211    2584 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 07 16:45:51.780338 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:51.779220    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-184.ec2.internal.18c00fbbd06bd122  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-184.ec2.internal,UID:ip-10-0-133-184.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-184.ec2.internal,},FirstTimestamp:2026-07-07 16:45:51.768744226 +0000 UTC m=+0.369952921,LastTimestamp:2026-07-07 16:45:51.768744226 +0000 UTC m=+0.369952921,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-184.ec2.internal,}"
Jul 07 16:45:51.785024 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.785003    2584 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 07 16:45:51.785024 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.785018    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 07 16:45:51.785538 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:51.785203    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-184.ec2.internal\" not found"
Jul 07 16:45:51.785791 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.785773    2584 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 07 16:45:51.785881 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.785776    2584 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 07 16:45:51.785881 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.785805    2584 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 07 16:45:51.785881 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.785866    2584 factory.go:55] Registering systemd factory
Jul 07 16:45:51.785881 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.785883    2584 factory.go:223] Registration of the systemd container factory successfully
Jul 07 16:45:51.786065 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.785923    2584 reconstruct.go:97] "Volume reconstruction finished"
Jul 07 16:45:51.786065 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.785932    2584 reconciler.go:29] "Reconciler: start to sync state"
Jul 07 16:45:51.786160 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.786148    2584 factory.go:153] Registering CRI-O factory
Jul 07 16:45:51.786191 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.786162    2584 factory.go:223] Registration of the crio container factory successfully
Jul 07 16:45:51.786221 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.786202    2584 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 07 16:45:51.786221 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.786219    2584 factory.go:103] Registering Raw factory
Jul 07 16:45:51.786276 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.786230    2584 manager.go:1196] Started watching for new ooms in manager
Jul 07 16:45:51.786651 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.786640    2584 manager.go:349] Starting recovery of all containers
Jul 07 16:45:51.797892 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.797722    2584 manager.go:354] Recovery completed
Jul 07 16:45:51.798461 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:51.798437    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 07 16:45:51.798644 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:51.798613    2584 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-184.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 07 16:45:51.802605 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.802592    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:45:51.805350 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.805332    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-184.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:45:51.805429 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.805370    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-184.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:45:51.805429 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.805382    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-184.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:45:51.805993 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.805981    2584 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 07 16:45:51.805993 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.805991    2584 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 07 16:45:51.806094 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.805994    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:45:51.806094 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.806008    2584 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 16:45:51.806809 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.806794    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-184.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:45:51.806906 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.806817    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-184.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:45:51.806906 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.806827    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-184.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:45:51.807422 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.807410    2584 policy_none.go:49] "None policy: Start"
Jul 07 16:45:51.807465 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.807428    2584 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 07 16:45:51.807465 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.807439    2584 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 07 16:45:51.807880 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:51.807807    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-184.ec2.internal.18c00fbbd29a64a1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-184.ec2.internal,UID:ip-10-0-133-184.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-184.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-184.ec2.internal,},FirstTimestamp:2026-07-07 16:45:51.805351073 +0000 UTC m=+0.406559771,LastTimestamp:2026-07-07 16:45:51.805351073 +0000 UTC m=+0.406559771,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-184.ec2.internal,}"
Jul 07 16:45:51.808419 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.808404    2584 policy_none.go:47] "Start"
Jul 07 16:45:51.817414 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.816539    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q25gf"
Jul 07 16:45:51.820644 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:51.820550    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-184.ec2.internal.18c00fbbd29ac49d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-184.ec2.internal,UID:ip-10-0-133-184.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-184.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-184.ec2.internal,},FirstTimestamp:2026-07-07 16:45:51.805375645 +0000 UTC m=+0.406584340,LastTimestamp:2026-07-07 16:45:51.805375645 +0000 UTC m=+0.406584340,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-184.ec2.internal,}"
Jul 07 16:45:51.826215 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.826195    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q25gf"
Jul 07 16:45:51.837741 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.837704    2584 manager.go:337] "Starting Device Plugin manager"
Jul 07 16:45:51.837826 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:51.837765    2584 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 07 16:45:51.837826 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.837779    2584 server.go:87] "Starting device plugin registration server"
Jul 07 16:45:51.838172 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.838156    2584 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 07 16:45:51.838242 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.838173    2584 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 07 16:45:51.838298 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.838283    2584 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 07 16:45:51.838386 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.838371    2584 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 07 16:45:51.838386 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.838386    2584 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 07 16:45:51.841385 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:51.841361    2584 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 07 16:45:51.861148 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:51.841419    2584 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-184.ec2.internal\" not found"
Jul 07 16:45:51.886105 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.886075    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 07 16:45:51.887360 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.887341    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 07 16:45:51.887470 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.887377    2584 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 07 16:45:51.887470 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.887400    2584 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 07 16:45:51.887470 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:51.887428    2584 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 07 16:45:51.890209 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.890184    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 16:45:51.938544 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.938513    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:45:51.939768 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.939747    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-184.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:45:51.939887 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.939781    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-184.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:45:51.939887 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.939792    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-184.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:45:51.939887 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.939823    2584 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-184.ec2.internal"
Jul 07 16:45:51.949923 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.949863    2584 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-184.ec2.internal"
Jul 07 16:45:51.949923 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:51.949888    2584 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-184.ec2.internal\": node \"ip-10-0-133-184.ec2.internal\" not found"
Jul 07 16:45:51.987564 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.987525    2584 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-184.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-184.ec2.internal"]
Jul 07 16:45:51.987744 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.987604    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:45:51.987744 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:51.987614    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-184.ec2.internal\" not found"
Jul 07 16:45:51.991302 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.991286    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-184.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:45:51.991398 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.991324    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-184.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:45:51.991398 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.991339    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-184.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:45:51.992860 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.992844    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:45:51.992987 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.992972    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-184.ec2.internal"
Jul 07 16:45:51.993032 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.993002    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:45:51.993669 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.993657    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-184.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:45:51.993708 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.993668    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-184.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:45:51.993708 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.993684    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-184.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:45:51.993708 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.993698    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-184.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:45:51.993814 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.993688    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-184.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:45:51.993814 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.993766    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-184.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:45:51.995956 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.995939    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-184.ec2.internal"
Jul 07 16:45:51.996068 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.995973    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 16:45:51.996691 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.996674    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-184.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 16:45:51.996793 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.996707    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-184.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 16:45:51.996793 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:51.996723    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-184.ec2.internal" event="NodeHasSufficientPID"
Jul 07 16:45:52.011682 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:52.011660    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-184.ec2.internal\" not found" node="ip-10-0-133-184.ec2.internal"
Jul 07 16:45:52.015364 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:52.015345    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-184.ec2.internal\" not found" node="ip-10-0-133-184.ec2.internal"
Jul 07 16:45:52.087267 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:52.087230    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/47045ed184587f2ba81f9c13a0f2aefa-config\") pod \"kube-apiserver-proxy-ip-10-0-133-184.ec2.internal\" (UID: \"47045ed184587f2ba81f9c13a0f2aefa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-184.ec2.internal"
Jul 07 16:45:52.087267 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:52.087265    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d26d465c12efb451fe9dcf0c005e8db7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-184.ec2.internal\" (UID: \"d26d465c12efb451fe9dcf0c005e8db7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-184.ec2.internal"
Jul 07 16:45:52.087461 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:52.087293    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d26d465c12efb451fe9dcf0c005e8db7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-184.ec2.internal\" (UID: \"d26d465c12efb451fe9dcf0c005e8db7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-184.ec2.internal"
Jul 07 16:45:52.088272 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:52.088254    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-184.ec2.internal\" not found"
Jul 07 16:45:52.187462 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:52.187425    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/47045ed184587f2ba81f9c13a0f2aefa-config\") pod \"kube-apiserver-proxy-ip-10-0-133-184.ec2.internal\" (UID: \"47045ed184587f2ba81f9c13a0f2aefa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-184.ec2.internal"
Jul 07 16:45:52.187635 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:52.187481    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d26d465c12efb451fe9dcf0c005e8db7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-184.ec2.internal\" (UID: \"d26d465c12efb451fe9dcf0c005e8db7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-184.ec2.internal"
Jul 07 16:45:52.187635 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:52.187505    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d26d465c12efb451fe9dcf0c005e8db7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-184.ec2.internal\" (UID: \"d26d465c12efb451fe9dcf0c005e8db7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-184.ec2.internal"
Jul 07 16:45:52.187635 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:52.187555    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d26d465c12efb451fe9dcf0c005e8db7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-184.ec2.internal\" (UID: \"d26d465c12efb451fe9dcf0c005e8db7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-184.ec2.internal"
Jul 07 16:45:52.187635 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:52.187554    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/47045ed184587f2ba81f9c13a0f2aefa-config\") pod \"kube-apiserver-proxy-ip-10-0-133-184.ec2.internal\" (UID: \"47045ed184587f2ba81f9c13a0f2aefa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-184.ec2.internal"
Jul 07 16:45:52.187635 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:52.187554    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d26d465c12efb451fe9dcf0c005e8db7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-184.ec2.internal\" (UID: \"d26d465c12efb451fe9dcf0c005e8db7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-184.ec2.internal"
Jul 07 16:45:52.188492 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:52.188472    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-184.ec2.internal\" not found"
Jul 07 16:45:52.289270 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:52.289199    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-184.ec2.internal\" not found"
Jul 07 16:45:52.314355 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:52.314329    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-184.ec2.internal"
Jul 07 16:45:52.318178 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:52.317977    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-184.ec2.internal"
Jul 07 16:45:52.389832 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:52.389794    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-184.ec2.internal\" not found"
Jul 07 16:45:52.490248 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:52.490218    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-184.ec2.internal\" not found"
Jul 07 16:45:52.590804 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:52.590722    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-184.ec2.internal\" not found"
Jul 07 16:45:52.691303 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:52.691282    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-184.ec2.internal\" not found"
Jul 07 16:45:52.716405 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:52.716378    2584 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 07 16:45:52.717132 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:52.716549    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 16:45:52.787025 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:52.786992    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd26d465c12efb451fe9dcf0c005e8db7.slice/crio-d859f88bc5e9144a99ae0fb9787d7317482d27906d1f0120cbe6e484b824c8a7 WatchSource:0}: Error finding container d859f88bc5e9144a99ae0fb9787d7317482d27906d1f0120cbe6e484b824c8a7: Status 404 returned error can't find the container with id d859f88bc5e9144a99ae0fb9787d7317482d27906d1f0120cbe6e484b824c8a7
Jul 07 16:45:52.787243 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:52.787225    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47045ed184587f2ba81f9c13a0f2aefa.slice/crio-b6aa90fcf6af0de8544082c05fd30c032ac661129ae6c48bfec3c1a688d2ac54 WatchSource:0}: Error finding container b6aa90fcf6af0de8544082c05fd30c032ac661129ae6c48bfec3c1a688d2ac54: Status 404 returned error can't find the container with id b6aa90fcf6af0de8544082c05fd30c032ac661129ae6c48bfec3c1a688d2ac54
Jul 07 16:45:52.791481 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:52.791460    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-184.ec2.internal\" not found"
Jul 07 16:45:52.792739 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:52.792724    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 16:45:52.820983 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:52.820957    2584 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 16:45:52.828708 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:52.828667    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 16:40:52 +0000 UTC" deadline="2028-02-18 06:04:32.591617659 +0000 UTC"
Jul 07 16:45:52.828708 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:52.828706    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14173h18m39.762914586s"
Jul 07 16:45:52.891432 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:52.891382    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-184.ec2.internal" event={"ID":"47045ed184587f2ba81f9c13a0f2aefa","Type":"ContainerStarted","Data":"b6aa90fcf6af0de8544082c05fd30c032ac661129ae6c48bfec3c1a688d2ac54"}
Jul 07 16:45:52.891591 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:52.891519    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-184.ec2.internal\" not found"
Jul 07 16:45:52.892429 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:52.892408    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-184.ec2.internal" event={"ID":"d26d465c12efb451fe9dcf0c005e8db7","Type":"ContainerStarted","Data":"d859f88bc5e9144a99ae0fb9787d7317482d27906d1f0120cbe6e484b824c8a7"}
Jul 07 16:45:52.906131 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:52.906108    2584 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 16:45:52.984902 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:52.984873    2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 16:45:52.985783 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:52.985767    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-184.ec2.internal"
Jul 07 16:45:52.995063 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:52.995027    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 16:45:52.996131 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:52.996117    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-184.ec2.internal"
Jul 07 16:45:53.011486 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.011465    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 16:45:53.759372 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.759279    2584 apiserver.go:52] "Watching apiserver"
Jul 07 16:45:53.773316 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.773284    2584 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 07 16:45:53.774555 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.774520    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-tgj8c","openshift-ovn-kubernetes/ovnkube-node-qlfcs","kube-system/global-pull-secret-syncer-2bkmf","kube-system/kube-apiserver-proxy-ip-10-0-133-184.ec2.internal","openshift-dns/node-resolver-ptgp6","openshift-image-registry/node-ca-cpbvp","openshift-multus/network-metrics-daemon-9kj8c","openshift-network-diagnostics/network-check-target-l6pr5","kube-system/konnectivity-agent-gjbh8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sgl5","openshift-cluster-node-tuning-operator/tuned-9jfs6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-184.ec2.internal","openshift-multus/multus-additional-cni-plugins-phn99","openshift-multus/multus-kdq9n"]
Jul 07 16:45:53.776432 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.776407    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6pr5"
Jul 07 16:45:53.776532 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:53.776492    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6pr5" podUID="0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c"
Jul 07 16:45:53.777935 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.777911    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.779047 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.779026    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.780738 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.780718    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.781926 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.781902    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ptgp6"
Jul 07 16:45:53.783138 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.783117    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sgl5"
Jul 07 16:45:53.784557 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.784539    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9kj8c"
Jul 07 16:45:53.784660 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:53.784616    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9kj8c" podUID="7f7e543c-8cdd-4f28-adb5-570b6004c796"
Jul 07 16:45:53.785342 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.785326    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 07 16:45:53.786729 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.786709    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gjbh8"
Jul 07 16:45:53.789320 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.788548    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cpbvp"
Jul 07 16:45:53.790002 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.789976    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tgj8c"
Jul 07 16:45:53.791821 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.791801    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-phn99"
Jul 07 16:45:53.792920 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.792903    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:45:53.793007 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:53.792964    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bkmf" podUID="b2627fd1-997d-4ce5-84aa-e4b5b85cc307"
Jul 07 16:45:53.794763 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.794743    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 07 16:45:53.795502 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.795483    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 07 16:45:53.795888 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.795864    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p88kp\" (UniqueName: \"kubernetes.io/projected/3d18f299-6f72-46ff-abf2-0ccf120fc875-kube-api-access-p88kp\") pod \"node-resolver-ptgp6\" (UID: \"3d18f299-6f72-46ff-abf2-0ccf120fc875\") " pod="openshift-dns/node-resolver-ptgp6"
Jul 07 16:45:53.796010 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.795900    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/16653221-598d-4229-a6d4-039bdaf5118d-etc-tuned\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.796010 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.795930    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f9035994-1125-405f-91db-cbe6f9c8af5a-registration-dir\") pod \"aws-ebs-csi-driver-node-7sgl5\" (UID: \"f9035994-1125-405f-91db-cbe6f9c8af5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sgl5"
Jul 07 16:45:53.796010 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.795958    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-os-release\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.796010 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.795998    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59sbs\" (UniqueName: \"kubernetes.io/projected/7f7e543c-8cdd-4f28-adb5-570b6004c796-kube-api-access-59sbs\") pod \"network-metrics-daemon-9kj8c\" (UID: \"7f7e543c-8cdd-4f28-adb5-570b6004c796\") " pod="openshift-multus/network-metrics-daemon-9kj8c"
Jul 07 16:45:53.796254 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.796030    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/16653221-598d-4229-a6d4-039bdaf5118d-etc-sysconfig\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.796254 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.796056    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16653221-598d-4229-a6d4-039bdaf5118d-sys\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.796254 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.796080    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-host-run-multus-certs\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.796254 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.796096    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6cpz\" (UniqueName: \"kubernetes.io/projected/73096d96-d52d-47ee-95a8-2242bcfb4888-kube-api-access-x6cpz\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.796254 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.796139    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a1a8fd82-5f36-474b-a780-b30b1d5122ca-konnectivity-ca\") pod \"konnectivity-agent-gjbh8\" (UID: \"a1a8fd82-5f36-474b-a780-b30b1d5122ca\") " pod="kube-system/konnectivity-agent-gjbh8"
Jul 07 16:45:53.796254 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.796181    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-multus-socket-dir-parent\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.796254 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.796218    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d18f299-6f72-46ff-abf2-0ccf120fc875-tmp-dir\") pod \"node-resolver-ptgp6\" (UID: \"3d18f299-6f72-46ff-abf2-0ccf120fc875\") " pod="openshift-dns/node-resolver-ptgp6"
Jul 07 16:45:53.796254 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.796250    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-run-openvswitch\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.796669 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.796277    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f9035994-1125-405f-91db-cbe6f9c8af5a-sys-fs\") pod \"aws-ebs-csi-driver-node-7sgl5\" (UID: \"f9035994-1125-405f-91db-cbe6f9c8af5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sgl5"
Jul 07 16:45:53.796669 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.796307    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-host-run-k8s-cni-cncf-io\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.796669 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.796370    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-host-var-lib-cni-multus\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.796669 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.796396    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-host-var-lib-kubelet\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.796669 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.796422    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16653221-598d-4229-a6d4-039bdaf5118d-lib-modules\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.796669 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.796448    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79sxd\" (UniqueName: \"kubernetes.io/projected/16653221-598d-4229-a6d4-039bdaf5118d-kube-api-access-79sxd\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.796669 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.796483    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/73096d96-d52d-47ee-95a8-2242bcfb4888-cni-binary-copy\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.796669 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.796513    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f7e543c-8cdd-4f28-adb5-570b6004c796-metrics-certs\") pod \"network-metrics-daemon-9kj8c\" (UID: \"7f7e543c-8cdd-4f28-adb5-570b6004c796\") " pod="openshift-multus/network-metrics-daemon-9kj8c"
Jul 07 16:45:53.796669 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.796562    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-etc-openvswitch\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.796669 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.796604    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16653221-598d-4229-a6d4-039bdaf5118d-var-lib-kubelet\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.796669 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.796641    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f9035994-1125-405f-91db-cbe6f9c8af5a-device-dir\") pod \"aws-ebs-csi-driver-node-7sgl5\" (UID: \"f9035994-1125-405f-91db-cbe6f9c8af5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sgl5"
Jul 07 16:45:53.796669 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.796667    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-host-run-netns\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.797271 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.796695    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-host-cni-bin\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.797271 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.796724    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/16653221-598d-4229-a6d4-039bdaf5118d-etc-systemd\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.797271 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.796750    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/16653221-598d-4229-a6d4-039bdaf5118d-run\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.797271 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.796779    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f9035994-1125-405f-91db-cbe6f9c8af5a-etc-selinux\") pod \"aws-ebs-csi-driver-node-7sgl5\" (UID: \"f9035994-1125-405f-91db-cbe6f9c8af5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sgl5"
Jul 07 16:45:53.797271 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.796803    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-host-var-lib-cni-bin\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.797271 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.796823    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4w7jn\""
Jul 07 16:45:53.797271 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.796832    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.797271 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.796867    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 07 16:45:53.797271 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.796860    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f959ebb9-6e2d-48da-88b8-647a596e1413-ovnkube-config\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.797271 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.796827    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 07 16:45:53.797271 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.796903    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/16653221-598d-4229-a6d4-039bdaf5118d-etc-sysctl-d\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.797271 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.796932    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16653221-598d-4229-a6d4-039bdaf5118d-tmp\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.797271 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.796963    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-hostroot\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.797271 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.797001    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f959ebb9-6e2d-48da-88b8-647a596e1413-ovnkube-script-lib\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.797271 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.797036    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-log-socket\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.797271 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.797064    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f959ebb9-6e2d-48da-88b8-647a596e1413-ovn-node-metrics-cert\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.797271 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.797095    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16653221-598d-4229-a6d4-039bdaf5118d-etc-kubernetes\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.797271 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.797144    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnwqk\" (UniqueName: \"kubernetes.io/projected/f9035994-1125-405f-91db-cbe6f9c8af5a-kube-api-access-pnwqk\") pod \"aws-ebs-csi-driver-node-7sgl5\" (UID: \"f9035994-1125-405f-91db-cbe6f9c8af5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sgl5"
Jul 07 16:45:53.797271 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.796834    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 07 16:45:53.798209 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.797175    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a1a8fd82-5f36-474b-a780-b30b1d5122ca-agent-certs\") pod \"konnectivity-agent-gjbh8\" (UID: \"a1a8fd82-5f36-474b-a780-b30b1d5122ca\") " pod="kube-system/konnectivity-agent-gjbh8"
Jul 07 16:45:53.798209 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.797226    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-system-cni-dir\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.798209 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.797254    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-host-run-netns\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.798209 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.797272    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmkv5\" (UniqueName: \"kubernetes.io/projected/f959ebb9-6e2d-48da-88b8-647a596e1413-kube-api-access-qmkv5\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.798209 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.797346    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/16653221-598d-4229-a6d4-039bdaf5118d-etc-sysctl-conf\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.798209 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.797379    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-multus-cni-dir\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.798209 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.797419    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-multus-conf-dir\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.798209 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.797456    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/73096d96-d52d-47ee-95a8-2242bcfb4888-multus-daemon-config\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.798209 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.797471    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-host-kubelet\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.798209 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.797531    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-node-log\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.798209 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.797546    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f9035994-1125-405f-91db-cbe6f9c8af5a-socket-dir\") pod \"aws-ebs-csi-driver-node-7sgl5\" (UID: \"f9035994-1125-405f-91db-cbe6f9c8af5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sgl5"
Jul 07 16:45:53.798209 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.797561    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-etc-kubernetes\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.798209 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.797584    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-systemd-units\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.798209 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.797619    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-run-ovn\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.798209 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.797633    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16653221-598d-4229-a6d4-039bdaf5118d-host\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.798209 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.797654    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hppvt\" (UniqueName: \"kubernetes.io/projected/0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c-kube-api-access-hppvt\") pod \"network-check-target-l6pr5\" (UID: \"0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c\") " pod="openshift-network-diagnostics/network-check-target-l6pr5"
Jul 07 16:45:53.798209 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.797692    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-host-slash\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.798772 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.797717    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-run-systemd\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.798772 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.797742    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-var-lib-openvswitch\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.798772 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.797760    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-host-run-ovn-kubernetes\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.798772 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.797790    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f959ebb9-6e2d-48da-88b8-647a596e1413-env-overrides\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.798772 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.797824    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f9035994-1125-405f-91db-cbe6f9c8af5a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7sgl5\" (UID: \"f9035994-1125-405f-91db-cbe6f9c8af5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sgl5"
Jul 07 16:45:53.798772 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.797845    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-cnibin\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.798772 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.797874    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3d18f299-6f72-46ff-abf2-0ccf120fc875-hosts-file\") pod \"node-resolver-ptgp6\" (UID: \"3d18f299-6f72-46ff-abf2-0ccf120fc875\") " pod="openshift-dns/node-resolver-ptgp6"
Jul 07 16:45:53.798772 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.797941    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/16653221-598d-4229-a6d4-039bdaf5118d-etc-modprobe-d\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.798772 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.797987    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-host-cni-netd\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.798772 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.798097    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 07 16:45:53.798772 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.798115    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 07 16:45:53.798772 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.798165    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 07 16:45:53.799664 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.799623    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rhppj\""
Jul 07 16:45:53.799772 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.799703    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8w826\""
Jul 07 16:45:53.799930 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.799903    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 07 16:45:53.800160 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.800110    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 07 16:45:53.800519 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.800498    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 07 16:45:53.801618 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.801412    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 07 16:45:53.802294 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.802274    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 07 16:45:53.802389 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.802280    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 07 16:45:53.802971 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.802953    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 07 16:45:53.805385 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.805362    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 07 16:45:53.805526 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.805505    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 07 16:45:53.805622 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.805528    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 07 16:45:53.805622 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.805610    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 07 16:45:53.805723 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.805708    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 07 16:45:53.805787 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.805773    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hkp9n\""
Jul 07 16:45:53.805902 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.805878    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-p979k\""
Jul 07 16:45:53.806085 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.806061    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 07 16:45:53.806124 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.806105    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 07 16:45:53.806124 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.806117    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6s7cq\""
Jul 07 16:45:53.806807 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.806786    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 07 16:45:53.807022 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.807008    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hjmkd\""
Jul 07 16:45:53.807136 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.807067    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 07 16:45:53.807189 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.807155    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 07 16:45:53.807320 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.807274    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-h2hv9\""
Jul 07 16:45:53.807514 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.807404    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-m4drx\""
Jul 07 16:45:53.807514 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.807480    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 07 16:45:53.807726 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.807707    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 07 16:45:53.816265 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.816231    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 16:45:53.838508 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.838483    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qq476"
Jul 07 16:45:53.849612 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.849591    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qq476"
Jul 07 16:45:53.887437 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.887397    2584 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 07 16:45:53.899073 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.899028    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f9035994-1125-405f-91db-cbe6f9c8af5a-sys-fs\") pod \"aws-ebs-csi-driver-node-7sgl5\" (UID: \"f9035994-1125-405f-91db-cbe6f9c8af5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sgl5"
Jul 07 16:45:53.899254 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.899085    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b4f517c-5cf2-440c-ad4a-bf3aa0934616-system-cni-dir\") pod \"multus-additional-cni-plugins-phn99\" (UID: \"3b4f517c-5cf2-440c-ad4a-bf3aa0934616\") " pod="openshift-multus/multus-additional-cni-plugins-phn99"
Jul 07 16:45:53.899254 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.899115    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b4f517c-5cf2-440c-ad4a-bf3aa0934616-cni-binary-copy\") pod \"multus-additional-cni-plugins-phn99\" (UID: \"3b4f517c-5cf2-440c-ad4a-bf3aa0934616\") " pod="openshift-multus/multus-additional-cni-plugins-phn99"
Jul 07 16:45:53.899254 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.899133    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2627fd1-997d-4ce5-84aa-e4b5b85cc307-original-pull-secret\") pod \"global-pull-secret-syncer-2bkmf\" (UID: \"b2627fd1-997d-4ce5-84aa-e4b5b85cc307\") " pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:45:53.899254 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.899151    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-host-run-k8s-cni-cncf-io\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.899254 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.899153    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f9035994-1125-405f-91db-cbe6f9c8af5a-sys-fs\") pod \"aws-ebs-csi-driver-node-7sgl5\" (UID: \"f9035994-1125-405f-91db-cbe6f9c8af5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sgl5"
Jul 07 16:45:53.899254 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.899174    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-host-var-lib-cni-multus\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.899254 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.899221    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-host-var-lib-kubelet\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.899254 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.899239    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-host-run-k8s-cni-cncf-io\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.899667 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.899282    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-host-var-lib-cni-multus\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.899667 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.899288    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16653221-598d-4229-a6d4-039bdaf5118d-lib-modules\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.899667 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.899293    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-host-var-lib-kubelet\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.899667 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.899319    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79sxd\" (UniqueName: \"kubernetes.io/projected/16653221-598d-4229-a6d4-039bdaf5118d-kube-api-access-79sxd\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.899667 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.899391    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a14a5b96-76aa-4d03-be55-8c23f342e4d9-host-slash\") pod \"iptables-alerter-tgj8c\" (UID: \"a14a5b96-76aa-4d03-be55-8c23f342e4d9\") " pod="openshift-network-operator/iptables-alerter-tgj8c"
Jul 07 16:45:53.899667 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.899476    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16653221-598d-4229-a6d4-039bdaf5118d-lib-modules\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.899667 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.899487    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/73096d96-d52d-47ee-95a8-2242bcfb4888-cni-binary-copy\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.899667 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.899523    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f7e543c-8cdd-4f28-adb5-570b6004c796-metrics-certs\") pod \"network-metrics-daemon-9kj8c\" (UID: \"7f7e543c-8cdd-4f28-adb5-570b6004c796\") " pod="openshift-multus/network-metrics-daemon-9kj8c"
Jul 07 16:45:53.899667 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.899606    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-etc-openvswitch\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.900122 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:53.899686    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:45:53.900122 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.899737    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-etc-openvswitch\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.900122 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:53.899759    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f7e543c-8cdd-4f28-adb5-570b6004c796-metrics-certs podName:7f7e543c-8cdd-4f28-adb5-570b6004c796 nodeName:}" failed. No retries permitted until 2026-07-07 16:45:54.399734695 +0000 UTC m=+3.000943404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f7e543c-8cdd-4f28-adb5-570b6004c796-metrics-certs") pod "network-metrics-daemon-9kj8c" (UID: "7f7e543c-8cdd-4f28-adb5-570b6004c796") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:45:53.900122 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.899781    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16653221-598d-4229-a6d4-039bdaf5118d-var-lib-kubelet\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.900122 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.899809    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f9035994-1125-405f-91db-cbe6f9c8af5a-device-dir\") pod \"aws-ebs-csi-driver-node-7sgl5\" (UID: \"f9035994-1125-405f-91db-cbe6f9c8af5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sgl5"
Jul 07 16:45:53.900122 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.899834    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-host-run-netns\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.900122 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.899861    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-host-cni-bin\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.900122 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.899878    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16653221-598d-4229-a6d4-039bdaf5118d-var-lib-kubelet\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.900122 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.899934    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f9035994-1125-405f-91db-cbe6f9c8af5a-device-dir\") pod \"aws-ebs-csi-driver-node-7sgl5\" (UID: \"f9035994-1125-405f-91db-cbe6f9c8af5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sgl5"
Jul 07 16:45:53.900122 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.899949    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/16653221-598d-4229-a6d4-039bdaf5118d-etc-systemd\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.900122 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.899886    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/16653221-598d-4229-a6d4-039bdaf5118d-etc-systemd\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.900122 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.899983    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-host-run-netns\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.900122 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.899994    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-host-cni-bin\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.900122 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.900007    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/16653221-598d-4229-a6d4-039bdaf5118d-run\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.900122 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.900034    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f9035994-1125-405f-91db-cbe6f9c8af5a-etc-selinux\") pod \"aws-ebs-csi-driver-node-7sgl5\" (UID: \"f9035994-1125-405f-91db-cbe6f9c8af5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sgl5"
Jul 07 16:45:53.900122 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.900080    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts2dw\" (UniqueName: \"kubernetes.io/projected/a14a5b96-76aa-4d03-be55-8c23f342e4d9-kube-api-access-ts2dw\") pod \"iptables-alerter-tgj8c\" (UID: \"a14a5b96-76aa-4d03-be55-8c23f342e4d9\") " pod="openshift-network-operator/iptables-alerter-tgj8c"
Jul 07 16:45:53.900122 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.900108    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-host-var-lib-cni-bin\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.900982 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.900135    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.900982 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.900169    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f959ebb9-6e2d-48da-88b8-647a596e1413-ovnkube-config\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.900982 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.900188    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-host-var-lib-cni-bin\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.900982 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.900208    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f9035994-1125-405f-91db-cbe6f9c8af5a-etc-selinux\") pod \"aws-ebs-csi-driver-node-7sgl5\" (UID: \"f9035994-1125-405f-91db-cbe6f9c8af5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sgl5"
Jul 07 16:45:53.900982 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.900210    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.900982 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.900210    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/16653221-598d-4229-a6d4-039bdaf5118d-etc-sysctl-d\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.900982 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.900250    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16653221-598d-4229-a6d4-039bdaf5118d-tmp\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.900982 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.900269    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-hostroot\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.900982 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.900290    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f959ebb9-6e2d-48da-88b8-647a596e1413-ovnkube-script-lib\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.900982 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.900312    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b4f517c-5cf2-440c-ad4a-bf3aa0934616-tuning-conf-dir\") pod \"multus-additional-cni-plugins-phn99\" (UID: \"3b4f517c-5cf2-440c-ad4a-bf3aa0934616\") " pod="openshift-multus/multus-additional-cni-plugins-phn99"
Jul 07 16:45:53.900982 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.900330    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-log-socket\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.900982 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.900333    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/16653221-598d-4229-a6d4-039bdaf5118d-etc-sysctl-d\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.900982 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.900346    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f959ebb9-6e2d-48da-88b8-647a596e1413-ovn-node-metrics-cert\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.900982 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.900389    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16653221-598d-4229-a6d4-039bdaf5118d-etc-kubernetes\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.900982 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.900417    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnwqk\" (UniqueName: \"kubernetes.io/projected/f9035994-1125-405f-91db-cbe6f9c8af5a-kube-api-access-pnwqk\") pod \"aws-ebs-csi-driver-node-7sgl5\" (UID: \"f9035994-1125-405f-91db-cbe6f9c8af5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sgl5"
Jul 07 16:45:53.900982 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.900442    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a1a8fd82-5f36-474b-a780-b30b1d5122ca-agent-certs\") pod \"konnectivity-agent-gjbh8\" (UID: \"a1a8fd82-5f36-474b-a780-b30b1d5122ca\") " pod="kube-system/konnectivity-agent-gjbh8"
Jul 07 16:45:53.900982 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.900468    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/266c4079-c003-4676-a480-fb0e19a14cce-host\") pod \"node-ca-cpbvp\" (UID: \"266c4079-c003-4676-a480-fb0e19a14cce\") " pod="openshift-image-registry/node-ca-cpbvp"
Jul 07 16:45:53.901863 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.900495    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/266c4079-c003-4676-a480-fb0e19a14cce-serviceca\") pod \"node-ca-cpbvp\" (UID: \"266c4079-c003-4676-a480-fb0e19a14cce\") " pod="openshift-image-registry/node-ca-cpbvp"
Jul 07 16:45:53.901863 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.900520    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msm4n\" (UniqueName: \"kubernetes.io/projected/266c4079-c003-4676-a480-fb0e19a14cce-kube-api-access-msm4n\") pod \"node-ca-cpbvp\" (UID: \"266c4079-c003-4676-a480-fb0e19a14cce\") " pod="openshift-image-registry/node-ca-cpbvp"
Jul 07 16:45:53.901863 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.900551    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-system-cni-dir\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.901863 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.900593    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-host-run-netns\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.901863 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.900627    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmkv5\" (UniqueName: \"kubernetes.io/projected/f959ebb9-6e2d-48da-88b8-647a596e1413-kube-api-access-qmkv5\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.901863 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.900651    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/16653221-598d-4229-a6d4-039bdaf5118d-etc-sysctl-conf\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.901863 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.900648    2584 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 07 16:45:53.901863 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.900671    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxdqb\" (UniqueName: \"kubernetes.io/projected/3b4f517c-5cf2-440c-ad4a-bf3aa0934616-kube-api-access-cxdqb\") pod \"multus-additional-cni-plugins-phn99\" (UID: \"3b4f517c-5cf2-440c-ad4a-bf3aa0934616\") " pod="openshift-multus/multus-additional-cni-plugins-phn99"
Jul 07 16:45:53.901863 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.900125    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/16653221-598d-4229-a6d4-039bdaf5118d-run\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.901863 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.900708    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16653221-598d-4229-a6d4-039bdaf5118d-etc-kubernetes\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.901863 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.900785    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f959ebb9-6e2d-48da-88b8-647a596e1413-ovnkube-config\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.901863 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.900648    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-host-run-netns\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.901863 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.900167    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/73096d96-d52d-47ee-95a8-2242bcfb4888-cni-binary-copy\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.901863 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.901044    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-log-socket\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.901863 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.901087    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-multus-cni-dir\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.901863 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.901117    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-system-cni-dir\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.901863 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.901164    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-multus-conf-dir\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.901863 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.901189    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/73096d96-d52d-47ee-95a8-2242bcfb4888-multus-daemon-config\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.902785 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.901214    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-host-kubelet\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.902785 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.901270    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-node-log\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.902785 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.901287    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/16653221-598d-4229-a6d4-039bdaf5118d-etc-sysctl-conf\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.902785 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.901306    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f9035994-1125-405f-91db-cbe6f9c8af5a-socket-dir\") pod \"aws-ebs-csi-driver-node-7sgl5\" (UID: \"f9035994-1125-405f-91db-cbe6f9c8af5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sgl5"
Jul 07 16:45:53.902785 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.901334    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-etc-kubernetes\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.902785 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.901338    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-hostroot\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.902785 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.901354    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f959ebb9-6e2d-48da-88b8-647a596e1413-ovnkube-script-lib\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.902785 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.901358    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-systemd-units\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.902785 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.901413    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-run-ovn\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.902785 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.901422    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-systemd-units\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.902785 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.901443    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16653221-598d-4229-a6d4-039bdaf5118d-host\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.902785 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.901476    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3b4f517c-5cf2-440c-ad4a-bf3aa0934616-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-phn99\" (UID: \"3b4f517c-5cf2-440c-ad4a-bf3aa0934616\") " pod="openshift-multus/multus-additional-cni-plugins-phn99"
Jul 07 16:45:53.902785 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.901481    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-host-kubelet\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.902785 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.901403    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-multus-conf-dir\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.902785 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.901538    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-multus-cni-dir\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.902785 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.901576    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-node-log\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.902785 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.901594    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-etc-kubernetes\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.902785 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.901613    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hppvt\" (UniqueName: \"kubernetes.io/projected/0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c-kube-api-access-hppvt\") pod \"network-check-target-l6pr5\" (UID: \"0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c\") " pod="openshift-network-diagnostics/network-check-target-l6pr5"
Jul 07 16:45:53.903679 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.901724    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-run-ovn\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.903679 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.901847    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f9035994-1125-405f-91db-cbe6f9c8af5a-socket-dir\") pod \"aws-ebs-csi-driver-node-7sgl5\" (UID: \"f9035994-1125-405f-91db-cbe6f9c8af5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sgl5"
Jul 07 16:45:53.903679 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.901868    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/73096d96-d52d-47ee-95a8-2242bcfb4888-multus-daemon-config\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.903679 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.901892    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-host-slash\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.903679 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.901926    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-run-systemd\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.903679 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.901928    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16653221-598d-4229-a6d4-039bdaf5118d-host\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.903679 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.901957    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-host-slash\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.903679 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.901997    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-var-lib-openvswitch\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.903679 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902021    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-host-run-ovn-kubernetes\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.903679 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902092    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f959ebb9-6e2d-48da-88b8-647a596e1413-env-overrides\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.903679 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902125    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-run-systemd\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.903679 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902165    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-host-run-ovn-kubernetes\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.903679 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902173    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-var-lib-openvswitch\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.903679 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902203    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f9035994-1125-405f-91db-cbe6f9c8af5a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7sgl5\" (UID: \"f9035994-1125-405f-91db-cbe6f9c8af5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sgl5"
Jul 07 16:45:53.903679 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902239    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b2627fd1-997d-4ce5-84aa-e4b5b85cc307-kubelet-config\") pod \"global-pull-secret-syncer-2bkmf\" (UID: \"b2627fd1-997d-4ce5-84aa-e4b5b85cc307\") " pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:45:53.903679 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902263    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f9035994-1125-405f-91db-cbe6f9c8af5a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7sgl5\" (UID: \"f9035994-1125-405f-91db-cbe6f9c8af5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sgl5"
Jul 07 16:45:53.903679 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902274    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-cnibin\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.904477 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902308    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3d18f299-6f72-46ff-abf2-0ccf120fc875-hosts-file\") pod \"node-resolver-ptgp6\" (UID: \"3d18f299-6f72-46ff-abf2-0ccf120fc875\") " pod="openshift-dns/node-resolver-ptgp6"
Jul 07 16:45:53.904477 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902324    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-cnibin\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.904477 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902335    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/16653221-598d-4229-a6d4-039bdaf5118d-etc-modprobe-d\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.904477 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902362    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-host-cni-netd\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.904477 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902389    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b2627fd1-997d-4ce5-84aa-e4b5b85cc307-dbus\") pod \"global-pull-secret-syncer-2bkmf\" (UID: \"b2627fd1-997d-4ce5-84aa-e4b5b85cc307\") " pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:45:53.904477 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902427    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3d18f299-6f72-46ff-abf2-0ccf120fc875-hosts-file\") pod \"node-resolver-ptgp6\" (UID: \"3d18f299-6f72-46ff-abf2-0ccf120fc875\") " pod="openshift-dns/node-resolver-ptgp6"
Jul 07 16:45:53.904477 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902442    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p88kp\" (UniqueName: \"kubernetes.io/projected/3d18f299-6f72-46ff-abf2-0ccf120fc875-kube-api-access-p88kp\") pod \"node-resolver-ptgp6\" (UID: \"3d18f299-6f72-46ff-abf2-0ccf120fc875\") " pod="openshift-dns/node-resolver-ptgp6"
Jul 07 16:45:53.904477 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902471    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/16653221-598d-4229-a6d4-039bdaf5118d-etc-tuned\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.904477 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902474    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/16653221-598d-4229-a6d4-039bdaf5118d-etc-modprobe-d\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.904477 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902497    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f9035994-1125-405f-91db-cbe6f9c8af5a-registration-dir\") pod \"aws-ebs-csi-driver-node-7sgl5\" (UID: \"f9035994-1125-405f-91db-cbe6f9c8af5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sgl5"
Jul 07 16:45:53.904477 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902507    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f959ebb9-6e2d-48da-88b8-647a596e1413-env-overrides\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.904477 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902527    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a14a5b96-76aa-4d03-be55-8c23f342e4d9-iptables-alerter-script\") pod \"iptables-alerter-tgj8c\" (UID: \"a14a5b96-76aa-4d03-be55-8c23f342e4d9\") " pod="openshift-network-operator/iptables-alerter-tgj8c"
Jul 07 16:45:53.904477 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902537    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-host-cni-netd\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.904477 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902554    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b4f517c-5cf2-440c-ad4a-bf3aa0934616-cnibin\") pod \"multus-additional-cni-plugins-phn99\" (UID: \"3b4f517c-5cf2-440c-ad4a-bf3aa0934616\") " pod="openshift-multus/multus-additional-cni-plugins-phn99"
Jul 07 16:45:53.904477 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902591    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3b4f517c-5cf2-440c-ad4a-bf3aa0934616-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-phn99\" (UID: \"3b4f517c-5cf2-440c-ad4a-bf3aa0934616\") " pod="openshift-multus/multus-additional-cni-plugins-phn99"
Jul 07 16:45:53.904477 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902636    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-os-release\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.904477 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902665    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59sbs\" (UniqueName: \"kubernetes.io/projected/7f7e543c-8cdd-4f28-adb5-570b6004c796-kube-api-access-59sbs\") pod \"network-metrics-daemon-9kj8c\" (UID: \"7f7e543c-8cdd-4f28-adb5-570b6004c796\") " pod="openshift-multus/network-metrics-daemon-9kj8c"
Jul 07 16:45:53.905178 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902693    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/16653221-598d-4229-a6d4-039bdaf5118d-etc-sysconfig\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.905178 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902692    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f9035994-1125-405f-91db-cbe6f9c8af5a-registration-dir\") pod \"aws-ebs-csi-driver-node-7sgl5\" (UID: \"f9035994-1125-405f-91db-cbe6f9c8af5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sgl5"
Jul 07 16:45:53.905178 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902717    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16653221-598d-4229-a6d4-039bdaf5118d-sys\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.905178 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902751    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3b4f517c-5cf2-440c-ad4a-bf3aa0934616-os-release\") pod \"multus-additional-cni-plugins-phn99\" (UID: \"3b4f517c-5cf2-440c-ad4a-bf3aa0934616\") " pod="openshift-multus/multus-additional-cni-plugins-phn99"
Jul 07 16:45:53.905178 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902784    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-host-run-multus-certs\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.905178 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902811    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6cpz\" (UniqueName: \"kubernetes.io/projected/73096d96-d52d-47ee-95a8-2242bcfb4888-kube-api-access-x6cpz\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.905178 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902819    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16653221-598d-4229-a6d4-039bdaf5118d-sys\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.905178 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902833    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-os-release\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.905178 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902838    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a1a8fd82-5f36-474b-a780-b30b1d5122ca-konnectivity-ca\") pod \"konnectivity-agent-gjbh8\" (UID: \"a1a8fd82-5f36-474b-a780-b30b1d5122ca\") " pod="kube-system/konnectivity-agent-gjbh8"
Jul 07 16:45:53.905178 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902873    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-multus-socket-dir-parent\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.905178 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902925    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-multus-socket-dir-parent\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.905178 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.902961    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/73096d96-d52d-47ee-95a8-2242bcfb4888-host-run-multus-certs\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.905178 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.903078    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d18f299-6f72-46ff-abf2-0ccf120fc875-tmp-dir\") pod \"node-resolver-ptgp6\" (UID: \"3d18f299-6f72-46ff-abf2-0ccf120fc875\") " pod="openshift-dns/node-resolver-ptgp6"
Jul 07 16:45:53.905178 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.903084    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/16653221-598d-4229-a6d4-039bdaf5118d-etc-sysconfig\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.905178 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.903145    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-run-openvswitch\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.905178 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.903383    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d18f299-6f72-46ff-abf2-0ccf120fc875-tmp-dir\") pod \"node-resolver-ptgp6\" (UID: \"3d18f299-6f72-46ff-abf2-0ccf120fc875\") " pod="openshift-dns/node-resolver-ptgp6"
Jul 07 16:45:53.905178 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.903411    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f959ebb9-6e2d-48da-88b8-647a596e1413-run-openvswitch\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.905178 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.903830    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a1a8fd82-5f36-474b-a780-b30b1d5122ca-konnectivity-ca\") pod \"konnectivity-agent-gjbh8\" (UID: \"a1a8fd82-5f36-474b-a780-b30b1d5122ca\") " pod="kube-system/konnectivity-agent-gjbh8"
Jul 07 16:45:53.905844 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.904765    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f959ebb9-6e2d-48da-88b8-647a596e1413-ovn-node-metrics-cert\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:53.905844 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.904878    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a1a8fd82-5f36-474b-a780-b30b1d5122ca-agent-certs\") pod \"konnectivity-agent-gjbh8\" (UID: \"a1a8fd82-5f36-474b-a780-b30b1d5122ca\") " pod="kube-system/konnectivity-agent-gjbh8"
Jul 07 16:45:53.905844 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.905293    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16653221-598d-4229-a6d4-039bdaf5118d-tmp\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.908398 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.908363    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/16653221-598d-4229-a6d4-039bdaf5118d-etc-tuned\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.913685 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:53.913650    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 16:45:53.913685 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:53.913677    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 16:45:53.914708 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:53.913693    2584 projected.go:196] Error preparing data for projected volume kube-api-access-hppvt for pod openshift-network-diagnostics/network-check-target-l6pr5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:45:53.914708 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:53.913762    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c-kube-api-access-hppvt podName:0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c nodeName:}" failed. No retries permitted until 2026-07-07 16:45:54.413743081 +0000 UTC m=+3.014951781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hppvt" (UniqueName: "kubernetes.io/projected/0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c-kube-api-access-hppvt") pod "network-check-target-l6pr5" (UID: "0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:45:53.916256 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.916214    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p88kp\" (UniqueName: \"kubernetes.io/projected/3d18f299-6f72-46ff-abf2-0ccf120fc875-kube-api-access-p88kp\") pod \"node-resolver-ptgp6\" (UID: \"3d18f299-6f72-46ff-abf2-0ccf120fc875\") " pod="openshift-dns/node-resolver-ptgp6"
Jul 07 16:45:53.917195 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.917144    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79sxd\" (UniqueName: \"kubernetes.io/projected/16653221-598d-4229-a6d4-039bdaf5118d-kube-api-access-79sxd\") pod \"tuned-9jfs6\" (UID: \"16653221-598d-4229-a6d4-039bdaf5118d\") " pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:53.917940 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.917909    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnwqk\" (UniqueName: \"kubernetes.io/projected/f9035994-1125-405f-91db-cbe6f9c8af5a-kube-api-access-pnwqk\") pod \"aws-ebs-csi-driver-node-7sgl5\" (UID: \"f9035994-1125-405f-91db-cbe6f9c8af5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sgl5"
Jul 07 16:45:53.918714 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.918683    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6cpz\" (UniqueName: \"kubernetes.io/projected/73096d96-d52d-47ee-95a8-2242bcfb4888-kube-api-access-x6cpz\") pod \"multus-kdq9n\" (UID: \"73096d96-d52d-47ee-95a8-2242bcfb4888\") " pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:53.919342 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.919318    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59sbs\" (UniqueName: \"kubernetes.io/projected/7f7e543c-8cdd-4f28-adb5-570b6004c796-kube-api-access-59sbs\") pod \"network-metrics-daemon-9kj8c\" (UID: \"7f7e543c-8cdd-4f28-adb5-570b6004c796\") " pod="openshift-multus/network-metrics-daemon-9kj8c"
Jul 07 16:45:53.920303 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:53.920281    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmkv5\" (UniqueName: \"kubernetes.io/projected/f959ebb9-6e2d-48da-88b8-647a596e1413-kube-api-access-qmkv5\") pod \"ovnkube-node-qlfcs\" (UID: \"f959ebb9-6e2d-48da-88b8-647a596e1413\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:54.004530 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.004485    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts2dw\" (UniqueName: \"kubernetes.io/projected/a14a5b96-76aa-4d03-be55-8c23f342e4d9-kube-api-access-ts2dw\") pod \"iptables-alerter-tgj8c\" (UID: \"a14a5b96-76aa-4d03-be55-8c23f342e4d9\") " pod="openshift-network-operator/iptables-alerter-tgj8c"
Jul 07 16:45:54.004732 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.004541    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b4f517c-5cf2-440c-ad4a-bf3aa0934616-tuning-conf-dir\") pod \"multus-additional-cni-plugins-phn99\" (UID: \"3b4f517c-5cf2-440c-ad4a-bf3aa0934616\") " pod="openshift-multus/multus-additional-cni-plugins-phn99"
Jul 07 16:45:54.004732 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.004570    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/266c4079-c003-4676-a480-fb0e19a14cce-host\") pod \"node-ca-cpbvp\" (UID: \"266c4079-c003-4676-a480-fb0e19a14cce\") " pod="openshift-image-registry/node-ca-cpbvp"
Jul 07 16:45:54.004732 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.004598    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/266c4079-c003-4676-a480-fb0e19a14cce-serviceca\") pod \"node-ca-cpbvp\" (UID: \"266c4079-c003-4676-a480-fb0e19a14cce\") " pod="openshift-image-registry/node-ca-cpbvp"
Jul 07 16:45:54.004732 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.004640    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msm4n\" (UniqueName: \"kubernetes.io/projected/266c4079-c003-4676-a480-fb0e19a14cce-kube-api-access-msm4n\") pod \"node-ca-cpbvp\" (UID: \"266c4079-c003-4676-a480-fb0e19a14cce\") " pod="openshift-image-registry/node-ca-cpbvp"
Jul 07 16:45:54.004732 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.004670    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxdqb\" (UniqueName: \"kubernetes.io/projected/3b4f517c-5cf2-440c-ad4a-bf3aa0934616-kube-api-access-cxdqb\") pod \"multus-additional-cni-plugins-phn99\" (UID: \"3b4f517c-5cf2-440c-ad4a-bf3aa0934616\") " pod="openshift-multus/multus-additional-cni-plugins-phn99"
Jul 07 16:45:54.004732 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.004707    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3b4f517c-5cf2-440c-ad4a-bf3aa0934616-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-phn99\" (UID: \"3b4f517c-5cf2-440c-ad4a-bf3aa0934616\") " pod="openshift-multus/multus-additional-cni-plugins-phn99"
Jul 07 16:45:54.004732 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.004712    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/266c4079-c003-4676-a480-fb0e19a14cce-host\") pod \"node-ca-cpbvp\" (UID: \"266c4079-c003-4676-a480-fb0e19a14cce\") " pod="openshift-image-registry/node-ca-cpbvp"
Jul 07 16:45:54.005122 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.004755    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b2627fd1-997d-4ce5-84aa-e4b5b85cc307-kubelet-config\") pod \"global-pull-secret-syncer-2bkmf\" (UID: \"b2627fd1-997d-4ce5-84aa-e4b5b85cc307\") " pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:45:54.005122 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.004784    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b2627fd1-997d-4ce5-84aa-e4b5b85cc307-dbus\") pod \"global-pull-secret-syncer-2bkmf\" (UID: \"b2627fd1-997d-4ce5-84aa-e4b5b85cc307\") " pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:45:54.005122 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.004795    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b4f517c-5cf2-440c-ad4a-bf3aa0934616-tuning-conf-dir\") pod \"multus-additional-cni-plugins-phn99\" (UID: \"3b4f517c-5cf2-440c-ad4a-bf3aa0934616\") " pod="openshift-multus/multus-additional-cni-plugins-phn99"
Jul 07 16:45:54.005122 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.004811    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a14a5b96-76aa-4d03-be55-8c23f342e4d9-iptables-alerter-script\") pod \"iptables-alerter-tgj8c\" (UID: \"a14a5b96-76aa-4d03-be55-8c23f342e4d9\") " pod="openshift-network-operator/iptables-alerter-tgj8c"
Jul 07 16:45:54.005122 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.004842    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b4f517c-5cf2-440c-ad4a-bf3aa0934616-cnibin\") pod \"multus-additional-cni-plugins-phn99\" (UID: \"3b4f517c-5cf2-440c-ad4a-bf3aa0934616\") " pod="openshift-multus/multus-additional-cni-plugins-phn99"
Jul 07 16:45:54.005122 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.004863    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3b4f517c-5cf2-440c-ad4a-bf3aa0934616-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-phn99\" (UID: \"3b4f517c-5cf2-440c-ad4a-bf3aa0934616\") " pod="openshift-multus/multus-additional-cni-plugins-phn99"
Jul 07 16:45:54.005122 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.004888    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3b4f517c-5cf2-440c-ad4a-bf3aa0934616-os-release\") pod \"multus-additional-cni-plugins-phn99\" (UID: \"3b4f517c-5cf2-440c-ad4a-bf3aa0934616\") " pod="openshift-multus/multus-additional-cni-plugins-phn99"
Jul 07 16:45:54.005122 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.004929    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b4f517c-5cf2-440c-ad4a-bf3aa0934616-system-cni-dir\") pod \"multus-additional-cni-plugins-phn99\" (UID: \"3b4f517c-5cf2-440c-ad4a-bf3aa0934616\") " pod="openshift-multus/multus-additional-cni-plugins-phn99"
Jul 07 16:45:54.005122 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.004945    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b4f517c-5cf2-440c-ad4a-bf3aa0934616-cni-binary-copy\") pod \"multus-additional-cni-plugins-phn99\" (UID: \"3b4f517c-5cf2-440c-ad4a-bf3aa0934616\") " pod="openshift-multus/multus-additional-cni-plugins-phn99"
Jul 07 16:45:54.005122 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.004961    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2627fd1-997d-4ce5-84aa-e4b5b85cc307-original-pull-secret\") pod \"global-pull-secret-syncer-2bkmf\" (UID: \"b2627fd1-997d-4ce5-84aa-e4b5b85cc307\") " pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:45:54.005122 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.004982    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a14a5b96-76aa-4d03-be55-8c23f342e4d9-host-slash\") pod \"iptables-alerter-tgj8c\" (UID: \"a14a5b96-76aa-4d03-be55-8c23f342e4d9\") " pod="openshift-network-operator/iptables-alerter-tgj8c"
Jul 07 16:45:54.005122 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.005050    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a14a5b96-76aa-4d03-be55-8c23f342e4d9-host-slash\") pod \"iptables-alerter-tgj8c\" (UID: \"a14a5b96-76aa-4d03-be55-8c23f342e4d9\") " pod="openshift-network-operator/iptables-alerter-tgj8c"
Jul 07 16:45:54.005122 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.005052    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b2627fd1-997d-4ce5-84aa-e4b5b85cc307-kubelet-config\") pod \"global-pull-secret-syncer-2bkmf\" (UID: \"b2627fd1-997d-4ce5-84aa-e4b5b85cc307\") " pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:45:54.005717 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.005140    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b4f517c-5cf2-440c-ad4a-bf3aa0934616-cnibin\") pod \"multus-additional-cni-plugins-phn99\" (UID: \"3b4f517c-5cf2-440c-ad4a-bf3aa0934616\") " pod="openshift-multus/multus-additional-cni-plugins-phn99"
Jul 07 16:45:54.005717 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.005182    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3b4f517c-5cf2-440c-ad4a-bf3aa0934616-os-release\") pod \"multus-additional-cni-plugins-phn99\" (UID: \"3b4f517c-5cf2-440c-ad4a-bf3aa0934616\") " pod="openshift-multus/multus-additional-cni-plugins-phn99"
Jul 07 16:45:54.005717 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.005190    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/266c4079-c003-4676-a480-fb0e19a14cce-serviceca\") pod \"node-ca-cpbvp\" (UID: \"266c4079-c003-4676-a480-fb0e19a14cce\") " pod="openshift-image-registry/node-ca-cpbvp"
Jul 07 16:45:54.005717 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.005254    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b4f517c-5cf2-440c-ad4a-bf3aa0934616-system-cni-dir\") pod \"multus-additional-cni-plugins-phn99\" (UID: \"3b4f517c-5cf2-440c-ad4a-bf3aa0934616\") " pod="openshift-multus/multus-additional-cni-plugins-phn99"
Jul 07 16:45:54.005717 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.005316    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3b4f517c-5cf2-440c-ad4a-bf3aa0934616-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-phn99\" (UID: \"3b4f517c-5cf2-440c-ad4a-bf3aa0934616\") " pod="openshift-multus/multus-additional-cni-plugins-phn99"
Jul 07 16:45:54.005717 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.005352    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b2627fd1-997d-4ce5-84aa-e4b5b85cc307-dbus\") pod \"global-pull-secret-syncer-2bkmf\" (UID: \"b2627fd1-997d-4ce5-84aa-e4b5b85cc307\") " pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:45:54.005717 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:54.005386    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 16:45:54.005717 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:54.005450    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2627fd1-997d-4ce5-84aa-e4b5b85cc307-original-pull-secret podName:b2627fd1-997d-4ce5-84aa-e4b5b85cc307 nodeName:}" failed. No retries permitted until 2026-07-07 16:45:54.505430757 +0000 UTC m=+3.106639446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2627fd1-997d-4ce5-84aa-e4b5b85cc307-original-pull-secret") pod "global-pull-secret-syncer-2bkmf" (UID: "b2627fd1-997d-4ce5-84aa-e4b5b85cc307") : object "kube-system"/"original-pull-secret" not registered
Jul 07 16:45:54.005717 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.005487    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a14a5b96-76aa-4d03-be55-8c23f342e4d9-iptables-alerter-script\") pod \"iptables-alerter-tgj8c\" (UID: \"a14a5b96-76aa-4d03-be55-8c23f342e4d9\") " pod="openshift-network-operator/iptables-alerter-tgj8c"
Jul 07 16:45:54.005717 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.005694    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3b4f517c-5cf2-440c-ad4a-bf3aa0934616-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-phn99\" (UID: \"3b4f517c-5cf2-440c-ad4a-bf3aa0934616\") " pod="openshift-multus/multus-additional-cni-plugins-phn99"
Jul 07 16:45:54.006116 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.006096    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b4f517c-5cf2-440c-ad4a-bf3aa0934616-cni-binary-copy\") pod \"multus-additional-cni-plugins-phn99\" (UID: \"3b4f517c-5cf2-440c-ad4a-bf3aa0934616\") " pod="openshift-multus/multus-additional-cni-plugins-phn99"
Jul 07 16:45:54.025659 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.025579    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msm4n\" (UniqueName: \"kubernetes.io/projected/266c4079-c003-4676-a480-fb0e19a14cce-kube-api-access-msm4n\") pod \"node-ca-cpbvp\" (UID: \"266c4079-c003-4676-a480-fb0e19a14cce\") " pod="openshift-image-registry/node-ca-cpbvp"
Jul 07 16:45:54.026009 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.025985    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxdqb\" (UniqueName: \"kubernetes.io/projected/3b4f517c-5cf2-440c-ad4a-bf3aa0934616-kube-api-access-cxdqb\") pod \"multus-additional-cni-plugins-phn99\" (UID: \"3b4f517c-5cf2-440c-ad4a-bf3aa0934616\") " pod="openshift-multus/multus-additional-cni-plugins-phn99"
Jul 07 16:45:54.026908 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.026886    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts2dw\" (UniqueName: \"kubernetes.io/projected/a14a5b96-76aa-4d03-be55-8c23f342e4d9-kube-api-access-ts2dw\") pod \"iptables-alerter-tgj8c\" (UID: \"a14a5b96-76aa-4d03-be55-8c23f342e4d9\") " pod="openshift-network-operator/iptables-alerter-tgj8c"
Jul 07 16:45:54.090231 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.090195    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:45:54.101559 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.101518    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ptgp6"
Jul 07 16:45:54.110652 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.110628    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kdq9n"
Jul 07 16:45:54.119596 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.119570    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9jfs6"
Jul 07 16:45:54.124775 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.124748    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sgl5"
Jul 07 16:45:54.131170 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.131142    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gjbh8"
Jul 07 16:45:54.140620 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.140596    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cpbvp"
Jul 07 16:45:54.146278 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.146255    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tgj8c"
Jul 07 16:45:54.151511 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.151491    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-phn99"
Jul 07 16:45:54.248134 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.248104    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 16:45:54.409017 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.408974    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f7e543c-8cdd-4f28-adb5-570b6004c796-metrics-certs\") pod \"network-metrics-daemon-9kj8c\" (UID: \"7f7e543c-8cdd-4f28-adb5-570b6004c796\") " pod="openshift-multus/network-metrics-daemon-9kj8c"
Jul 07 16:45:54.409217 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:54.409187    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:45:54.409276 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:54.409267    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f7e543c-8cdd-4f28-adb5-570b6004c796-metrics-certs podName:7f7e543c-8cdd-4f28-adb5-570b6004c796 nodeName:}" failed. No retries permitted until 2026-07-07 16:45:55.409245074 +0000 UTC m=+4.010453777 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f7e543c-8cdd-4f28-adb5-570b6004c796-metrics-certs") pod "network-metrics-daemon-9kj8c" (UID: "7f7e543c-8cdd-4f28-adb5-570b6004c796") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:45:54.509986 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.509938    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hppvt\" (UniqueName: \"kubernetes.io/projected/0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c-kube-api-access-hppvt\") pod \"network-check-target-l6pr5\" (UID: \"0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c\") " pod="openshift-network-diagnostics/network-check-target-l6pr5"
Jul 07 16:45:54.510202 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.510015    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2627fd1-997d-4ce5-84aa-e4b5b85cc307-original-pull-secret\") pod \"global-pull-secret-syncer-2bkmf\" (UID: \"b2627fd1-997d-4ce5-84aa-e4b5b85cc307\") " pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:45:54.510202 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:54.510170    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 16:45:54.510202 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:54.510182    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 16:45:54.510202 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:54.510194    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 16:45:54.510366 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:54.510210    2584 projected.go:196] Error preparing data for projected volume kube-api-access-hppvt for pod openshift-network-diagnostics/network-check-target-l6pr5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:45:54.510366 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:54.510266    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2627fd1-997d-4ce5-84aa-e4b5b85cc307-original-pull-secret podName:b2627fd1-997d-4ce5-84aa-e4b5b85cc307 nodeName:}" failed. No retries permitted until 2026-07-07 16:45:55.510242322 +0000 UTC m=+4.111451021 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2627fd1-997d-4ce5-84aa-e4b5b85cc307-original-pull-secret") pod "global-pull-secret-syncer-2bkmf" (UID: "b2627fd1-997d-4ce5-84aa-e4b5b85cc307") : object "kube-system"/"original-pull-secret" not registered
Jul 07 16:45:54.510366 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:54.510287    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c-kube-api-access-hppvt podName:0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c nodeName:}" failed. No retries permitted until 2026-07-07 16:45:55.510277668 +0000 UTC m=+4.111486357 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hppvt" (UniqueName: "kubernetes.io/projected/0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c-kube-api-access-hppvt") pod "network-check-target-l6pr5" (UID: "0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:45:54.743842 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:54.743765    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d18f299_6f72_46ff_abf2_0ccf120fc875.slice/crio-1bb7e0fcc205465f98537f9578f8ad3fdd74093b9e4920add08599380a349f38 WatchSource:0}: Error finding container 1bb7e0fcc205465f98537f9578f8ad3fdd74093b9e4920add08599380a349f38: Status 404 returned error can't find the container with id 1bb7e0fcc205465f98537f9578f8ad3fdd74093b9e4920add08599380a349f38
Jul 07 16:45:54.744968 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:54.744926    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b4f517c_5cf2_440c_ad4a_bf3aa0934616.slice/crio-90259c2bb2c76ac5a5a0aecc508de0e17bd3775e1a16abda6b2c897076a592d4 WatchSource:0}: Error finding container 90259c2bb2c76ac5a5a0aecc508de0e17bd3775e1a16abda6b2c897076a592d4: Status 404 returned error can't find the container with id 90259c2bb2c76ac5a5a0aecc508de0e17bd3775e1a16abda6b2c897076a592d4
Jul 07 16:45:54.746848 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:54.746784    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73096d96_d52d_47ee_95a8_2242bcfb4888.slice/crio-60e135a5a0e6abcce4e73c813a4d23ce4e63844c49042880208e2aa0f3940b37 WatchSource:0}: Error finding container 60e135a5a0e6abcce4e73c813a4d23ce4e63844c49042880208e2aa0f3940b37: Status 404 returned error can't find the container with id 60e135a5a0e6abcce4e73c813a4d23ce4e63844c49042880208e2aa0f3940b37
Jul 07 16:45:54.749331 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:54.749297    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16653221_598d_4229_a6d4_039bdaf5118d.slice/crio-140c121d7e14e650cf735f6758638f1e623d8f33e23d0b584563f6196f49e964 WatchSource:0}: Error finding container 140c121d7e14e650cf735f6758638f1e623d8f33e23d0b584563f6196f49e964: Status 404 returned error can't find the container with id 140c121d7e14e650cf735f6758638f1e623d8f33e23d0b584563f6196f49e964
Jul 07 16:45:54.750278 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:54.750254    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf959ebb9_6e2d_48da_88b8_647a596e1413.slice/crio-4cd94dc9ad0db0a03aa0021560770c7615f79eb8d02af8bae63ae0bdc8c567a9 WatchSource:0}: Error finding container 4cd94dc9ad0db0a03aa0021560770c7615f79eb8d02af8bae63ae0bdc8c567a9: Status 404 returned error can't find the container with id 4cd94dc9ad0db0a03aa0021560770c7615f79eb8d02af8bae63ae0bdc8c567a9
Jul 07 16:45:54.751723 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:54.751698    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a8fd82_5f36_474b_a780_b30b1d5122ca.slice/crio-a48f7cfcc9b8a2fa15252dc556b2cad37e7560d8808690884c050e58ce32cde7 WatchSource:0}: Error finding container a48f7cfcc9b8a2fa15252dc556b2cad37e7560d8808690884c050e58ce32cde7: Status 404 returned error can't find the container with id a48f7cfcc9b8a2fa15252dc556b2cad37e7560d8808690884c050e58ce32cde7
Jul 07 16:45:54.752684 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:54.752660    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda14a5b96_76aa_4d03_be55_8c23f342e4d9.slice/crio-7374dc999eec48ba5d744334df93d4863953791872e023e7bd485f7ffa215b40 WatchSource:0}: Error finding container 7374dc999eec48ba5d744334df93d4863953791872e023e7bd485f7ffa215b40: Status 404 returned error can't find the container with id 7374dc999eec48ba5d744334df93d4863953791872e023e7bd485f7ffa215b40
Jul 07 16:45:54.753898 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:54.753877    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9035994_1125_405f_91db_cbe6f9c8af5a.slice/crio-d19e148d5e99f0bf1d18d3643c4ef8359d4d7d641b72569990b4f80c2fa3f453 WatchSource:0}: Error finding container d19e148d5e99f0bf1d18d3643c4ef8359d4d7d641b72569990b4f80c2fa3f453: Status 404 returned error can't find the container with id d19e148d5e99f0bf1d18d3643c4ef8359d4d7d641b72569990b4f80c2fa3f453
Jul 07 16:45:54.757328 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:45:54.757279    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod266c4079_c003_4676_a480_fb0e19a14cce.slice/crio-5024050f54fb434a43573eb448a55b54c45076c640e980d6696979971f49939e WatchSource:0}: Error finding container 5024050f54fb434a43573eb448a55b54c45076c640e980d6696979971f49939e: Status 404 returned error can't find the container with id 5024050f54fb434a43573eb448a55b54c45076c640e980d6696979971f49939e
Jul 07 16:45:54.852486 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.852257    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 16:40:54 +0000 UTC" deadline="2028-04-19 21:14:03.84234621 +0000 UTC"
Jul 07 16:45:54.852486 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.852487    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15652h28m8.989866523s"
Jul 07 16:45:54.888635 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.888600    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9kj8c"
Jul 07 16:45:54.888823 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:54.888718    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9kj8c" podUID="7f7e543c-8cdd-4f28-adb5-570b6004c796"
Jul 07 16:45:54.905139 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.905102    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9jfs6" event={"ID":"16653221-598d-4229-a6d4-039bdaf5118d","Type":"ContainerStarted","Data":"140c121d7e14e650cf735f6758638f1e623d8f33e23d0b584563f6196f49e964"}
Jul 07 16:45:54.906294 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.906263    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kdq9n" event={"ID":"73096d96-d52d-47ee-95a8-2242bcfb4888","Type":"ContainerStarted","Data":"60e135a5a0e6abcce4e73c813a4d23ce4e63844c49042880208e2aa0f3940b37"}
Jul 07 16:45:54.908209 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.908182    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-184.ec2.internal" event={"ID":"47045ed184587f2ba81f9c13a0f2aefa","Type":"ContainerStarted","Data":"485f34a51d51b631e3a178fc9ee91ab5c000aced4aec4d8c34d50cba1d3a30d3"}
Jul 07 16:45:54.909310 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.909280    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cpbvp" event={"ID":"266c4079-c003-4676-a480-fb0e19a14cce","Type":"ContainerStarted","Data":"5024050f54fb434a43573eb448a55b54c45076c640e980d6696979971f49939e"}
Jul 07 16:45:54.910463 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.910432    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs" event={"ID":"f959ebb9-6e2d-48da-88b8-647a596e1413","Type":"ContainerStarted","Data":"4cd94dc9ad0db0a03aa0021560770c7615f79eb8d02af8bae63ae0bdc8c567a9"}
Jul 07 16:45:54.911649 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.911627    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sgl5" event={"ID":"f9035994-1125-405f-91db-cbe6f9c8af5a","Type":"ContainerStarted","Data":"d19e148d5e99f0bf1d18d3643c4ef8359d4d7d641b72569990b4f80c2fa3f453"}
Jul 07 16:45:54.912781 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.912753    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tgj8c" event={"ID":"a14a5b96-76aa-4d03-be55-8c23f342e4d9","Type":"ContainerStarted","Data":"7374dc999eec48ba5d744334df93d4863953791872e023e7bd485f7ffa215b40"}
Jul 07 16:45:54.913775 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.913752    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phn99" event={"ID":"3b4f517c-5cf2-440c-ad4a-bf3aa0934616","Type":"ContainerStarted","Data":"90259c2bb2c76ac5a5a0aecc508de0e17bd3775e1a16abda6b2c897076a592d4"}
Jul 07 16:45:54.914897 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.914879    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ptgp6" event={"ID":"3d18f299-6f72-46ff-abf2-0ccf120fc875","Type":"ContainerStarted","Data":"1bb7e0fcc205465f98537f9578f8ad3fdd74093b9e4920add08599380a349f38"}
Jul 07 16:45:54.920497 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.920470    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gjbh8" event={"ID":"a1a8fd82-5f36-474b-a780-b30b1d5122ca","Type":"ContainerStarted","Data":"a48f7cfcc9b8a2fa15252dc556b2cad37e7560d8808690884c050e58ce32cde7"}
Jul 07 16:45:54.933358 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:54.933304    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-184.ec2.internal" podStartSLOduration=1.9332874009999998 podStartE2EDuration="1.933287401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:45:54.93278781 +0000 UTC m=+3.533996518" watchObservedRunningTime="2026-07-07 16:45:54.933287401 +0000 UTC m=+3.534496115"
Jul 07 16:45:55.417203 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:55.416533    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f7e543c-8cdd-4f28-adb5-570b6004c796-metrics-certs\") pod \"network-metrics-daemon-9kj8c\" (UID: \"7f7e543c-8cdd-4f28-adb5-570b6004c796\") " pod="openshift-multus/network-metrics-daemon-9kj8c"
Jul 07 16:45:55.417203 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:55.416717    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:45:55.417203 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:55.416788    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f7e543c-8cdd-4f28-adb5-570b6004c796-metrics-certs podName:7f7e543c-8cdd-4f28-adb5-570b6004c796 nodeName:}" failed. No retries permitted until 2026-07-07 16:45:57.416766826 +0000 UTC m=+6.017975532 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f7e543c-8cdd-4f28-adb5-570b6004c796-metrics-certs") pod "network-metrics-daemon-9kj8c" (UID: "7f7e543c-8cdd-4f28-adb5-570b6004c796") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:45:55.517730 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:55.516979    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2627fd1-997d-4ce5-84aa-e4b5b85cc307-original-pull-secret\") pod \"global-pull-secret-syncer-2bkmf\" (UID: \"b2627fd1-997d-4ce5-84aa-e4b5b85cc307\") " pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:45:55.517730 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:55.517275    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 16:45:55.517730 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:55.517343    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2627fd1-997d-4ce5-84aa-e4b5b85cc307-original-pull-secret podName:b2627fd1-997d-4ce5-84aa-e4b5b85cc307 nodeName:}" failed. No retries permitted until 2026-07-07 16:45:57.517323825 +0000 UTC m=+6.118532513 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2627fd1-997d-4ce5-84aa-e4b5b85cc307-original-pull-secret") pod "global-pull-secret-syncer-2bkmf" (UID: "b2627fd1-997d-4ce5-84aa-e4b5b85cc307") : object "kube-system"/"original-pull-secret" not registered
Jul 07 16:45:55.517730 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:55.517368    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 16:45:55.517730 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:55.517395    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 16:45:55.517730 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:55.517410    2584 projected.go:196] Error preparing data for projected volume kube-api-access-hppvt for pod openshift-network-diagnostics/network-check-target-l6pr5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:45:55.517730 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:55.517465    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c-kube-api-access-hppvt podName:0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c nodeName:}" failed. No retries permitted until 2026-07-07 16:45:57.517445351 +0000 UTC m=+6.118654038 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hppvt" (UniqueName: "kubernetes.io/projected/0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c-kube-api-access-hppvt") pod "network-check-target-l6pr5" (UID: "0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:45:55.517730 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:55.517541    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hppvt\" (UniqueName: \"kubernetes.io/projected/0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c-kube-api-access-hppvt\") pod \"network-check-target-l6pr5\" (UID: \"0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c\") " pod="openshift-network-diagnostics/network-check-target-l6pr5"
Jul 07 16:45:55.852879 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:55.852797    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 16:40:54 +0000 UTC" deadline="2027-12-26 11:06:55.592583839 +0000 UTC"
Jul 07 16:45:55.852879 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:55.852853    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12882h20m59.739735308s"
Jul 07 16:45:55.889317 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:55.889278    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6pr5"
Jul 07 16:45:55.889490 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:55.889421    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6pr5" podUID="0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c"
Jul 07 16:45:55.890100 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:55.889921    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:45:55.890100 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:55.890024    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bkmf" podUID="b2627fd1-997d-4ce5-84aa-e4b5b85cc307"
Jul 07 16:45:55.942656 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:55.941944    2584 generic.go:356] "Generic (PLEG): container finished" podID="d26d465c12efb451fe9dcf0c005e8db7" containerID="5bf2ca91e029054f00ff7e2c7af985cef87cbdc2c6b3a118d1ccc93af0b6b698" exitCode=0
Jul 07 16:45:55.942656 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:55.942483    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-184.ec2.internal" event={"ID":"d26d465c12efb451fe9dcf0c005e8db7","Type":"ContainerDied","Data":"5bf2ca91e029054f00ff7e2c7af985cef87cbdc2c6b3a118d1ccc93af0b6b698"}
Jul 07 16:45:56.888518 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:56.888480    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9kj8c"
Jul 07 16:45:56.889066 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:56.888637    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9kj8c" podUID="7f7e543c-8cdd-4f28-adb5-570b6004c796"
Jul 07 16:45:56.970227 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:56.970172    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-184.ec2.internal" event={"ID":"d26d465c12efb451fe9dcf0c005e8db7","Type":"ContainerStarted","Data":"6cec61ccf097ab1e22e380ab8f7b3fcb257c4af824483bfdae2259ef2be5b969"}
Jul 07 16:45:56.991378 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:56.991283    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-184.ec2.internal" podStartSLOduration=3.991260073 podStartE2EDuration="3.991260073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:45:56.990948044 +0000 UTC m=+5.592156753" watchObservedRunningTime="2026-07-07 16:45:56.991260073 +0000 UTC m=+5.592468799"
Jul 07 16:45:57.434914 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:57.434866    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f7e543c-8cdd-4f28-adb5-570b6004c796-metrics-certs\") pod \"network-metrics-daemon-9kj8c\" (UID: \"7f7e543c-8cdd-4f28-adb5-570b6004c796\") " pod="openshift-multus/network-metrics-daemon-9kj8c"
Jul 07 16:45:57.435154 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:57.435047    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:45:57.435154 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:57.435151    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f7e543c-8cdd-4f28-adb5-570b6004c796-metrics-certs podName:7f7e543c-8cdd-4f28-adb5-570b6004c796 nodeName:}" failed. No retries permitted until 2026-07-07 16:46:01.435128839 +0000 UTC m=+10.036337524 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f7e543c-8cdd-4f28-adb5-570b6004c796-metrics-certs") pod "network-metrics-daemon-9kj8c" (UID: "7f7e543c-8cdd-4f28-adb5-570b6004c796") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:45:57.536457 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:57.536394    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hppvt\" (UniqueName: \"kubernetes.io/projected/0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c-kube-api-access-hppvt\") pod \"network-check-target-l6pr5\" (UID: \"0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c\") " pod="openshift-network-diagnostics/network-check-target-l6pr5"
Jul 07 16:45:57.536654 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:57.536527    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2627fd1-997d-4ce5-84aa-e4b5b85cc307-original-pull-secret\") pod \"global-pull-secret-syncer-2bkmf\" (UID: \"b2627fd1-997d-4ce5-84aa-e4b5b85cc307\") " pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:45:57.536726 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:57.536689    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 16:45:57.536788 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:57.536780    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2627fd1-997d-4ce5-84aa-e4b5b85cc307-original-pull-secret podName:b2627fd1-997d-4ce5-84aa-e4b5b85cc307 nodeName:}" failed. No retries permitted until 2026-07-07 16:46:01.536740315 +0000 UTC m=+10.137949005 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2627fd1-997d-4ce5-84aa-e4b5b85cc307-original-pull-secret") pod "global-pull-secret-syncer-2bkmf" (UID: "b2627fd1-997d-4ce5-84aa-e4b5b85cc307") : object "kube-system"/"original-pull-secret" not registered
Jul 07 16:45:57.536939 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:57.536923    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 16:45:57.537015 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:57.536946    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 16:45:57.537015 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:57.536962    2584 projected.go:196] Error preparing data for projected volume kube-api-access-hppvt for pod openshift-network-diagnostics/network-check-target-l6pr5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:45:57.537015 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:57.537007    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c-kube-api-access-hppvt podName:0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c nodeName:}" failed. No retries permitted until 2026-07-07 16:46:01.536990628 +0000 UTC m=+10.138199318 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hppvt" (UniqueName: "kubernetes.io/projected/0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c-kube-api-access-hppvt") pod "network-check-target-l6pr5" (UID: "0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:45:57.890392 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:57.890359    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6pr5"
Jul 07 16:45:57.890906 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:57.890488    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6pr5" podUID="0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c"
Jul 07 16:45:57.891745 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:57.891467    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:45:57.891745 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:57.891586    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bkmf" podUID="b2627fd1-997d-4ce5-84aa-e4b5b85cc307"
Jul 07 16:45:58.887729 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:58.887689    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9kj8c"
Jul 07 16:45:58.887921 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:58.887859    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9kj8c" podUID="7f7e543c-8cdd-4f28-adb5-570b6004c796"
Jul 07 16:45:59.889223 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:59.889185    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6pr5"
Jul 07 16:45:59.889802 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:45:59.889182    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:45:59.889802 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:59.889329    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6pr5" podUID="0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c"
Jul 07 16:45:59.889802 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:45:59.889415    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bkmf" podUID="b2627fd1-997d-4ce5-84aa-e4b5b85cc307"
Jul 07 16:46:00.888226 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:00.888186    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9kj8c"
Jul 07 16:46:00.888409 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:00.888319    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9kj8c" podUID="7f7e543c-8cdd-4f28-adb5-570b6004c796"
Jul 07 16:46:01.480063 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:01.480006    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f7e543c-8cdd-4f28-adb5-570b6004c796-metrics-certs\") pod \"network-metrics-daemon-9kj8c\" (UID: \"7f7e543c-8cdd-4f28-adb5-570b6004c796\") " pod="openshift-multus/network-metrics-daemon-9kj8c"
Jul 07 16:46:01.480506 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:01.480157    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:46:01.480506 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:01.480221    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f7e543c-8cdd-4f28-adb5-570b6004c796-metrics-certs podName:7f7e543c-8cdd-4f28-adb5-570b6004c796 nodeName:}" failed. No retries permitted until 2026-07-07 16:46:09.480204319 +0000 UTC m=+18.081413017 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f7e543c-8cdd-4f28-adb5-570b6004c796-metrics-certs") pod "network-metrics-daemon-9kj8c" (UID: "7f7e543c-8cdd-4f28-adb5-570b6004c796") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:46:01.580409 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:01.580369    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2627fd1-997d-4ce5-84aa-e4b5b85cc307-original-pull-secret\") pod \"global-pull-secret-syncer-2bkmf\" (UID: \"b2627fd1-997d-4ce5-84aa-e4b5b85cc307\") " pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:46:01.580656 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:01.580471    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hppvt\" (UniqueName: \"kubernetes.io/projected/0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c-kube-api-access-hppvt\") pod \"network-check-target-l6pr5\" (UID: \"0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c\") " pod="openshift-network-diagnostics/network-check-target-l6pr5"
Jul 07 16:46:01.580656 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:01.580534    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 16:46:01.580656 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:01.580596    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 16:46:01.580656 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:01.580609    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 16:46:01.580656 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:01.580622    2584 projected.go:196] Error preparing data for projected volume kube-api-access-hppvt for pod openshift-network-diagnostics/network-check-target-l6pr5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:46:01.580656 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:01.580610    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2627fd1-997d-4ce5-84aa-e4b5b85cc307-original-pull-secret podName:b2627fd1-997d-4ce5-84aa-e4b5b85cc307 nodeName:}" failed. No retries permitted until 2026-07-07 16:46:09.58059212 +0000 UTC m=+18.181800808 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2627fd1-997d-4ce5-84aa-e4b5b85cc307-original-pull-secret") pod "global-pull-secret-syncer-2bkmf" (UID: "b2627fd1-997d-4ce5-84aa-e4b5b85cc307") : object "kube-system"/"original-pull-secret" not registered
Jul 07 16:46:01.580877 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:01.580670    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c-kube-api-access-hppvt podName:0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c nodeName:}" failed. No retries permitted until 2026-07-07 16:46:09.580658644 +0000 UTC m=+18.181867328 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hppvt" (UniqueName: "kubernetes.io/projected/0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c-kube-api-access-hppvt") pod "network-check-target-l6pr5" (UID: "0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:46:01.890082 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:01.889589    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:46:01.890082 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:01.889716    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bkmf" podUID="b2627fd1-997d-4ce5-84aa-e4b5b85cc307"
Jul 07 16:46:01.890082 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:01.889823    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6pr5"
Jul 07 16:46:01.890082 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:01.889969    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6pr5" podUID="0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c"
Jul 07 16:46:02.888321 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:02.888142    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9kj8c"
Jul 07 16:46:02.888321 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:02.888276    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9kj8c" podUID="7f7e543c-8cdd-4f28-adb5-570b6004c796"
Jul 07 16:46:03.888224 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:03.888193    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:46:03.888402 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:03.888328    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bkmf" podUID="b2627fd1-997d-4ce5-84aa-e4b5b85cc307"
Jul 07 16:46:03.890644 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:03.890373    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6pr5"
Jul 07 16:46:03.890644 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:03.890485    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6pr5" podUID="0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c"
Jul 07 16:46:04.888255 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:04.888219    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9kj8c"
Jul 07 16:46:04.888429 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:04.888379    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9kj8c" podUID="7f7e543c-8cdd-4f28-adb5-570b6004c796"
Jul 07 16:46:05.888562 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:05.888514    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:46:05.889006 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:05.888573    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6pr5"
Jul 07 16:46:05.889006 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:05.888682    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bkmf" podUID="b2627fd1-997d-4ce5-84aa-e4b5b85cc307"
Jul 07 16:46:05.889006 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:05.888791    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6pr5" podUID="0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c"
Jul 07 16:46:06.888115 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:06.888081    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9kj8c"
Jul 07 16:46:06.888309 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:06.888200    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9kj8c" podUID="7f7e543c-8cdd-4f28-adb5-570b6004c796"
Jul 07 16:46:07.887898 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:07.887856    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:46:07.887898 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:07.887887    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6pr5"
Jul 07 16:46:07.888443 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:07.887983    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6pr5" podUID="0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c"
Jul 07 16:46:07.888443 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:07.888128    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bkmf" podUID="b2627fd1-997d-4ce5-84aa-e4b5b85cc307"
Jul 07 16:46:08.887586 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:08.887547    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9kj8c"
Jul 07 16:46:08.887832 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:08.887697    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9kj8c" podUID="7f7e543c-8cdd-4f28-adb5-570b6004c796"
Jul 07 16:46:09.544784 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:09.544742    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f7e543c-8cdd-4f28-adb5-570b6004c796-metrics-certs\") pod \"network-metrics-daemon-9kj8c\" (UID: \"7f7e543c-8cdd-4f28-adb5-570b6004c796\") " pod="openshift-multus/network-metrics-daemon-9kj8c"
Jul 07 16:46:09.545377 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:09.544923    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:46:09.545377 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:09.544993    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f7e543c-8cdd-4f28-adb5-570b6004c796-metrics-certs podName:7f7e543c-8cdd-4f28-adb5-570b6004c796 nodeName:}" failed. No retries permitted until 2026-07-07 16:46:25.544971929 +0000 UTC m=+34.146180631 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f7e543c-8cdd-4f28-adb5-570b6004c796-metrics-certs") pod "network-metrics-daemon-9kj8c" (UID: "7f7e543c-8cdd-4f28-adb5-570b6004c796") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:46:09.645945 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:09.645914    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hppvt\" (UniqueName: \"kubernetes.io/projected/0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c-kube-api-access-hppvt\") pod \"network-check-target-l6pr5\" (UID: \"0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c\") " pod="openshift-network-diagnostics/network-check-target-l6pr5"
Jul 07 16:46:09.646135 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:09.645970    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2627fd1-997d-4ce5-84aa-e4b5b85cc307-original-pull-secret\") pod \"global-pull-secret-syncer-2bkmf\" (UID: \"b2627fd1-997d-4ce5-84aa-e4b5b85cc307\") " pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:46:09.646135 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:09.646094    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 16:46:09.646135 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:09.646098    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 16:46:09.646135 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:09.646118    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 16:46:09.646135 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:09.646131    2584 projected.go:196] Error preparing data for projected volume kube-api-access-hppvt for pod openshift-network-diagnostics/network-check-target-l6pr5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:46:09.646294 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:09.646151    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2627fd1-997d-4ce5-84aa-e4b5b85cc307-original-pull-secret podName:b2627fd1-997d-4ce5-84aa-e4b5b85cc307 nodeName:}" failed. No retries permitted until 2026-07-07 16:46:25.646137436 +0000 UTC m=+34.247346119 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2627fd1-997d-4ce5-84aa-e4b5b85cc307-original-pull-secret") pod "global-pull-secret-syncer-2bkmf" (UID: "b2627fd1-997d-4ce5-84aa-e4b5b85cc307") : object "kube-system"/"original-pull-secret" not registered
Jul 07 16:46:09.646294 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:09.646176    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c-kube-api-access-hppvt podName:0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c nodeName:}" failed. No retries permitted until 2026-07-07 16:46:25.646164203 +0000 UTC m=+34.247372902 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hppvt" (UniqueName: "kubernetes.io/projected/0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c-kube-api-access-hppvt") pod "network-check-target-l6pr5" (UID: "0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:46:09.888625 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:09.888196    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:46:09.888625 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:09.888293    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6pr5"
Jul 07 16:46:09.888625 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:09.888316    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bkmf" podUID="b2627fd1-997d-4ce5-84aa-e4b5b85cc307"
Jul 07 16:46:09.888625 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:09.888395    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6pr5" podUID="0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c"
Jul 07 16:46:10.888058 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:10.888008    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9kj8c"
Jul 07 16:46:10.888541 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:10.888155    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9kj8c" podUID="7f7e543c-8cdd-4f28-adb5-570b6004c796"
Jul 07 16:46:11.889034 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:11.888996    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6pr5"
Jul 07 16:46:11.889467 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:11.889103    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:46:11.889467 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:11.889137    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6pr5" podUID="0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c"
Jul 07 16:46:11.889467 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:11.889200    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bkmf" podUID="b2627fd1-997d-4ce5-84aa-e4b5b85cc307"
Jul 07 16:46:12.887781 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:12.887737    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9kj8c"
Jul 07 16:46:12.887958 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:12.887885    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9kj8c" podUID="7f7e543c-8cdd-4f28-adb5-570b6004c796"
Jul 07 16:46:13.888722 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:13.888230    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:46:13.888722 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:13.888369    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bkmf" podUID="b2627fd1-997d-4ce5-84aa-e4b5b85cc307"
Jul 07 16:46:13.888722 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:13.888427    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6pr5"
Jul 07 16:46:13.888722 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:13.888526    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6pr5" podUID="0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c"
Jul 07 16:46:14.888458 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:14.888146    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9kj8c"
Jul 07 16:46:14.888648 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:14.888564    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9kj8c" podUID="7f7e543c-8cdd-4f28-adb5-570b6004c796"
Jul 07 16:46:15.010944 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:15.010917    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cpbvp" event={"ID":"266c4079-c003-4676-a480-fb0e19a14cce","Type":"ContainerStarted","Data":"be8858edd04e7c5536fc7f51ea1a4f341fd719b6fc978aca174739e1a3cdfe83"}
Jul 07 16:46:15.013465 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:15.013444    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs" event={"ID":"f959ebb9-6e2d-48da-88b8-647a596e1413","Type":"ContainerStarted","Data":"625da638765c1f78d33c1f301b4aac9c09c008a842fb177cc54ba807d18f964b"}
Jul 07 16:46:15.013548 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:15.013508    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs" event={"ID":"f959ebb9-6e2d-48da-88b8-647a596e1413","Type":"ContainerStarted","Data":"353c035ac7b24eada9da6e64451d789f90a9a70c40d690cb86ac513bf66a40bd"}
Jul 07 16:46:15.013548 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:15.013538    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs" event={"ID":"f959ebb9-6e2d-48da-88b8-647a596e1413","Type":"ContainerStarted","Data":"b78cb560c4e28fd7bb0ad6498ba4b4f54344522ee8b9d6e51cb63aa3a2131fbb"}
Jul 07 16:46:15.013628 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:15.013571    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs" event={"ID":"f959ebb9-6e2d-48da-88b8-647a596e1413","Type":"ContainerStarted","Data":"d4dada0154c342d884f9f97863d1b071f7c747dd2eaf60e8f36d4154fc77c8b4"}
Jul 07 16:46:15.013628 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:15.013601    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs" event={"ID":"f959ebb9-6e2d-48da-88b8-647a596e1413","Type":"ContainerStarted","Data":"9d317c6c20f6b376afb87f90769d529f9e1fd712b794b741ee87a5a158fa4359"}
Jul 07 16:46:15.014772 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:15.014745    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sgl5" event={"ID":"f9035994-1125-405f-91db-cbe6f9c8af5a","Type":"ContainerStarted","Data":"5d6cdfb46891609859d75dd39471373fcab8512334ab1f6ec11e7dbc78c9a81c"}
Jul 07 16:46:15.016190 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:15.016167    2584 generic.go:356] "Generic (PLEG): container finished" podID="3b4f517c-5cf2-440c-ad4a-bf3aa0934616" containerID="f3785eda54ce16b48592cb7bc1ce838e8ef84c0165d12ab6a2b39e8532995520" exitCode=0
Jul 07 16:46:15.016287 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:15.016226    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phn99" event={"ID":"3b4f517c-5cf2-440c-ad4a-bf3aa0934616","Type":"ContainerDied","Data":"f3785eda54ce16b48592cb7bc1ce838e8ef84c0165d12ab6a2b39e8532995520"}
Jul 07 16:46:15.017667 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:15.017642    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ptgp6" event={"ID":"3d18f299-6f72-46ff-abf2-0ccf120fc875","Type":"ContainerStarted","Data":"7222208fabd28fd9f9e1ddf97948573852c2bbea66005a54bea45a3d10f2839f"}
Jul 07 16:46:15.018987 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:15.018963    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gjbh8" event={"ID":"a1a8fd82-5f36-474b-a780-b30b1d5122ca","Type":"ContainerStarted","Data":"265e4ec2f3df15ce27c543df4631724c5d91d7710a78f99adf7e1f7dc11fd3d8"}
Jul 07 16:46:15.020324 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:15.020296    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9jfs6" event={"ID":"16653221-598d-4229-a6d4-039bdaf5118d","Type":"ContainerStarted","Data":"ddde3ae1b3c03988635f99e86ed23b898c8841b1b6c87d03d7fbe238de07a9e6"}
Jul 07 16:46:15.021542 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:15.021524    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kdq9n" event={"ID":"73096d96-d52d-47ee-95a8-2242bcfb4888","Type":"ContainerStarted","Data":"68dbfa8d47ba487eb01da18be4255ed06a63a9894cacb651f2485b6b503692ec"}
Jul 07 16:46:15.082092 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:15.082018    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-kdq9n" podStartSLOduration=3.412657156 podStartE2EDuration="23.082004388s" totalImagesPullingTime="19.669347232s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:45:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:46:15.081852884 +0000 UTC m=+23.683061588" watchObservedRunningTime="2026-07-07 16:46:15.082004388 +0000 UTC m=+23.683213091"
Jul 07 16:46:15.106949 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:15.106900    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9jfs6" podStartSLOduration=3.495486596 podStartE2EDuration="23.106885087s" totalImagesPullingTime="19.611398491s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:45:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:46:15.106712931 +0000 UTC m=+23.707921627" watchObservedRunningTime="2026-07-07 16:46:15.106885087 +0000 UTC m=+23.708093788"
Jul 07 16:46:15.131762 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:15.131657    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gjbh8" podStartSLOduration=10.63939031 podStartE2EDuration="22.131635711s" totalImagesPullingTime="11.492245401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:45:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:46:15.130913459 +0000 UTC m=+23.732122164" watchObservedRunningTime="2026-07-07 16:46:15.131635711 +0000 UTC m=+23.732844416"
Jul 07 16:46:15.151966 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:15.151910    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ptgp6" podStartSLOduration=3.534540151 podStartE2EDuration="23.151897173s" totalImagesPullingTime="19.617357022s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:45:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:46:15.151554682 +0000 UTC m=+23.752763387" watchObservedRunningTime="2026-07-07 16:46:15.151897173 +0000 UTC m=+23.753105877"
Jul 07 16:46:15.169487 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:15.169439    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cpbvp" podStartSLOduration=2.565260449 podStartE2EDuration="22.169423589s" totalImagesPullingTime="19.60416314s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:45:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:46:15.169408618 +0000 UTC m=+23.770617321" watchObservedRunningTime="2026-07-07 16:46:15.169423589 +0000 UTC m=+23.770632293"
Jul 07 16:46:15.888282 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:15.888198    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:46:15.888282 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:15.888217    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6pr5"
Jul 07 16:46:15.888605 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:15.888331    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bkmf" podUID="b2627fd1-997d-4ce5-84aa-e4b5b85cc307"
Jul 07 16:46:15.888605 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:15.888446    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6pr5" podUID="0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c"
Jul 07 16:46:16.027425 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:16.027392    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs" event={"ID":"f959ebb9-6e2d-48da-88b8-647a596e1413","Type":"ContainerStarted","Data":"75e971d0375c225ffd6147977761cf968634df3763ae15fdbe6c7f1b0c326270"}
Jul 07 16:46:16.028995 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:16.028917    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tgj8c" event={"ID":"a14a5b96-76aa-4d03-be55-8c23f342e4d9","Type":"ContainerStarted","Data":"0e618d2a87c6fe81542ade60c5c5bc3addb09f1a4c8bebe567584f053b1a9767"}
Jul 07 16:46:16.274886 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:16.274704    2584 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 07 16:46:16.283889 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:16.283726    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gjbh8"
Jul 07 16:46:16.284507 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:16.284490    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gjbh8"
Jul 07 16:46:16.307147 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:16.307098    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tgj8c" podStartSLOduration=3.699357719 podStartE2EDuration="23.307080896s" totalImagesPullingTime="19.607723177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:45:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:46:16.056269488 +0000 UTC m=+24.657478193" watchObservedRunningTime="2026-07-07 16:46:16.307080896 +0000 UTC m=+24.908289600"
Jul 07 16:46:16.851949 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:16.851824    2584 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T16:46:16.274871191Z","UUID":"db100d87-d7a8-45d3-b6fb-6ca79e59dd57","Handler":null,"Name":"","Endpoint":""}
Jul 07 16:46:16.853993 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:16.853968    2584 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 07 16:46:16.854172 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:16.854017    2584 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 07 16:46:16.888618 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:16.888588    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9kj8c"
Jul 07 16:46:16.888795 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:16.888699    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9kj8c" podUID="7f7e543c-8cdd-4f28-adb5-570b6004c796"
Jul 07 16:46:17.034597 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:17.034535    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sgl5" event={"ID":"f9035994-1125-405f-91db-cbe6f9c8af5a","Type":"ContainerStarted","Data":"85eb2cce758ffe45f7fc959abd31dd5abe3db037cbece04afdb24e8c8236108c"}
Jul 07 16:46:17.035084 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:17.035041    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gjbh8"
Jul 07 16:46:17.035553 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:17.035533    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gjbh8"
Jul 07 16:46:17.888469 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:17.888433    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:46:17.888642 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:17.888544    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bkmf" podUID="b2627fd1-997d-4ce5-84aa-e4b5b85cc307"
Jul 07 16:46:17.888642 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:17.888603    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6pr5"
Jul 07 16:46:17.888726 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:17.888690    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6pr5" podUID="0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c"
Jul 07 16:46:18.041465 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:18.041303    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs" event={"ID":"f959ebb9-6e2d-48da-88b8-647a596e1413","Type":"ContainerStarted","Data":"a0217660615bd7644bbafd7625cc6aae47b2c58a8c8bf95793db547a184b9603"}
Jul 07 16:46:18.045240 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:18.045204    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sgl5" event={"ID":"f9035994-1125-405f-91db-cbe6f9c8af5a","Type":"ContainerStarted","Data":"abb59f64097fd06fe75fec18e56adde53e9af065622472e09403b418874f118d"}
Jul 07 16:46:18.076262 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:18.076199    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7sgl5" podStartSLOduration=3.655006857 podStartE2EDuration="26.07618404s" totalImagesPullingTime="22.421177183s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:45:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:46:18.075229256 +0000 UTC m=+26.676437960" watchObservedRunningTime="2026-07-07 16:46:18.07618404 +0000 UTC m=+26.677392743"
Jul 07 16:46:18.888487 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:18.888451    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9kj8c"
Jul 07 16:46:18.888650 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:18.888580    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9kj8c" podUID="7f7e543c-8cdd-4f28-adb5-570b6004c796"
Jul 07 16:46:19.888534 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:19.888501    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6pr5"
Jul 07 16:46:19.889244 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:19.888501    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:46:19.889244 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:19.888643    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6pr5" podUID="0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c"
Jul 07 16:46:19.889244 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:19.888711    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bkmf" podUID="b2627fd1-997d-4ce5-84aa-e4b5b85cc307"
Jul 07 16:46:20.889414 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:20.888452    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9kj8c"
Jul 07 16:46:20.889414 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:20.888818    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9kj8c" podUID="7f7e543c-8cdd-4f28-adb5-570b6004c796"
Jul 07 16:46:21.059040 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:21.059000    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs" event={"ID":"f959ebb9-6e2d-48da-88b8-647a596e1413","Type":"ContainerStarted","Data":"9b51979ba8591acb7afd13ac1642cfef50ecd50b2988f8b2f3653f3596cbc362"}
Jul 07 16:46:21.059320 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:21.059274    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:46:21.059438 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:21.059418    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:46:21.059497 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:21.059486    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:46:21.061915 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:21.061883    2584 generic.go:356] "Generic (PLEG): container finished" podID="3b4f517c-5cf2-440c-ad4a-bf3aa0934616" containerID="6ac8eaa068ada7f93f69f6e64ceef4eb963139a2e503bf9f9b9c7b6e6f13e448" exitCode=0
Jul 07 16:46:21.062113 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:21.061935    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phn99" event={"ID":"3b4f517c-5cf2-440c-ad4a-bf3aa0934616","Type":"ContainerDied","Data":"6ac8eaa068ada7f93f69f6e64ceef4eb963139a2e503bf9f9b9c7b6e6f13e448"}
Jul 07 16:46:21.076065 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:21.076021    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:46:21.076211 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:21.076159    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:46:21.093674 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:21.093628    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs" podStartSLOduration=9.418261057 podStartE2EDuration="29.093613092s" totalImagesPullingTime="19.675352035s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:45:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:46:21.093119411 +0000 UTC m=+29.694328116" watchObservedRunningTime="2026-07-07 16:46:21.093613092 +0000 UTC m=+29.694821796"
Jul 07 16:46:21.892117 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:21.889483    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:46:21.892117 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:21.889605    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bkmf" podUID="b2627fd1-997d-4ce5-84aa-e4b5b85cc307"
Jul 07 16:46:21.892117 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:21.889651    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6pr5"
Jul 07 16:46:21.892117 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:21.889757    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6pr5" podUID="0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c"
Jul 07 16:46:22.816650 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:22.816455    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2bkmf"]
Jul 07 16:46:22.816857 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:22.816759    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:46:22.816922 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:22.816859    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bkmf" podUID="b2627fd1-997d-4ce5-84aa-e4b5b85cc307"
Jul 07 16:46:22.818449 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:22.818424    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l6pr5"]
Jul 07 16:46:22.818588 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:22.818536    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6pr5"
Jul 07 16:46:22.818683 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:22.818657    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6pr5" podUID="0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c"
Jul 07 16:46:22.821675 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:22.821637    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9kj8c"]
Jul 07 16:46:22.821821 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:22.821765    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9kj8c"
Jul 07 16:46:22.821891 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:22.821870    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9kj8c" podUID="7f7e543c-8cdd-4f28-adb5-570b6004c796"
Jul 07 16:46:23.068933 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:23.068856    2584 generic.go:356] "Generic (PLEG): container finished" podID="3b4f517c-5cf2-440c-ad4a-bf3aa0934616" containerID="2864c642d977f9e1abff11314d8b074a22ea35d25fcc41a7ad849bd4e0ed93f2" exitCode=0
Jul 07 16:46:23.069333 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:23.068947    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phn99" event={"ID":"3b4f517c-5cf2-440c-ad4a-bf3aa0934616","Type":"ContainerDied","Data":"2864c642d977f9e1abff11314d8b074a22ea35d25fcc41a7ad849bd4e0ed93f2"}
Jul 07 16:46:23.888139 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:23.888109    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:46:23.888326 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:23.888228    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bkmf" podUID="b2627fd1-997d-4ce5-84aa-e4b5b85cc307"
Jul 07 16:46:24.888650 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:24.888611    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6pr5"
Jul 07 16:46:24.889317 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:24.888618    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9kj8c"
Jul 07 16:46:24.889317 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:24.888713    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6pr5" podUID="0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c"
Jul 07 16:46:24.889317 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:24.888823    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9kj8c" podUID="7f7e543c-8cdd-4f28-adb5-570b6004c796"
Jul 07 16:46:25.076954 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:25.076859    2584 generic.go:356] "Generic (PLEG): container finished" podID="3b4f517c-5cf2-440c-ad4a-bf3aa0934616" containerID="b4385815437eb54d0b786362c26f37cc9ddcd3013dea161b41be807844314879" exitCode=0
Jul 07 16:46:25.076954 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:25.076906    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phn99" event={"ID":"3b4f517c-5cf2-440c-ad4a-bf3aa0934616","Type":"ContainerDied","Data":"b4385815437eb54d0b786362c26f37cc9ddcd3013dea161b41be807844314879"}
Jul 07 16:46:25.561218 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:25.561181    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f7e543c-8cdd-4f28-adb5-570b6004c796-metrics-certs\") pod \"network-metrics-daemon-9kj8c\" (UID: \"7f7e543c-8cdd-4f28-adb5-570b6004c796\") " pod="openshift-multus/network-metrics-daemon-9kj8c"
Jul 07 16:46:25.561382 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:25.561319    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:46:25.561429 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:25.561400    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f7e543c-8cdd-4f28-adb5-570b6004c796-metrics-certs podName:7f7e543c-8cdd-4f28-adb5-570b6004c796 nodeName:}" failed. No retries permitted until 2026-07-07 16:46:57.561378095 +0000 UTC m=+66.162586780 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f7e543c-8cdd-4f28-adb5-570b6004c796-metrics-certs") pod "network-metrics-daemon-9kj8c" (UID: "7f7e543c-8cdd-4f28-adb5-570b6004c796") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 16:46:25.662176 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:25.662116    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2627fd1-997d-4ce5-84aa-e4b5b85cc307-original-pull-secret\") pod \"global-pull-secret-syncer-2bkmf\" (UID: \"b2627fd1-997d-4ce5-84aa-e4b5b85cc307\") " pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:46:25.662366 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:25.662198    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hppvt\" (UniqueName: \"kubernetes.io/projected/0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c-kube-api-access-hppvt\") pod \"network-check-target-l6pr5\" (UID: \"0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c\") " pod="openshift-network-diagnostics/network-check-target-l6pr5"
Jul 07 16:46:25.662366 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:25.662277    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 16:46:25.662366 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:25.662306    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 16:46:25.662366 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:25.662326    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 16:46:25.662366 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:25.662336    2584 projected.go:196] Error preparing data for projected volume kube-api-access-hppvt for pod openshift-network-diagnostics/network-check-target-l6pr5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:46:25.662366 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:25.662348    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2627fd1-997d-4ce5-84aa-e4b5b85cc307-original-pull-secret podName:b2627fd1-997d-4ce5-84aa-e4b5b85cc307 nodeName:}" failed. No retries permitted until 2026-07-07 16:46:57.662334023 +0000 UTC m=+66.263542705 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2627fd1-997d-4ce5-84aa-e4b5b85cc307-original-pull-secret") pod "global-pull-secret-syncer-2bkmf" (UID: "b2627fd1-997d-4ce5-84aa-e4b5b85cc307") : object "kube-system"/"original-pull-secret" not registered
Jul 07 16:46:25.662747 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:25.662372    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c-kube-api-access-hppvt podName:0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c nodeName:}" failed. No retries permitted until 2026-07-07 16:46:57.662361978 +0000 UTC m=+66.263570660 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hppvt" (UniqueName: "kubernetes.io/projected/0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c-kube-api-access-hppvt") pod "network-check-target-l6pr5" (UID: "0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 16:46:25.888355 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:25.888322    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:46:25.888530 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:25.888435    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bkmf" podUID="b2627fd1-997d-4ce5-84aa-e4b5b85cc307"
Jul 07 16:46:26.888009 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:26.887973    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9kj8c"
Jul 07 16:46:26.888494 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:26.887977    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6pr5"
Jul 07 16:46:26.888494 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:26.888109    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9kj8c" podUID="7f7e543c-8cdd-4f28-adb5-570b6004c796"
Jul 07 16:46:26.888494 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:26.888150    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6pr5" podUID="0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c"
Jul 07 16:46:27.888538 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:27.888509    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:46:27.888942 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:27.888634    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2bkmf" podUID="b2627fd1-997d-4ce5-84aa-e4b5b85cc307"
Jul 07 16:46:28.231543 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.231468    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-184.ec2.internal" event="NodeReady"
Jul 07 16:46:28.231681 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.231606    2584 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 07 16:46:28.275170 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.275133    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-54b88fb4c5-zhjwb"]
Jul 07 16:46:28.277208 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.277185    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:46:28.280884 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.280859    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 07 16:46:28.281437 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.281416    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 07 16:46:28.281565 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.281445    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 07 16:46:28.281565 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.281538    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xv5k5\""
Jul 07 16:46:28.291694 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.291673    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 07 16:46:28.299610 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.299583    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54b88fb4c5-zhjwb"]
Jul 07 16:46:28.313515 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.313483    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ml4f2"]
Jul 07 16:46:28.315513 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.315477    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wv8wf"]
Jul 07 16:46:28.315650 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.315617    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ml4f2"
Jul 07 16:46:28.317479 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.317456    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wv8wf"
Jul 07 16:46:28.320488 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.320466    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 07 16:46:28.322109 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.322090    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 07 16:46:28.322283 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.322143    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-f72zk\""
Jul 07 16:46:28.322283 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.322217    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9pvjc\""
Jul 07 16:46:28.322395 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.322363    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 07 16:46:28.322444 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.322419    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 07 16:46:28.327628 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.327605    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 07 16:46:28.337522 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.337499    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wv8wf"]
Jul 07 16:46:28.340966 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.340943    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ml4f2"]
Jul 07 16:46:28.383369 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.383328    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-installation-pull-secrets\") pod \"image-registry-54b88fb4c5-zhjwb\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") " pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:46:28.383529 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.383401    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-image-registry-private-configuration\") pod \"image-registry-54b88fb4c5-zhjwb\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") " pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:46:28.383529 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.383443    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-trusted-ca\") pod \"image-registry-54b88fb4c5-zhjwb\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") " pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:46:28.383529 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.383473    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-registry-certificates\") pod \"image-registry-54b88fb4c5-zhjwb\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") " pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:46:28.383529 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.383510    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-registry-tls\") pod \"image-registry-54b88fb4c5-zhjwb\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") " pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:46:28.383724 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.383532    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-bound-sa-token\") pod \"image-registry-54b88fb4c5-zhjwb\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") " pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:46:28.383724 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.383563    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-ca-trust-extracted\") pod \"image-registry-54b88fb4c5-zhjwb\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") " pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:46:28.383724 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.383614    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvt6c\" (UniqueName: \"kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-kube-api-access-xvt6c\") pod \"image-registry-54b88fb4c5-zhjwb\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") " pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:46:28.484591 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.484509    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-registry-tls\") pod \"image-registry-54b88fb4c5-zhjwb\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") " pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:46:28.484591 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.484553    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-bound-sa-token\") pod \"image-registry-54b88fb4c5-zhjwb\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") " pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:46:28.484833 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:28.484668    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 16:46:28.484833 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:28.484689    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54b88fb4c5-zhjwb: secret "image-registry-tls" not found
Jul 07 16:46:28.484833 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.484676    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3b39298-a197-40cb-8363-912fdc7dbcc8-tmp-dir\") pod \"dns-default-wv8wf\" (UID: \"f3b39298-a197-40cb-8363-912fdc7dbcc8\") " pod="openshift-dns/dns-default-wv8wf"
Jul 07 16:46:28.484833 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:28.484751    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-registry-tls podName:2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2 nodeName:}" failed. No retries permitted until 2026-07-07 16:46:28.984730361 +0000 UTC m=+37.585939062 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-registry-tls") pod "image-registry-54b88fb4c5-zhjwb" (UID: "2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2") : secret "image-registry-tls" not found
Jul 07 16:46:28.484833 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.484771    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-ca-trust-extracted\") pod \"image-registry-54b88fb4c5-zhjwb\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") " pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:46:28.484833 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.484791    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b39298-a197-40cb-8363-912fdc7dbcc8-config-volume\") pod \"dns-default-wv8wf\" (UID: \"f3b39298-a197-40cb-8363-912fdc7dbcc8\") " pod="openshift-dns/dns-default-wv8wf"
Jul 07 16:46:28.485158 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.484843    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvt6c\" (UniqueName: \"kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-kube-api-access-xvt6c\") pod \"image-registry-54b88fb4c5-zhjwb\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") " pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:46:28.485158 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.484895    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zswl8\" (UniqueName: \"kubernetes.io/projected/b8e32c6b-206f-4882-b1c0-066d26486d41-kube-api-access-zswl8\") pod \"ingress-canary-ml4f2\" (UID: \"b8e32c6b-206f-4882-b1c0-066d26486d41\") " pod="openshift-ingress-canary/ingress-canary-ml4f2"
Jul 07 16:46:28.485158 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.484936    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-installation-pull-secrets\") pod \"image-registry-54b88fb4c5-zhjwb\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") " pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:46:28.485158 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.484962    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3b39298-a197-40cb-8363-912fdc7dbcc8-metrics-tls\") pod \"dns-default-wv8wf\" (UID: \"f3b39298-a197-40cb-8363-912fdc7dbcc8\") " pod="openshift-dns/dns-default-wv8wf"
Jul 07 16:46:28.485158 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.485024    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-image-registry-private-configuration\") pod \"image-registry-54b88fb4c5-zhjwb\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") " pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:46:28.485158 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.485082    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-trusted-ca\") pod \"image-registry-54b88fb4c5-zhjwb\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") " pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:46:28.485158 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.485115    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-registry-certificates\") pod \"image-registry-54b88fb4c5-zhjwb\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") " pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:46:28.485158 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.485139    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8e32c6b-206f-4882-b1c0-066d26486d41-cert\") pod \"ingress-canary-ml4f2\" (UID: \"b8e32c6b-206f-4882-b1c0-066d26486d41\") " pod="openshift-ingress-canary/ingress-canary-ml4f2"
Jul 07 16:46:28.485523 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.485166    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc4jn\" (UniqueName: \"kubernetes.io/projected/f3b39298-a197-40cb-8363-912fdc7dbcc8-kube-api-access-qc4jn\") pod \"dns-default-wv8wf\" (UID: \"f3b39298-a197-40cb-8363-912fdc7dbcc8\") " pod="openshift-dns/dns-default-wv8wf"
Jul 07 16:46:28.485523 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.485223    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-ca-trust-extracted\") pod \"image-registry-54b88fb4c5-zhjwb\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") " pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:46:28.485918 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.485891    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-registry-certificates\") pod \"image-registry-54b88fb4c5-zhjwb\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") " pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:46:28.486241 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.486220    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-trusted-ca\") pod \"image-registry-54b88fb4c5-zhjwb\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") " pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:46:28.489976 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.489955    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-installation-pull-secrets\") pod \"image-registry-54b88fb4c5-zhjwb\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") " pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:46:28.490091 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.490020    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-image-registry-private-configuration\") pod \"image-registry-54b88fb4c5-zhjwb\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") " pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:46:28.501990 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.501956    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-bound-sa-token\") pod \"image-registry-54b88fb4c5-zhjwb\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") " pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:46:28.506267 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.506244    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvt6c\" (UniqueName: \"kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-kube-api-access-xvt6c\") pod \"image-registry-54b88fb4c5-zhjwb\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") " pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:46:28.586311 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.586267    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3b39298-a197-40cb-8363-912fdc7dbcc8-tmp-dir\") pod \"dns-default-wv8wf\" (UID: \"f3b39298-a197-40cb-8363-912fdc7dbcc8\") " pod="openshift-dns/dns-default-wv8wf"
Jul 07 16:46:28.586311 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.586311    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b39298-a197-40cb-8363-912fdc7dbcc8-config-volume\") pod \"dns-default-wv8wf\" (UID: \"f3b39298-a197-40cb-8363-912fdc7dbcc8\") " pod="openshift-dns/dns-default-wv8wf"
Jul 07 16:46:28.586508 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.586436    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zswl8\" (UniqueName: \"kubernetes.io/projected/b8e32c6b-206f-4882-b1c0-066d26486d41-kube-api-access-zswl8\") pod \"ingress-canary-ml4f2\" (UID: \"b8e32c6b-206f-4882-b1c0-066d26486d41\") " pod="openshift-ingress-canary/ingress-canary-ml4f2"
Jul 07 16:46:28.586508 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.586478    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3b39298-a197-40cb-8363-912fdc7dbcc8-metrics-tls\") pod \"dns-default-wv8wf\" (UID: \"f3b39298-a197-40cb-8363-912fdc7dbcc8\") " pod="openshift-dns/dns-default-wv8wf"
Jul 07 16:46:28.586593 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.586535    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8e32c6b-206f-4882-b1c0-066d26486d41-cert\") pod \"ingress-canary-ml4f2\" (UID: \"b8e32c6b-206f-4882-b1c0-066d26486d41\") " pod="openshift-ingress-canary/ingress-canary-ml4f2"
Jul 07 16:46:28.586593 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.586560    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc4jn\" (UniqueName: \"kubernetes.io/projected/f3b39298-a197-40cb-8363-912fdc7dbcc8-kube-api-access-qc4jn\") pod \"dns-default-wv8wf\" (UID: \"f3b39298-a197-40cb-8363-912fdc7dbcc8\") " pod="openshift-dns/dns-default-wv8wf"
Jul 07 16:46:28.586679 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:28.586655    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 16:46:28.586728 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.586705    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3b39298-a197-40cb-8363-912fdc7dbcc8-tmp-dir\") pod \"dns-default-wv8wf\" (UID: \"f3b39298-a197-40cb-8363-912fdc7dbcc8\") " pod="openshift-dns/dns-default-wv8wf"
Jul 07 16:46:28.586782 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:28.586719    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3b39298-a197-40cb-8363-912fdc7dbcc8-metrics-tls podName:f3b39298-a197-40cb-8363-912fdc7dbcc8 nodeName:}" failed. No retries permitted until 2026-07-07 16:46:29.086700739 +0000 UTC m=+37.687909437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f3b39298-a197-40cb-8363-912fdc7dbcc8-metrics-tls") pod "dns-default-wv8wf" (UID: "f3b39298-a197-40cb-8363-912fdc7dbcc8") : secret "dns-default-metrics-tls" not found
Jul 07 16:46:28.586900 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.586881    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b39298-a197-40cb-8363-912fdc7dbcc8-config-volume\") pod \"dns-default-wv8wf\" (UID: \"f3b39298-a197-40cb-8363-912fdc7dbcc8\") " pod="openshift-dns/dns-default-wv8wf"
Jul 07 16:46:28.589301 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.589281    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8e32c6b-206f-4882-b1c0-066d26486d41-cert\") pod \"ingress-canary-ml4f2\" (UID: \"b8e32c6b-206f-4882-b1c0-066d26486d41\") " pod="openshift-ingress-canary/ingress-canary-ml4f2"
Jul 07 16:46:28.602886 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.602859    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zswl8\" (UniqueName: \"kubernetes.io/projected/b8e32c6b-206f-4882-b1c0-066d26486d41-kube-api-access-zswl8\") pod \"ingress-canary-ml4f2\" (UID: \"b8e32c6b-206f-4882-b1c0-066d26486d41\") " pod="openshift-ingress-canary/ingress-canary-ml4f2"
Jul 07 16:46:28.604325 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.604304    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc4jn\" (UniqueName: \"kubernetes.io/projected/f3b39298-a197-40cb-8363-912fdc7dbcc8-kube-api-access-qc4jn\") pod \"dns-default-wv8wf\" (UID: \"f3b39298-a197-40cb-8363-912fdc7dbcc8\") " pod="openshift-dns/dns-default-wv8wf"
Jul 07 16:46:28.629421 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.629390    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ml4f2"
Jul 07 16:46:28.888884 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.888670    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6pr5"
Jul 07 16:46:28.889425 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.888723    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9kj8c"
Jul 07 16:46:28.894916 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.894892    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bvb6g\""
Jul 07 16:46:28.895095 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.894944    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 16:46:28.895251 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.895231    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 16:46:28.895546 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.895522    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rsbn5\""
Jul 07 16:46:28.895644 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.895555    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 16:46:28.990196 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:28.990160    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-registry-tls\") pod \"image-registry-54b88fb4c5-zhjwb\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") " pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:46:28.990348 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:28.990318    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 16:46:28.990348 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:28.990340    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54b88fb4c5-zhjwb: secret "image-registry-tls" not found
Jul 07 16:46:28.990441 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:28.990405    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-registry-tls podName:2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2 nodeName:}" failed. No retries permitted until 2026-07-07 16:46:29.99038406 +0000 UTC m=+38.591592745 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-registry-tls") pod "image-registry-54b88fb4c5-zhjwb" (UID: "2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2") : secret "image-registry-tls" not found
Jul 07 16:46:29.091502 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:29.091460    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3b39298-a197-40cb-8363-912fdc7dbcc8-metrics-tls\") pod \"dns-default-wv8wf\" (UID: \"f3b39298-a197-40cb-8363-912fdc7dbcc8\") " pod="openshift-dns/dns-default-wv8wf"
Jul 07 16:46:29.091677 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:29.091625    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 16:46:29.091723 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:29.091692    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3b39298-a197-40cb-8363-912fdc7dbcc8-metrics-tls podName:f3b39298-a197-40cb-8363-912fdc7dbcc8 nodeName:}" failed. No retries permitted until 2026-07-07 16:46:30.091677932 +0000 UTC m=+38.692886614 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f3b39298-a197-40cb-8363-912fdc7dbcc8-metrics-tls") pod "dns-default-wv8wf" (UID: "f3b39298-a197-40cb-8363-912fdc7dbcc8") : secret "dns-default-metrics-tls" not found
Jul 07 16:46:29.888082 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:29.888026    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:46:29.891320 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:29.891290    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 16:46:29.999221 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:29.999189    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-registry-tls\") pod \"image-registry-54b88fb4c5-zhjwb\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") " pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:46:29.999398 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:29.999360    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 16:46:29.999398 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:29.999375    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54b88fb4c5-zhjwb: secret "image-registry-tls" not found
Jul 07 16:46:29.999515 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:29.999461    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-registry-tls podName:2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2 nodeName:}" failed. No retries permitted until 2026-07-07 16:46:31.999447176 +0000 UTC m=+40.600655858 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-registry-tls") pod "image-registry-54b88fb4c5-zhjwb" (UID: "2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2") : secret "image-registry-tls" not found
Jul 07 16:46:30.099950 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:30.099911    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3b39298-a197-40cb-8363-912fdc7dbcc8-metrics-tls\") pod \"dns-default-wv8wf\" (UID: \"f3b39298-a197-40cb-8363-912fdc7dbcc8\") " pod="openshift-dns/dns-default-wv8wf"
Jul 07 16:46:30.100124 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:30.100024    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 16:46:30.100124 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:30.100115    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3b39298-a197-40cb-8363-912fdc7dbcc8-metrics-tls podName:f3b39298-a197-40cb-8363-912fdc7dbcc8 nodeName:}" failed. No retries permitted until 2026-07-07 16:46:32.100094776 +0000 UTC m=+40.701303458 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f3b39298-a197-40cb-8363-912fdc7dbcc8-metrics-tls") pod "dns-default-wv8wf" (UID: "f3b39298-a197-40cb-8363-912fdc7dbcc8") : secret "dns-default-metrics-tls" not found
Jul 07 16:46:30.251147 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:30.251115    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ml4f2"]
Jul 07 16:46:30.256249 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:46:30.256220    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8e32c6b_206f_4882_b1c0_066d26486d41.slice/crio-41e7318613a0f61026a1306a259e15271df648fe8bbf22c445a060452655f2cc WatchSource:0}: Error finding container 41e7318613a0f61026a1306a259e15271df648fe8bbf22c445a060452655f2cc: Status 404 returned error can't find the container with id 41e7318613a0f61026a1306a259e15271df648fe8bbf22c445a060452655f2cc
Jul 07 16:46:31.091469 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:31.091256    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ml4f2" event={"ID":"b8e32c6b-206f-4882-b1c0-066d26486d41","Type":"ContainerStarted","Data":"41e7318613a0f61026a1306a259e15271df648fe8bbf22c445a060452655f2cc"}
Jul 07 16:46:31.094578 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:31.094547    2584 generic.go:356] "Generic (PLEG): container finished" podID="3b4f517c-5cf2-440c-ad4a-bf3aa0934616" containerID="8d4368ab281e0a7684fdc1c4c830b8acde5d1e8596859e45071eae4e2355f651" exitCode=0
Jul 07 16:46:31.094697 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:31.094597    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phn99" event={"ID":"3b4f517c-5cf2-440c-ad4a-bf3aa0934616","Type":"ContainerDied","Data":"8d4368ab281e0a7684fdc1c4c830b8acde5d1e8596859e45071eae4e2355f651"}
Jul 07 16:46:31.270066 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:31.270019    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:46:31.451250 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:31.451216    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:46:31.566097 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:31.566035    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:46:31.761565 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:31.761491    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:46:31.870006 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:31.869892    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:46:32.018629 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:32.018548    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-registry-tls\") pod \"image-registry-54b88fb4c5-zhjwb\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") " pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:46:32.018779 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:32.018692    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 16:46:32.018779 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:32.018711    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54b88fb4c5-zhjwb: secret "image-registry-tls" not found
Jul 07 16:46:32.018779 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:32.018775    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-registry-tls podName:2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2 nodeName:}" failed. No retries permitted until 2026-07-07 16:46:36.018754266 +0000 UTC m=+44.619962972 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-registry-tls") pod "image-registry-54b88fb4c5-zhjwb" (UID: "2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2") : secret "image-registry-tls" not found
Jul 07 16:46:32.044899 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:32.044865    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:46:32.101614 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:32.101583    2584 generic.go:356] "Generic (PLEG): container finished" podID="3b4f517c-5cf2-440c-ad4a-bf3aa0934616" containerID="5805aa254e0d856b6b62430bd364e34c2cd34235dc0e57ca15a717c450020ea4" exitCode=0
Jul 07 16:46:32.101614 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:32.101620    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phn99" event={"ID":"3b4f517c-5cf2-440c-ad4a-bf3aa0934616","Type":"ContainerDied","Data":"5805aa254e0d856b6b62430bd364e34c2cd34235dc0e57ca15a717c450020ea4"}
Jul 07 16:46:32.119298 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:32.119259    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3b39298-a197-40cb-8363-912fdc7dbcc8-metrics-tls\") pod \"dns-default-wv8wf\" (UID: \"f3b39298-a197-40cb-8363-912fdc7dbcc8\") " pod="openshift-dns/dns-default-wv8wf"
Jul 07 16:46:32.119431 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:32.119411    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 16:46:32.119485 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:32.119472    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3b39298-a197-40cb-8363-912fdc7dbcc8-metrics-tls podName:f3b39298-a197-40cb-8363-912fdc7dbcc8 nodeName:}" failed. No retries permitted until 2026-07-07 16:46:36.119452167 +0000 UTC m=+44.720660850 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f3b39298-a197-40cb-8363-912fdc7dbcc8-metrics-tls") pod "dns-default-wv8wf" (UID: "f3b39298-a197-40cb-8363-912fdc7dbcc8") : secret "dns-default-metrics-tls" not found
Jul 07 16:46:32.147214 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:32.147172    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:46:32.492812 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:32.492782    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:46:32.591841 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:32.591807    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:46:33.109031 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:33.108984    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phn99" event={"ID":"3b4f517c-5cf2-440c-ad4a-bf3aa0934616","Type":"ContainerStarted","Data":"0f9b4bb3b9b7270fee444bcee4a88f95053c4de8852d18968515275404d38741"}
Jul 07 16:46:33.110587 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:33.110557    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ml4f2" event={"ID":"b8e32c6b-206f-4882-b1c0-066d26486d41","Type":"ContainerStarted","Data":"0df2757e74b0159a222645010753a5efdc2088a3c15ffea4101dbeb97634ea0b"}
Jul 07 16:46:33.147187 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:33.147130    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-phn99" podStartSLOduration=7.55031228 podStartE2EDuration="40.147114237s" totalImagesPullingTime="32.596801957s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:45:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:46:33.144309892 +0000 UTC m=+41.745518596" watchObservedRunningTime="2026-07-07 16:46:33.147114237 +0000 UTC m=+41.748322941"
Jul 07 16:46:33.162651 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:33.162599    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ml4f2" podStartSLOduration=2.462910695 podStartE2EDuration="5.162582449s" totalImagesPullingTime="2.699671754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:46:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:46:33.161971363 +0000 UTC m=+41.763180068" watchObservedRunningTime="2026-07-07 16:46:33.162582449 +0000 UTC m=+41.763791153"
Jul 07 16:46:33.248596 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:33.248513    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:46:33.350510 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:33.350475    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:46:33.628463 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:33.628422    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:46:33.729367 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:33.729330    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:46:34.431897 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:34.431864    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:46:34.532163 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:34.532131    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:46:34.827233 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:34.827141    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:46:34.928098 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:34.928063    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:46:36.052480 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:36.052444    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-registry-tls\") pod \"image-registry-54b88fb4c5-zhjwb\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") " pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:46:36.052831 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:36.052606    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 16:46:36.052831 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:36.052619    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54b88fb4c5-zhjwb: secret "image-registry-tls" not found
Jul 07 16:46:36.052831 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:36.052675    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-registry-tls podName:2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2 nodeName:}" failed. No retries permitted until 2026-07-07 16:46:44.0526586 +0000 UTC m=+52.653867305 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-registry-tls") pod "image-registry-54b88fb4c5-zhjwb" (UID: "2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2") : secret "image-registry-tls" not found
Jul 07 16:46:36.153381 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:36.153339    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3b39298-a197-40cb-8363-912fdc7dbcc8-metrics-tls\") pod \"dns-default-wv8wf\" (UID: \"f3b39298-a197-40cb-8363-912fdc7dbcc8\") " pod="openshift-dns/dns-default-wv8wf"
Jul 07 16:46:36.153536 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:36.153471    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 16:46:36.153536 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:36.153519    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3b39298-a197-40cb-8363-912fdc7dbcc8-metrics-tls podName:f3b39298-a197-40cb-8363-912fdc7dbcc8 nodeName:}" failed. No retries permitted until 2026-07-07 16:46:44.153506014 +0000 UTC m=+52.754714697 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f3b39298-a197-40cb-8363-912fdc7dbcc8-metrics-tls") pod "dns-default-wv8wf" (UID: "f3b39298-a197-40cb-8363-912fdc7dbcc8") : secret "dns-default-metrics-tls" not found
Jul 07 16:46:41.036767 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:41.036733    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:46:42.521028 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:42.520990    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:46:44.106753 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:44.106717    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-registry-tls\") pod \"image-registry-54b88fb4c5-zhjwb\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") " pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:46:44.107181 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:44.106919    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 16:46:44.107181 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:44.106938    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54b88fb4c5-zhjwb: secret "image-registry-tls" not found
Jul 07 16:46:44.107181 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:44.106992    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-registry-tls podName:2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:00.106976751 +0000 UTC m=+68.708185466 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-registry-tls") pod "image-registry-54b88fb4c5-zhjwb" (UID: "2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2") : secret "image-registry-tls" not found
Jul 07 16:46:44.207655 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:44.207615    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3b39298-a197-40cb-8363-912fdc7dbcc8-metrics-tls\") pod \"dns-default-wv8wf\" (UID: \"f3b39298-a197-40cb-8363-912fdc7dbcc8\") " pod="openshift-dns/dns-default-wv8wf"
Jul 07 16:46:44.207821 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:44.207802    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 16:46:44.207881 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:44.207870    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3b39298-a197-40cb-8363-912fdc7dbcc8-metrics-tls podName:f3b39298-a197-40cb-8363-912fdc7dbcc8 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:00.207849003 +0000 UTC m=+68.809057686 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f3b39298-a197-40cb-8363-912fdc7dbcc8-metrics-tls") pod "dns-default-wv8wf" (UID: "f3b39298-a197-40cb-8363-912fdc7dbcc8") : secret "dns-default-metrics-tls" not found
Jul 07 16:46:53.086013 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:53.085987    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qlfcs"
Jul 07 16:46:53.266834 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:53.266795    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:46:53.372903 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:53.372862    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:46:54.349628 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:54.349590    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:46:54.425257 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:54.425214    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:46:54.524474 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:54.524436    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:46:54.596716 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:54.596680    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:46:55.455706 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:55.455670    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:46:55.528275 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:55.528244    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:46:57.604450 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:57.604407    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f7e543c-8cdd-4f28-adb5-570b6004c796-metrics-certs\") pod \"network-metrics-daemon-9kj8c\" (UID: \"7f7e543c-8cdd-4f28-adb5-570b6004c796\") " pod="openshift-multus/network-metrics-daemon-9kj8c"
Jul 07 16:46:57.607712 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:57.607693    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 16:46:57.615348 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:57.615330    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 07 16:46:57.615404 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:46:57.615388    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f7e543c-8cdd-4f28-adb5-570b6004c796-metrics-certs podName:7f7e543c-8cdd-4f28-adb5-570b6004c796 nodeName:}" failed. No retries permitted until 2026-07-07 16:48:01.615370947 +0000 UTC m=+130.216579641 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f7e543c-8cdd-4f28-adb5-570b6004c796-metrics-certs") pod "network-metrics-daemon-9kj8c" (UID: "7f7e543c-8cdd-4f28-adb5-570b6004c796") : secret "metrics-daemon-secret" not found
Jul 07 16:46:57.705290 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:57.705256    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hppvt\" (UniqueName: \"kubernetes.io/projected/0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c-kube-api-access-hppvt\") pod \"network-check-target-l6pr5\" (UID: \"0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c\") " pod="openshift-network-diagnostics/network-check-target-l6pr5"
Jul 07 16:46:57.705392 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:57.705310    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2627fd1-997d-4ce5-84aa-e4b5b85cc307-original-pull-secret\") pod \"global-pull-secret-syncer-2bkmf\" (UID: \"b2627fd1-997d-4ce5-84aa-e4b5b85cc307\") " pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:46:57.709040 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:57.709026    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 16:46:57.709216 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:57.709198    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 16:46:57.719325 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:57.719306    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 16:46:57.720118 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:57.720100    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2627fd1-997d-4ce5-84aa-e4b5b85cc307-original-pull-secret\") pod \"global-pull-secret-syncer-2bkmf\" (UID: \"b2627fd1-997d-4ce5-84aa-e4b5b85cc307\") " pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:46:57.729454 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:57.729427    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hppvt\" (UniqueName: \"kubernetes.io/projected/0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c-kube-api-access-hppvt\") pod \"network-check-target-l6pr5\" (UID: \"0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c\") " pod="openshift-network-diagnostics/network-check-target-l6pr5"
Jul 07 16:46:57.799767 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:57.799728    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2bkmf"
Jul 07 16:46:57.927234 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:57.927197    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2bkmf"]
Jul 07 16:46:57.929948 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:46:57.929923    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2627fd1_997d_4ce5_84aa_e4b5b85cc307.slice/crio-99b4516938e28d6f6f76e5240f1902d46294d63f487e018e506e105d1c842c2e WatchSource:0}: Error finding container 99b4516938e28d6f6f76e5240f1902d46294d63f487e018e506e105d1c842c2e: Status 404 returned error can't find the container with id 99b4516938e28d6f6f76e5240f1902d46294d63f487e018e506e105d1c842c2e
Jul 07 16:46:58.003686 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:58.003652    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bvb6g\""
Jul 07 16:46:58.011520 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:58.011495    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6pr5"
Jul 07 16:46:58.155038 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:58.154960    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l6pr5"]
Jul 07 16:46:58.158108 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:46:58.158083    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dc25bd5_419b_41d0_84a9_3ab96e1e5f9c.slice/crio-f98e244e5aac0443fa4db5e71f1217c7ada4e716481f2b8482ce4acda3d26c41 WatchSource:0}: Error finding container f98e244e5aac0443fa4db5e71f1217c7ada4e716481f2b8482ce4acda3d26c41: Status 404 returned error can't find the container with id f98e244e5aac0443fa4db5e71f1217c7ada4e716481f2b8482ce4acda3d26c41
Jul 07 16:46:58.169378 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:58.169347    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l6pr5" event={"ID":"0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c","Type":"ContainerStarted","Data":"f98e244e5aac0443fa4db5e71f1217c7ada4e716481f2b8482ce4acda3d26c41"}
Jul 07 16:46:58.170434 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:58.170415    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2bkmf" event={"ID":"b2627fd1-997d-4ce5-84aa-e4b5b85cc307","Type":"ContainerStarted","Data":"99b4516938e28d6f6f76e5240f1902d46294d63f487e018e506e105d1c842c2e"}
Jul 07 16:46:58.370973 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:46:58.370938    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:00.133029 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:00.132973    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-registry-tls\") pod \"image-registry-54b88fb4c5-zhjwb\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") " pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:47:00.133517 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:47:00.133170    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 16:47:00.133517 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:47:00.133194    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54b88fb4c5-zhjwb: secret "image-registry-tls" not found
Jul 07 16:47:00.133517 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:47:00.133269    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-registry-tls podName:2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:32.13324628 +0000 UTC m=+100.734454985 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-registry-tls") pod "image-registry-54b88fb4c5-zhjwb" (UID: "2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2") : secret "image-registry-tls" not found
Jul 07 16:47:00.234268 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:00.234228    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3b39298-a197-40cb-8363-912fdc7dbcc8-metrics-tls\") pod \"dns-default-wv8wf\" (UID: \"f3b39298-a197-40cb-8363-912fdc7dbcc8\") " pod="openshift-dns/dns-default-wv8wf"
Jul 07 16:47:00.234469 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:47:00.234381    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 16:47:00.234469 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:47:00.234447    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3b39298-a197-40cb-8363-912fdc7dbcc8-metrics-tls podName:f3b39298-a197-40cb-8363-912fdc7dbcc8 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:32.234429836 +0000 UTC m=+100.835638535 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f3b39298-a197-40cb-8363-912fdc7dbcc8-metrics-tls") pod "dns-default-wv8wf" (UID: "f3b39298-a197-40cb-8363-912fdc7dbcc8") : secret "dns-default-metrics-tls" not found
Jul 07 16:47:01.373010 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:01.372971    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:01.461280 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:01.461231    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:03.186773 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:03.186737    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l6pr5" event={"ID":"0dc25bd5-419b-41d0-84a9-3ab96e1e5f9c","Type":"ContainerStarted","Data":"aa34e734a628a5b9ae0b8f7d4313fe983136ccf597e0c2ef57b9973e70d9cfc0"}
Jul 07 16:47:03.187270 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:03.186817    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-l6pr5"
Jul 07 16:47:03.188123 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:03.188102    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2bkmf" event={"ID":"b2627fd1-997d-4ce5-84aa-e4b5b85cc307","Type":"ContainerStarted","Data":"44dd4bf41d1547aebf2d1fc92949ee828de7bd0ca2af999d996d3276a709d63d"}
Jul 07 16:47:03.206761 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:03.206709    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-l6pr5" podStartSLOduration=66.760974838 podStartE2EDuration="1m11.206693963s" totalImagesPullingTime="4.445719125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:45:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:47:03.20582478 +0000 UTC m=+71.807033483" watchObservedRunningTime="2026-07-07 16:47:03.206693963 +0000 UTC m=+71.807902644"
Jul 07 16:47:03.226009 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:03.225956    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2bkmf" podStartSLOduration=65.549295601 podStartE2EDuration="1m10.225940845s" totalImagesPullingTime="4.676645244s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:45:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:47:03.225547398 +0000 UTC m=+71.826756102" watchObservedRunningTime="2026-07-07 16:47:03.225940845 +0000 UTC m=+71.827149550"
Jul 07 16:47:06.062907 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:06.062868    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:06.160384 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:06.160347    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:10.418633 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:10.418593    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:10.519807 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:10.519777    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:10.588597 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:10.588555    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:12.984971 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:12.984843    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:13.066201 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:13.066164    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:14.680450 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:14.680414    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-dcrfk"]
Jul 07 16:47:14.683931 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:14.683905    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-dcrfk"
Jul 07 16:47:14.695561 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:14.695540    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-ncdvr\""
Jul 07 16:47:14.695811 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:14.695797    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 07 16:47:14.696892 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:14.696876    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 07 16:47:14.696949 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:14.696892    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 07 16:47:14.699224 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:14.699204    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 07 16:47:14.699559 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:14.699544    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-dcrfk"]
Jul 07 16:47:14.735905 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:14.735865    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f45b3a44-6bd5-4cee-912e-5edc4d056972-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-dcrfk\" (UID: \"f45b3a44-6bd5-4cee-912e-5edc4d056972\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-dcrfk"
Jul 07 16:47:14.736097 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:14.735912    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m22x\" (UniqueName: \"kubernetes.io/projected/f45b3a44-6bd5-4cee-912e-5edc4d056972-kube-api-access-9m22x\") pod \"kube-storage-version-migrator-operator-846bf668f7-dcrfk\" (UID: \"f45b3a44-6bd5-4cee-912e-5edc4d056972\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-dcrfk"
Jul 07 16:47:14.736097 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:14.735991    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f45b3a44-6bd5-4cee-912e-5edc4d056972-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-dcrfk\" (UID: \"f45b3a44-6bd5-4cee-912e-5edc4d056972\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-dcrfk"
Jul 07 16:47:14.836901 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:14.836861    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f45b3a44-6bd5-4cee-912e-5edc4d056972-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-dcrfk\" (UID: \"f45b3a44-6bd5-4cee-912e-5edc4d056972\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-dcrfk"
Jul 07 16:47:14.836901 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:14.836900    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m22x\" (UniqueName: \"kubernetes.io/projected/f45b3a44-6bd5-4cee-912e-5edc4d056972-kube-api-access-9m22x\") pod \"kube-storage-version-migrator-operator-846bf668f7-dcrfk\" (UID: \"f45b3a44-6bd5-4cee-912e-5edc4d056972\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-dcrfk"
Jul 07 16:47:14.837138 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:14.836952    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f45b3a44-6bd5-4cee-912e-5edc4d056972-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-dcrfk\" (UID: \"f45b3a44-6bd5-4cee-912e-5edc4d056972\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-dcrfk"
Jul 07 16:47:14.837602 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:14.837581    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f45b3a44-6bd5-4cee-912e-5edc4d056972-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-dcrfk\" (UID: \"f45b3a44-6bd5-4cee-912e-5edc4d056972\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-dcrfk"
Jul 07 16:47:14.839293 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:14.839276    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f45b3a44-6bd5-4cee-912e-5edc4d056972-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-dcrfk\" (UID: \"f45b3a44-6bd5-4cee-912e-5edc4d056972\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-dcrfk"
Jul 07 16:47:14.847171 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:14.847140    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m22x\" (UniqueName: \"kubernetes.io/projected/f45b3a44-6bd5-4cee-912e-5edc4d056972-kube-api-access-9m22x\") pod \"kube-storage-version-migrator-operator-846bf668f7-dcrfk\" (UID: \"f45b3a44-6bd5-4cee-912e-5edc4d056972\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-dcrfk"
Jul 07 16:47:14.992298 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:14.992220    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-dcrfk"
Jul 07 16:47:15.114963 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:15.114933    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-dcrfk"]
Jul 07 16:47:15.118039 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:47:15.118015    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf45b3a44_6bd5_4cee_912e_5edc4d056972.slice/crio-9ad7625d709542f33de32c13a9711e1c34e1a0c55e9265b89b6155e968bc3c6e WatchSource:0}: Error finding container 9ad7625d709542f33de32c13a9711e1c34e1a0c55e9265b89b6155e968bc3c6e: Status 404 returned error can't find the container with id 9ad7625d709542f33de32c13a9711e1c34e1a0c55e9265b89b6155e968bc3c6e
Jul 07 16:47:15.218036 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:15.218006    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-dcrfk" event={"ID":"f45b3a44-6bd5-4cee-912e-5edc4d056972","Type":"ContainerStarted","Data":"9ad7625d709542f33de32c13a9711e1c34e1a0c55e9265b89b6155e968bc3c6e"}
Jul 07 16:47:15.425514 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:15.425473    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:15.495066 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:15.495010    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:15.756184 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:15.756101    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jk247"]
Jul 07 16:47:15.758134 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:15.758111    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jk247"
Jul 07 16:47:15.761305 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:15.761283    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 07 16:47:15.761417 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:15.761309    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 07 16:47:15.761417 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:15.761338    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2mgsv\""
Jul 07 16:47:15.761417 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:15.761285    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 07 16:47:15.762605 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:15.762593    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 07 16:47:15.775710 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:15.775687    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jk247"]
Jul 07 16:47:15.806130 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:15.806087    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:15.845284 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:15.845247    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/600fd20e-5eba-4886-9bda-68a5c995d9a8-crio-socket\") pod \"insights-runtime-extractor-jk247\" (UID: \"600fd20e-5eba-4886-9bda-68a5c995d9a8\") " pod="openshift-insights/insights-runtime-extractor-jk247"
Jul 07 16:47:15.845469 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:15.845290    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/600fd20e-5eba-4886-9bda-68a5c995d9a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jk247\" (UID: \"600fd20e-5eba-4886-9bda-68a5c995d9a8\") " pod="openshift-insights/insights-runtime-extractor-jk247"
Jul 07 16:47:15.845469 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:15.845403    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs8tm\" (UniqueName: \"kubernetes.io/projected/600fd20e-5eba-4886-9bda-68a5c995d9a8-kube-api-access-bs8tm\") pod \"insights-runtime-extractor-jk247\" (UID: \"600fd20e-5eba-4886-9bda-68a5c995d9a8\") " pod="openshift-insights/insights-runtime-extractor-jk247"
Jul 07 16:47:15.845469 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:15.845431    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/600fd20e-5eba-4886-9bda-68a5c995d9a8-data-volume\") pod \"insights-runtime-extractor-jk247\" (UID: \"600fd20e-5eba-4886-9bda-68a5c995d9a8\") " pod="openshift-insights/insights-runtime-extractor-jk247"
Jul 07 16:47:15.845616 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:15.845504    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/600fd20e-5eba-4886-9bda-68a5c995d9a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jk247\" (UID: \"600fd20e-5eba-4886-9bda-68a5c995d9a8\") " pod="openshift-insights/insights-runtime-extractor-jk247"
Jul 07 16:47:15.870473 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:15.870432    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:15.946376 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:15.946334    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs8tm\" (UniqueName: \"kubernetes.io/projected/600fd20e-5eba-4886-9bda-68a5c995d9a8-kube-api-access-bs8tm\") pod \"insights-runtime-extractor-jk247\" (UID: \"600fd20e-5eba-4886-9bda-68a5c995d9a8\") " pod="openshift-insights/insights-runtime-extractor-jk247"
Jul 07 16:47:15.946566 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:15.946389    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/600fd20e-5eba-4886-9bda-68a5c995d9a8-data-volume\") pod \"insights-runtime-extractor-jk247\" (UID: \"600fd20e-5eba-4886-9bda-68a5c995d9a8\") " pod="openshift-insights/insights-runtime-extractor-jk247"
Jul 07 16:47:15.946566 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:15.946443    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/600fd20e-5eba-4886-9bda-68a5c995d9a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jk247\" (UID: \"600fd20e-5eba-4886-9bda-68a5c995d9a8\") " pod="openshift-insights/insights-runtime-extractor-jk247"
Jul 07 16:47:15.946566 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:15.946476    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/600fd20e-5eba-4886-9bda-68a5c995d9a8-crio-socket\") pod \"insights-runtime-extractor-jk247\" (UID: \"600fd20e-5eba-4886-9bda-68a5c995d9a8\") " pod="openshift-insights/insights-runtime-extractor-jk247"
Jul 07 16:47:15.946566 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:15.946500    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/600fd20e-5eba-4886-9bda-68a5c995d9a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jk247\" (UID: \"600fd20e-5eba-4886-9bda-68a5c995d9a8\") " pod="openshift-insights/insights-runtime-extractor-jk247"
Jul 07 16:47:15.946750 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:47:15.946641    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 16:47:15.946750 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:47:15.946736    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/600fd20e-5eba-4886-9bda-68a5c995d9a8-kube-rbac-proxy-cm podName:600fd20e-5eba-4886-9bda-68a5c995d9a8 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:16.446716497 +0000 UTC m=+85.047925198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/600fd20e-5eba-4886-9bda-68a5c995d9a8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jk247" (UID: "600fd20e-5eba-4886-9bda-68a5c995d9a8") : configmap "kube-rbac-proxy" not found
Jul 07 16:47:15.947374 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:15.947321    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/600fd20e-5eba-4886-9bda-68a5c995d9a8-crio-socket\") pod \"insights-runtime-extractor-jk247\" (UID: \"600fd20e-5eba-4886-9bda-68a5c995d9a8\") " pod="openshift-insights/insights-runtime-extractor-jk247"
Jul 07 16:47:15.947374 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:47:15.947357    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 16:47:15.947547 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:15.947408    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/600fd20e-5eba-4886-9bda-68a5c995d9a8-data-volume\") pod \"insights-runtime-extractor-jk247\" (UID: \"600fd20e-5eba-4886-9bda-68a5c995d9a8\") " pod="openshift-insights/insights-runtime-extractor-jk247"
Jul 07 16:47:15.947547 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:47:15.947415    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/600fd20e-5eba-4886-9bda-68a5c995d9a8-insights-runtime-extractor-tls podName:600fd20e-5eba-4886-9bda-68a5c995d9a8 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:16.447396998 +0000 UTC m=+85.048605680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/600fd20e-5eba-4886-9bda-68a5c995d9a8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jk247" (UID: "600fd20e-5eba-4886-9bda-68a5c995d9a8") : secret "insights-runtime-extractor-tls" not found
Jul 07 16:47:15.956406 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:15.956375    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs8tm\" (UniqueName: \"kubernetes.io/projected/600fd20e-5eba-4886-9bda-68a5c995d9a8-kube-api-access-bs8tm\") pod \"insights-runtime-extractor-jk247\" (UID: \"600fd20e-5eba-4886-9bda-68a5c995d9a8\") " pod="openshift-insights/insights-runtime-extractor-jk247"
Jul 07 16:47:16.451362 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:16.451324    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/600fd20e-5eba-4886-9bda-68a5c995d9a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jk247\" (UID: \"600fd20e-5eba-4886-9bda-68a5c995d9a8\") " pod="openshift-insights/insights-runtime-extractor-jk247"
Jul 07 16:47:16.451549 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:16.451454    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/600fd20e-5eba-4886-9bda-68a5c995d9a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jk247\" (UID: \"600fd20e-5eba-4886-9bda-68a5c995d9a8\") " pod="openshift-insights/insights-runtime-extractor-jk247"
Jul 07 16:47:16.451549 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:47:16.451465    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 16:47:16.451549 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:47:16.451527    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/600fd20e-5eba-4886-9bda-68a5c995d9a8-kube-rbac-proxy-cm podName:600fd20e-5eba-4886-9bda-68a5c995d9a8 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:17.451509434 +0000 UTC m=+86.052718116 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/600fd20e-5eba-4886-9bda-68a5c995d9a8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jk247" (UID: "600fd20e-5eba-4886-9bda-68a5c995d9a8") : configmap "kube-rbac-proxy" not found
Jul 07 16:47:16.451700 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:47:16.451557    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 16:47:16.451700 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:47:16.451606    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/600fd20e-5eba-4886-9bda-68a5c995d9a8-insights-runtime-extractor-tls podName:600fd20e-5eba-4886-9bda-68a5c995d9a8 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:17.451590617 +0000 UTC m=+86.052799317 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/600fd20e-5eba-4886-9bda-68a5c995d9a8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jk247" (UID: "600fd20e-5eba-4886-9bda-68a5c995d9a8") : secret "insights-runtime-extractor-tls" not found
Jul 07 16:47:17.224666 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:17.224580    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-dcrfk" event={"ID":"f45b3a44-6bd5-4cee-912e-5edc4d056972","Type":"ContainerStarted","Data":"73bca516dce8d0764a95f986014c811e5f03b457c5e03b6dbd3891a83987f7ba"}
Jul 07 16:47:17.245910 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:17.245861    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-dcrfk" podStartSLOduration=1.4858454380000001 podStartE2EDuration="3.245843154s" totalImagesPullingTime="1.759997716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:47:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:47:17.245353748 +0000 UTC m=+85.846562454" watchObservedRunningTime="2026-07-07 16:47:17.245843154 +0000 UTC m=+85.847051894"
Jul 07 16:47:17.458748 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:17.458700    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/600fd20e-5eba-4886-9bda-68a5c995d9a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jk247\" (UID: \"600fd20e-5eba-4886-9bda-68a5c995d9a8\") " pod="openshift-insights/insights-runtime-extractor-jk247"
Jul 07 16:47:17.458925 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:47:17.458826    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 16:47:17.458925 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:17.458830    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/600fd20e-5eba-4886-9bda-68a5c995d9a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jk247\" (UID: \"600fd20e-5eba-4886-9bda-68a5c995d9a8\") " pod="openshift-insights/insights-runtime-extractor-jk247"
Jul 07 16:47:17.458925 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:47:17.458900    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/600fd20e-5eba-4886-9bda-68a5c995d9a8-kube-rbac-proxy-cm podName:600fd20e-5eba-4886-9bda-68a5c995d9a8 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:19.458881216 +0000 UTC m=+88.060089902 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/600fd20e-5eba-4886-9bda-68a5c995d9a8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jk247" (UID: "600fd20e-5eba-4886-9bda-68a5c995d9a8") : configmap "kube-rbac-proxy" not found
Jul 07 16:47:17.458925 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:47:17.458905    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 16:47:17.459124 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:47:17.458955    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/600fd20e-5eba-4886-9bda-68a5c995d9a8-insights-runtime-extractor-tls podName:600fd20e-5eba-4886-9bda-68a5c995d9a8 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:19.45894384 +0000 UTC m=+88.060152525 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/600fd20e-5eba-4886-9bda-68a5c995d9a8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jk247" (UID: "600fd20e-5eba-4886-9bda-68a5c995d9a8") : secret "insights-runtime-extractor-tls" not found
Jul 07 16:47:17.643550 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:17.643514    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-v9xbq"]
Jul 07 16:47:17.648589 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:17.648552    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-v9xbq"
Jul 07 16:47:17.651258 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:17.651231    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 07 16:47:17.651258 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:17.651243    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 07 16:47:17.651488 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:17.651321    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 07 16:47:17.651488 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:17.651324    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-lpczp\""
Jul 07 16:47:17.651606 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:17.651524    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 07 16:47:17.657469 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:17.657442    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-v9xbq"]
Jul 07 16:47:17.755281 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:17.755250    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-9v7lb"]
Jul 07 16:47:17.755990 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:17.755969    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ptgp6_3d18f299-6f72-46ff-abf2-0ccf120fc875/dns-node-resolver/0.log"
Jul 07 16:47:17.757283 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:17.757268    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-9v7lb"
Jul 07 16:47:17.760210 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:17.760191    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-rqlhh\""
Jul 07 16:47:17.761121 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:17.761098    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7b9d7134-446c-4c14-a1fb-92202cdde701-config\") pod \"service-ca-operator-5d46d9ffd4-v9xbq\" (UID: \"7b9d7134-446c-4c14-a1fb-92202cdde701\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-v9xbq"
Jul 07 16:47:17.761219 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:17.761132    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97hhv\" (UniqueName: \"kubernetes.io/projected/7b9d7134-446c-4c14-a1fb-92202cdde701-kube-api-access-97hhv\") pod \"service-ca-operator-5d46d9ffd4-v9xbq\" (UID: \"7b9d7134-446c-4c14-a1fb-92202cdde701\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-v9xbq"
Jul 07 16:47:17.761219 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:17.761192    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b9d7134-446c-4c14-a1fb-92202cdde701-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-v9xbq\" (UID: \"7b9d7134-446c-4c14-a1fb-92202cdde701\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-v9xbq"
Jul 07 16:47:17.766644 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:17.766619    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-9v7lb"]
Jul 07 16:47:17.862013 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:17.861969    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b9d7134-446c-4c14-a1fb-92202cdde701-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-v9xbq\" (UID: \"7b9d7134-446c-4c14-a1fb-92202cdde701\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-v9xbq"
Jul 07 16:47:17.862211 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:17.862072    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs87j\" (UniqueName: \"kubernetes.io/projected/7530ad6b-84c6-48bd-b307-e7f5e3ce4a08-kube-api-access-hs87j\") pod \"network-check-source-989f8b8dc-9v7lb\" (UID: \"7530ad6b-84c6-48bd-b307-e7f5e3ce4a08\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-9v7lb"
Jul 07 16:47:17.862211 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:17.862118    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7b9d7134-446c-4c14-a1fb-92202cdde701-config\") pod \"service-ca-operator-5d46d9ffd4-v9xbq\" (UID: \"7b9d7134-446c-4c14-a1fb-92202cdde701\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-v9xbq"
Jul 07 16:47:17.862211 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:17.862135    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97hhv\" (UniqueName: \"kubernetes.io/projected/7b9d7134-446c-4c14-a1fb-92202cdde701-kube-api-access-97hhv\") pod \"service-ca-operator-5d46d9ffd4-v9xbq\" (UID: \"7b9d7134-446c-4c14-a1fb-92202cdde701\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-v9xbq"
Jul 07 16:47:17.862784 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:17.862742    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7b9d7134-446c-4c14-a1fb-92202cdde701-config\") pod \"service-ca-operator-5d46d9ffd4-v9xbq\" (UID: \"7b9d7134-446c-4c14-a1fb-92202cdde701\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-v9xbq"
Jul 07 16:47:17.864947 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:17.864925    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b9d7134-446c-4c14-a1fb-92202cdde701-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-v9xbq\" (UID: \"7b9d7134-446c-4c14-a1fb-92202cdde701\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-v9xbq"
Jul 07 16:47:17.872783 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:17.872759    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97hhv\" (UniqueName: \"kubernetes.io/projected/7b9d7134-446c-4c14-a1fb-92202cdde701-kube-api-access-97hhv\") pod \"service-ca-operator-5d46d9ffd4-v9xbq\" (UID: \"7b9d7134-446c-4c14-a1fb-92202cdde701\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-v9xbq"
Jul 07 16:47:17.958633 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:17.958538    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-v9xbq"
Jul 07 16:47:17.962869 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:17.962842    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs87j\" (UniqueName: \"kubernetes.io/projected/7530ad6b-84c6-48bd-b307-e7f5e3ce4a08-kube-api-access-hs87j\") pod \"network-check-source-989f8b8dc-9v7lb\" (UID: \"7530ad6b-84c6-48bd-b307-e7f5e3ce4a08\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-9v7lb"
Jul 07 16:47:17.975341 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:17.975305    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs87j\" (UniqueName: \"kubernetes.io/projected/7530ad6b-84c6-48bd-b307-e7f5e3ce4a08-kube-api-access-hs87j\") pod \"network-check-source-989f8b8dc-9v7lb\" (UID: \"7530ad6b-84c6-48bd-b307-e7f5e3ce4a08\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-9v7lb"
Jul 07 16:47:18.063178 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:18.063144    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:18.065937 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:18.065916    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-9v7lb"
Jul 07 16:47:18.084393 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:18.084366    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-2q4wx"]
Jul 07 16:47:18.086682 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:18.086656    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-2q4wx"
Jul 07 16:47:18.089408 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:18.089390    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 07 16:47:18.089530 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:18.089421    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-pcwzk\""
Jul 07 16:47:18.089530 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:18.089390    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 07 16:47:18.096423 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:18.096400    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-2q4wx"]
Jul 07 16:47:18.102836 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:18.102801    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-v9xbq"]
Jul 07 16:47:18.105534 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:47:18.105510    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b9d7134_446c_4c14_a1fb_92202cdde701.slice/crio-f27dcca1ec7db6b3334e2bdad7edb0dc8be3d11a586c0cc3c046a145f4eff65b WatchSource:0}: Error finding container f27dcca1ec7db6b3334e2bdad7edb0dc8be3d11a586c0cc3c046a145f4eff65b: Status 404 returned error can't find the container with id f27dcca1ec7db6b3334e2bdad7edb0dc8be3d11a586c0cc3c046a145f4eff65b
Jul 07 16:47:18.147980 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:18.147947    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:18.164147 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:18.164114    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5nwh\" (UniqueName: \"kubernetes.io/projected/edc60915-17c5-41d2-b611-4d875a7f18a0-kube-api-access-f5nwh\") pod \"migrator-d57d58cc9-2q4wx\" (UID: \"edc60915-17c5-41d2-b611-4d875a7f18a0\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-2q4wx"
Jul 07 16:47:18.199069 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:18.199024    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-9v7lb"]
Jul 07 16:47:18.202210 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:47:18.202186    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7530ad6b_84c6_48bd_b307_e7f5e3ce4a08.slice/crio-ac18e0b9d0b953bb06925d9c08a641c3cad63a3d2c0a2a3e7abd15af88188245 WatchSource:0}: Error finding container ac18e0b9d0b953bb06925d9c08a641c3cad63a3d2c0a2a3e7abd15af88188245: Status 404 returned error can't find the container with id ac18e0b9d0b953bb06925d9c08a641c3cad63a3d2c0a2a3e7abd15af88188245
Jul 07 16:47:18.227599 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:18.227572    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-9v7lb" event={"ID":"7530ad6b-84c6-48bd-b307-e7f5e3ce4a08","Type":"ContainerStarted","Data":"ac18e0b9d0b953bb06925d9c08a641c3cad63a3d2c0a2a3e7abd15af88188245"}
Jul 07 16:47:18.228674 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:18.228644    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-v9xbq" event={"ID":"7b9d7134-446c-4c14-a1fb-92202cdde701","Type":"ContainerStarted","Data":"f27dcca1ec7db6b3334e2bdad7edb0dc8be3d11a586c0cc3c046a145f4eff65b"}
Jul 07 16:47:18.248391 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:18.248339    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:18.264683 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:18.264658    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5nwh\" (UniqueName: \"kubernetes.io/projected/edc60915-17c5-41d2-b611-4d875a7f18a0-kube-api-access-f5nwh\") pod \"migrator-d57d58cc9-2q4wx\" (UID: \"edc60915-17c5-41d2-b611-4d875a7f18a0\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-2q4wx"
Jul 07 16:47:18.276892 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:18.276863    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5nwh\" (UniqueName: \"kubernetes.io/projected/edc60915-17c5-41d2-b611-4d875a7f18a0-kube-api-access-f5nwh\") pod \"migrator-d57d58cc9-2q4wx\" (UID: \"edc60915-17c5-41d2-b611-4d875a7f18a0\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-2q4wx"
Jul 07 16:47:18.396927 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:18.396886    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-2q4wx"
Jul 07 16:47:18.539387 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:18.539352    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-2q4wx"]
Jul 07 16:47:18.542562 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:47:18.542536    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc60915_17c5_41d2_b611_4d875a7f18a0.slice/crio-fd90834136253a2477d3c223a02689b2d22e280e3e6e24bb66d438a156905365 WatchSource:0}: Error finding container fd90834136253a2477d3c223a02689b2d22e280e3e6e24bb66d438a156905365: Status 404 returned error can't find the container with id fd90834136253a2477d3c223a02689b2d22e280e3e6e24bb66d438a156905365
Jul 07 16:47:18.556115 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:18.556097    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cpbvp_266c4079-c003-4676-a480-fb0e19a14cce/node-ca/0.log"
Jul 07 16:47:19.233317 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:19.233275    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-2q4wx" event={"ID":"edc60915-17c5-41d2-b611-4d875a7f18a0","Type":"ContainerStarted","Data":"fd90834136253a2477d3c223a02689b2d22e280e3e6e24bb66d438a156905365"}
Jul 07 16:47:19.235252 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:19.235180    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-9v7lb" event={"ID":"7530ad6b-84c6-48bd-b307-e7f5e3ce4a08","Type":"ContainerStarted","Data":"dd431e3bed107a972c39a03520baf7183ff9c454ffe1e987cf270723523ab295"}
Jul 07 16:47:19.352990 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:19.352962    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ml4f2_b8e32c6b-206f-4882-b1c0-066d26486d41/serve-healthcheck-canary/0.log"
Jul 07 16:47:19.477370 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:19.477332    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/600fd20e-5eba-4886-9bda-68a5c995d9a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jk247\" (UID: \"600fd20e-5eba-4886-9bda-68a5c995d9a8\") " pod="openshift-insights/insights-runtime-extractor-jk247"
Jul 07 16:47:19.477576 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:19.477455    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/600fd20e-5eba-4886-9bda-68a5c995d9a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jk247\" (UID: \"600fd20e-5eba-4886-9bda-68a5c995d9a8\") " pod="openshift-insights/insights-runtime-extractor-jk247"
Jul 07 16:47:19.477576 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:47:19.477509    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 16:47:19.477576 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:47:19.477572    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 16:47:19.477739 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:47:19.477582    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/600fd20e-5eba-4886-9bda-68a5c995d9a8-insights-runtime-extractor-tls podName:600fd20e-5eba-4886-9bda-68a5c995d9a8 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:23.477561599 +0000 UTC m=+92.078770295 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/600fd20e-5eba-4886-9bda-68a5c995d9a8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jk247" (UID: "600fd20e-5eba-4886-9bda-68a5c995d9a8") : secret "insights-runtime-extractor-tls" not found
Jul 07 16:47:19.477739 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:47:19.477636    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/600fd20e-5eba-4886-9bda-68a5c995d9a8-kube-rbac-proxy-cm podName:600fd20e-5eba-4886-9bda-68a5c995d9a8 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:23.477621763 +0000 UTC m=+92.078830460 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/600fd20e-5eba-4886-9bda-68a5c995d9a8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jk247" (UID: "600fd20e-5eba-4886-9bda-68a5c995d9a8") : configmap "kube-rbac-proxy" not found
Jul 07 16:47:20.240120 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:20.240071    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-2q4wx" event={"ID":"edc60915-17c5-41d2-b611-4d875a7f18a0","Type":"ContainerStarted","Data":"ada45b6d03c4973a0e38ec764b7e77875a8fd54781859e36e6e802c741b9fe0f"}
Jul 07 16:47:20.240495 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:20.240150    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-2q4wx" event={"ID":"edc60915-17c5-41d2-b611-4d875a7f18a0","Type":"ContainerStarted","Data":"bacd0e56921ab17872ebef4bacbc7f572900703bcad69a7bd7c5dfac69327bbb"}
Jul 07 16:47:20.264381 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:20.264333    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-9v7lb" podStartSLOduration=3.264318142 podStartE2EDuration="3.264318142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:47:19.254926593 +0000 UTC m=+87.856135297" watchObservedRunningTime="2026-07-07 16:47:20.264318142 +0000 UTC m=+88.865526905"
Jul 07 16:47:20.264541 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:20.264411    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-2q4wx" podStartSLOduration=1.236702811 podStartE2EDuration="2.264405924s" totalImagesPullingTime="1.027703113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:47:20.262815639 +0000 UTC m=+88.864024342" watchObservedRunningTime="2026-07-07 16:47:20.264405924 +0000 UTC m=+88.865614628"
Jul 07 16:47:20.358889 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:20.358853    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:20.433774 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:20.433740    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:21.245593 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:21.245557    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-v9xbq" event={"ID":"7b9d7134-446c-4c14-a1fb-92202cdde701","Type":"ContainerStarted","Data":"847f86386a357e2635e855505f4084107cb7d92bfe355b681dc5bafdaae8abdc"}
Jul 07 16:47:21.266095 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:21.266015    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-v9xbq" podStartSLOduration=2.181344147 podStartE2EDuration="4.265998101s" totalImagesPullingTime="2.084653954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:47:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:47:21.265627947 +0000 UTC m=+89.866836650" watchObservedRunningTime="2026-07-07 16:47:21.265998101 +0000 UTC m=+89.867206804"
Jul 07 16:47:21.659659 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:21.659626    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:21.754945 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:21.754903    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:21.869070 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:21.869014    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:21.971946 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:21.971869    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:22.337539 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:22.337456    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:22.445976 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:22.445938    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:23.510369 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:23.510329    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/600fd20e-5eba-4886-9bda-68a5c995d9a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jk247\" (UID: \"600fd20e-5eba-4886-9bda-68a5c995d9a8\") " pod="openshift-insights/insights-runtime-extractor-jk247"
Jul 07 16:47:23.510763 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:23.510387    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/600fd20e-5eba-4886-9bda-68a5c995d9a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jk247\" (UID: \"600fd20e-5eba-4886-9bda-68a5c995d9a8\") " pod="openshift-insights/insights-runtime-extractor-jk247"
Jul 07 16:47:23.510763 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:47:23.510457    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 16:47:23.510763 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:47:23.510470    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 16:47:23.510763 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:47:23.510524    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/600fd20e-5eba-4886-9bda-68a5c995d9a8-kube-rbac-proxy-cm podName:600fd20e-5eba-4886-9bda-68a5c995d9a8 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:31.510506615 +0000 UTC m=+100.111715298 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/600fd20e-5eba-4886-9bda-68a5c995d9a8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jk247" (UID: "600fd20e-5eba-4886-9bda-68a5c995d9a8") : configmap "kube-rbac-proxy" not found
Jul 07 16:47:23.510763 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:47:23.510539    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/600fd20e-5eba-4886-9bda-68a5c995d9a8-insights-runtime-extractor-tls podName:600fd20e-5eba-4886-9bda-68a5c995d9a8 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:31.5105328 +0000 UTC m=+100.111741482 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/600fd20e-5eba-4886-9bda-68a5c995d9a8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jk247" (UID: "600fd20e-5eba-4886-9bda-68a5c995d9a8") : secret "insights-runtime-extractor-tls" not found
Jul 07 16:47:23.938994 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:23.938962    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-nvkb6"]
Jul 07 16:47:23.941293 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:23.941276    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-nvkb6"
Jul 07 16:47:23.944203 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:23.944179    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 07 16:47:23.945508 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:23.945484    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 07 16:47:23.945606 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:23.945509    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 07 16:47:23.945606 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:23.945568    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 07 16:47:23.945704 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:23.945609    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-nb7bj\""
Jul 07 16:47:23.952893 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:23.952868    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-nvkb6"]
Jul 07 16:47:24.016372 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:24.016332    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcz6l\" (UniqueName: \"kubernetes.io/projected/0a1311fe-3876-4d1f-ad53-d1fcaede2eda-kube-api-access-hcz6l\") pod \"service-ca-758f599c5-nvkb6\" (UID: \"0a1311fe-3876-4d1f-ad53-d1fcaede2eda\") " pod="openshift-service-ca/service-ca-758f599c5-nvkb6"
Jul 07 16:47:24.016542 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:24.016399    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0a1311fe-3876-4d1f-ad53-d1fcaede2eda-signing-key\") pod \"service-ca-758f599c5-nvkb6\" (UID: \"0a1311fe-3876-4d1f-ad53-d1fcaede2eda\") " pod="openshift-service-ca/service-ca-758f599c5-nvkb6"
Jul 07 16:47:24.016542 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:24.016477    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0a1311fe-3876-4d1f-ad53-d1fcaede2eda-signing-cabundle\") pod \"service-ca-758f599c5-nvkb6\" (UID: \"0a1311fe-3876-4d1f-ad53-d1fcaede2eda\") " pod="openshift-service-ca/service-ca-758f599c5-nvkb6"
Jul 07 16:47:24.058302 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:24.058263    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:24.117585 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:24.117543    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcz6l\" (UniqueName: \"kubernetes.io/projected/0a1311fe-3876-4d1f-ad53-d1fcaede2eda-kube-api-access-hcz6l\") pod \"service-ca-758f599c5-nvkb6\" (UID: \"0a1311fe-3876-4d1f-ad53-d1fcaede2eda\") " pod="openshift-service-ca/service-ca-758f599c5-nvkb6"
Jul 07 16:47:24.117734 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:24.117616    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0a1311fe-3876-4d1f-ad53-d1fcaede2eda-signing-key\") pod \"service-ca-758f599c5-nvkb6\" (UID: \"0a1311fe-3876-4d1f-ad53-d1fcaede2eda\") " pod="openshift-service-ca/service-ca-758f599c5-nvkb6"
Jul 07 16:47:24.117734 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:24.117661    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0a1311fe-3876-4d1f-ad53-d1fcaede2eda-signing-cabundle\") pod \"service-ca-758f599c5-nvkb6\" (UID: \"0a1311fe-3876-4d1f-ad53-d1fcaede2eda\") " pod="openshift-service-ca/service-ca-758f599c5-nvkb6"
Jul 07 16:47:24.118336 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:24.118310    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0a1311fe-3876-4d1f-ad53-d1fcaede2eda-signing-cabundle\") pod \"service-ca-758f599c5-nvkb6\" (UID: \"0a1311fe-3876-4d1f-ad53-d1fcaede2eda\") " pod="openshift-service-ca/service-ca-758f599c5-nvkb6"
Jul 07 16:47:24.120023 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:24.120006    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0a1311fe-3876-4d1f-ad53-d1fcaede2eda-signing-key\") pod \"service-ca-758f599c5-nvkb6\" (UID: \"0a1311fe-3876-4d1f-ad53-d1fcaede2eda\") " pod="openshift-service-ca/service-ca-758f599c5-nvkb6"
Jul 07 16:47:24.128723 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:24.128694    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcz6l\" (UniqueName: \"kubernetes.io/projected/0a1311fe-3876-4d1f-ad53-d1fcaede2eda-kube-api-access-hcz6l\") pod \"service-ca-758f599c5-nvkb6\" (UID: \"0a1311fe-3876-4d1f-ad53-d1fcaede2eda\") " pod="openshift-service-ca/service-ca-758f599c5-nvkb6"
Jul 07 16:47:24.155033 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:24.154997    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:24.250568 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:24.250478    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-nvkb6"
Jul 07 16:47:24.376097 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:24.376040    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-nvkb6"]
Jul 07 16:47:24.379247 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:47:24.379221    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a1311fe_3876_4d1f_ad53_d1fcaede2eda.slice/crio-60c5b5bbc89e7becfe1e27af9fb239dbfbd41b07d152ccf7ed0176f419c99f20 WatchSource:0}: Error finding container 60c5b5bbc89e7becfe1e27af9fb239dbfbd41b07d152ccf7ed0176f419c99f20: Status 404 returned error can't find the container with id 60c5b5bbc89e7becfe1e27af9fb239dbfbd41b07d152ccf7ed0176f419c99f20
Jul 07 16:47:25.169489 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:25.169452    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:25.262193 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:25.262153    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-nvkb6" event={"ID":"0a1311fe-3876-4d1f-ad53-d1fcaede2eda","Type":"ContainerStarted","Data":"9a27f953419c2e3fbff03bb97f1bcaef68a7f9509b80586841b24d113b0897a8"}
Jul 07 16:47:25.262389 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:25.262211    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-nvkb6" event={"ID":"0a1311fe-3876-4d1f-ad53-d1fcaede2eda","Type":"ContainerStarted","Data":"60c5b5bbc89e7becfe1e27af9fb239dbfbd41b07d152ccf7ed0176f419c99f20"}
Jul 07 16:47:25.292668 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:25.292616    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-nvkb6" podStartSLOduration=2.292598547 podStartE2EDuration="2.292598547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:47:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:47:25.292567348 +0000 UTC m=+93.893776053" watchObservedRunningTime="2026-07-07 16:47:25.292598547 +0000 UTC m=+93.893807248"
Jul 07 16:47:25.412340 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:25.412296    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:25.683257 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:25.683219    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:25.866788 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:25.866750    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:26.015164 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:26.015073    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:26.123873 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:26.123838    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:26.680623 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:26.680568    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:27.295406 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:27.295365    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:27.682392 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:27.682350    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:28.094100 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:28.094038    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:28.297656 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:28.297615    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:28.651652 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:28.651617    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:28.748334 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:28.748304    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:29.031832 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:29.031732    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:29.132573 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:29.132536    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:29.343717 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:29.343685    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:29.493502 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:29.493458    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:29.887095 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:29.887059    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:30.081890 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:30.081845    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:30.149115 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:30.149009    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:30.288435 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:30.288404    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:30.688652 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:30.688596    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:31.093939 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:31.093910    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:31.198599 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:31.198565    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:31.301578 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:31.301527    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:31.582830 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:31.582731    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/600fd20e-5eba-4886-9bda-68a5c995d9a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jk247\" (UID: \"600fd20e-5eba-4886-9bda-68a5c995d9a8\") " pod="openshift-insights/insights-runtime-extractor-jk247"
Jul 07 16:47:31.582830 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:31.582810    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/600fd20e-5eba-4886-9bda-68a5c995d9a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jk247\" (UID: \"600fd20e-5eba-4886-9bda-68a5c995d9a8\") " pod="openshift-insights/insights-runtime-extractor-jk247"
Jul 07 16:47:31.583024 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:47:31.582959    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 16:47:31.583072 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:47:31.583061    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/600fd20e-5eba-4886-9bda-68a5c995d9a8-insights-runtime-extractor-tls podName:600fd20e-5eba-4886-9bda-68a5c995d9a8 nodeName:}" failed. No retries permitted until 2026-07-07 16:47:47.583022588 +0000 UTC m=+116.184231269 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/600fd20e-5eba-4886-9bda-68a5c995d9a8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jk247" (UID: "600fd20e-5eba-4886-9bda-68a5c995d9a8") : secret "insights-runtime-extractor-tls" not found
Jul 07 16:47:31.583263 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:31.583246    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/600fd20e-5eba-4886-9bda-68a5c995d9a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jk247\" (UID: \"600fd20e-5eba-4886-9bda-68a5c995d9a8\") " pod="openshift-insights/insights-runtime-extractor-jk247"
Jul 07 16:47:31.721931 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:31.721893    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:32.101097 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:32.101026    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:32.187355 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:32.187320    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-registry-tls\") pod \"image-registry-54b88fb4c5-zhjwb\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") " pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:47:32.190076 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:32.190020    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-registry-tls\") pod \"image-registry-54b88fb4c5-zhjwb\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") " pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:47:32.288332 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:32.288298    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3b39298-a197-40cb-8363-912fdc7dbcc8-metrics-tls\") pod \"dns-default-wv8wf\" (UID: \"f3b39298-a197-40cb-8363-912fdc7dbcc8\") " pod="openshift-dns/dns-default-wv8wf"
Jul 07 16:47:32.290865 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:32.290829    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3b39298-a197-40cb-8363-912fdc7dbcc8-metrics-tls\") pod \"dns-default-wv8wf\" (UID: \"f3b39298-a197-40cb-8363-912fdc7dbcc8\") " pod="openshift-dns/dns-default-wv8wf"
Jul 07 16:47:32.291437 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:32.291406    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:32.494457 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:32.494428    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xv5k5\""
Jul 07 16:47:32.499527 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:32.499506    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:47:32.512952 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:32.512919    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:32.538068 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:32.538022    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9pvjc\""
Jul 07 16:47:32.545038 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:32.545012    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wv8wf"
Jul 07 16:47:32.647718 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:32.647689    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54b88fb4c5-zhjwb"]
Jul 07 16:47:32.654715 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:47:32.654686    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cbfbc07_f19d_49f8_8cb3_a1ea8e712eb2.slice/crio-b35bda9feacc2bd7a4f8ea6e5fff0bca21f9fa43d38e3d994790d3ce0130ffe8 WatchSource:0}: Error finding container b35bda9feacc2bd7a4f8ea6e5fff0bca21f9fa43d38e3d994790d3ce0130ffe8: Status 404 returned error can't find the container with id b35bda9feacc2bd7a4f8ea6e5fff0bca21f9fa43d38e3d994790d3ce0130ffe8
Jul 07 16:47:32.659622 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:32.659594    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:32.689561 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:32.689538    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wv8wf"]
Jul 07 16:47:32.692351 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:47:32.692325    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b39298_a197_40cb_8363_912fdc7dbcc8.slice/crio-5d88f72d6742934707128d97b1f3b677211cb747eef5bda4ee18befcea9edab1 WatchSource:0}: Error finding container 5d88f72d6742934707128d97b1f3b677211cb747eef5bda4ee18befcea9edab1: Status 404 returned error can't find the container with id 5d88f72d6742934707128d97b1f3b677211cb747eef5bda4ee18befcea9edab1
Jul 07 16:47:32.906521 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:32.906427    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:33.292584 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:33.292491    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb" event={"ID":"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2","Type":"ContainerStarted","Data":"5928cbf409274726e135f3f1b4364d3a261eb8c217186800723c102db8df5059"}
Jul 07 16:47:33.292584 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:33.292551    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb" event={"ID":"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2","Type":"ContainerStarted","Data":"b35bda9feacc2bd7a4f8ea6e5fff0bca21f9fa43d38e3d994790d3ce0130ffe8"}
Jul 07 16:47:33.293107 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:33.292773    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:47:33.293910 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:33.293886    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wv8wf" event={"ID":"f3b39298-a197-40cb-8363-912fdc7dbcc8","Type":"ContainerStarted","Data":"5d88f72d6742934707128d97b1f3b677211cb747eef5bda4ee18befcea9edab1"}
Jul 07 16:47:33.298185 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:33.298158    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:33.326443 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:33.326382    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb" podStartSLOduration=100.326365298 podStartE2EDuration="1m40.326365298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:47:33.32586669 +0000 UTC m=+101.927075448" watchObservedRunningTime="2026-07-07 16:47:33.326365298 +0000 UTC m=+101.927574001"
Jul 07 16:47:33.503961 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:33.503922    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:33.888870 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:33.888834    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:34.193902 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:34.193825    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-l6pr5"
Jul 07 16:47:34.485069 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:34.484942    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:34.894248 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:34.894221    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:35.110229 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:35.110194    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:35.302869 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:35.302836    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wv8wf" event={"ID":"f3b39298-a197-40cb-8363-912fdc7dbcc8","Type":"ContainerStarted","Data":"8df7bc9fda984f55042356c56ff0cddd103405a2c0c60ba51502264a60bb8246"}
Jul 07 16:47:35.303033 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:35.302890    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wv8wf" event={"ID":"f3b39298-a197-40cb-8363-912fdc7dbcc8","Type":"ContainerStarted","Data":"2c5240d4db13014aad8d85f7001547c94abb0cf9d1615c5aa017dfa8d496eb4b"}
Jul 07 16:47:35.303033 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:35.302956    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wv8wf"
Jul 07 16:47:35.325717 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:35.325663    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wv8wf" podStartSLOduration=65.366103913 podStartE2EDuration="1m7.325646532s" totalImagesPullingTime="1.959542619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:46:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:47:35.324452957 +0000 UTC m=+103.925661663" watchObservedRunningTime="2026-07-07 16:47:35.325646532 +0000 UTC m=+103.926855213"
Jul 07 16:47:35.685530 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:35.685491    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:35.888692 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:35.888652    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:36.083367 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:36.083286    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:36.368271 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:36.368236    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:36.469196 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:36.469158    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:36.544990 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:36.544952    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:36.661565 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:36.661487    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:36.850492 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:36.850456    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:37.049513 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:37.049435    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:37.160498 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:37.160463    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:37.238624 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:37.238579    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:37.307609 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:37.307524    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:37.445329 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:37.445290    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:37.516310 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:37.516279    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:38.063610 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:38.063571    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:38.164708 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:38.164676    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:38.449993 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:38.449956    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:39.079213 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:39.079180    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:39.146511 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:39.146468    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:45.308916 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:45.308886    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wv8wf"
Jul 07 16:47:45.673671 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:45.673633    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:45.771475 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:45.771424    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:47.042932 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:47.042896    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:47.153468 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:47.153432    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:47.278013 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:47.277974    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:47.480546 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:47.480509    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:47.569748 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:47.569695    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:47.605062 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:47.605014    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/600fd20e-5eba-4886-9bda-68a5c995d9a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jk247\" (UID: \"600fd20e-5eba-4886-9bda-68a5c995d9a8\") " pod="openshift-insights/insights-runtime-extractor-jk247"
Jul 07 16:47:47.607391 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:47.607370    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/600fd20e-5eba-4886-9bda-68a5c995d9a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jk247\" (UID: \"600fd20e-5eba-4886-9bda-68a5c995d9a8\") " pod="openshift-insights/insights-runtime-extractor-jk247"
Jul 07 16:47:47.668208 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:47.668166    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:47.731706 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:47.731623    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:47.867007 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:47.866973    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jk247"
Jul 07 16:47:47.997880 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:47.997801    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jk247"]
Jul 07 16:47:48.001027 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:47:48.000999    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod600fd20e_5eba_4886_9bda_68a5c995d9a8.slice/crio-4d2e300cf4d36998e8e173b611f4567d766cfa225f03df282fbd67b077ed8147 WatchSource:0}: Error finding container 4d2e300cf4d36998e8e173b611f4567d766cfa225f03df282fbd67b077ed8147: Status 404 returned error can't find the container with id 4d2e300cf4d36998e8e173b611f4567d766cfa225f03df282fbd67b077ed8147
Jul 07 16:47:48.346601 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:48.346567    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jk247" event={"ID":"600fd20e-5eba-4886-9bda-68a5c995d9a8","Type":"ContainerStarted","Data":"4d2e300cf4d36998e8e173b611f4567d766cfa225f03df282fbd67b077ed8147"}
Jul 07 16:47:49.319699 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.319665    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6zkdb"]
Jul 07 16:47:49.321817 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.321798    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6zkdb"
Jul 07 16:47:49.326988 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:47:49.326956    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"prometheus-operator-admission-webhook-d559d66d5-6zkdb\" is forbidden: User \"system:node:ip-10-0-133-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-133-184.ec2.internal' and this object" podUID="25fef8f4-e7f3-4274-a226-e07d12bc3efd" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6zkdb"
Jul 07 16:47:49.327159 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.327143    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-mkvgm\""
Jul 07 16:47:49.329349 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.329323    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 07 16:47:49.346751 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.346718    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-xvlzm"]
Jul 07 16:47:49.350503 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.350479    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-xvlzm"
Jul 07 16:47:49.352338 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.352314    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c9f468564-9bc2g"]
Jul 07 16:47:49.354219 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.354199    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6zkdb"]
Jul 07 16:47:49.354321 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.354290    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c9f468564-9bc2g"
Jul 07 16:47:49.355024 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.355005    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-fft6t\""
Jul 07 16:47:49.357676 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.357657    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 07 16:47:49.362524 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.362495    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 07 16:47:49.376882 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.376857    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 07 16:47:49.377007 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.376890    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 07 16:47:49.388279 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.388259    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 07 16:47:49.389724 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.389699    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2q9xq\""
Jul 07 16:47:49.389724 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.389716    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 07 16:47:49.390141 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.390122    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 07 16:47:49.408083 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.408034    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-xvlzm"]
Jul 07 16:47:49.411453 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.411422    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 07 16:47:49.420799 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.420766    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-oauth-serving-cert\") pod \"console-5c9f468564-9bc2g\" (UID: \"83acf15b-a80a-4a21-bcdb-08f6080ccb0f\") " pod="openshift-console/console-5c9f468564-9bc2g"
Jul 07 16:47:49.420970 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.420812    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-console-config\") pod \"console-5c9f468564-9bc2g\" (UID: \"83acf15b-a80a-4a21-bcdb-08f6080ccb0f\") " pod="openshift-console/console-5c9f468564-9bc2g"
Jul 07 16:47:49.420970 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.420843    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldktj\" (UniqueName: \"kubernetes.io/projected/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-kube-api-access-ldktj\") pod \"console-5c9f468564-9bc2g\" (UID: \"83acf15b-a80a-4a21-bcdb-08f6080ccb0f\") " pod="openshift-console/console-5c9f468564-9bc2g"
Jul 07 16:47:49.420970 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.420930    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-service-ca\") pod \"console-5c9f468564-9bc2g\" (UID: \"83acf15b-a80a-4a21-bcdb-08f6080ccb0f\") " pod="openshift-console/console-5c9f468564-9bc2g"
Jul 07 16:47:49.421189 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.420995    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-console-serving-cert\") pod \"console-5c9f468564-9bc2g\" (UID: \"83acf15b-a80a-4a21-bcdb-08f6080ccb0f\") " pod="openshift-console/console-5c9f468564-9bc2g"
Jul 07 16:47:49.421189 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.421071    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-console-oauth-config\") pod \"console-5c9f468564-9bc2g\" (UID: \"83acf15b-a80a-4a21-bcdb-08f6080ccb0f\") " pod="openshift-console/console-5c9f468564-9bc2g"
Jul 07 16:47:49.421189 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.421108    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/25fef8f4-e7f3-4274-a226-e07d12bc3efd-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-6zkdb\" (UID: \"25fef8f4-e7f3-4274-a226-e07d12bc3efd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6zkdb"
Jul 07 16:47:49.421189 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.421140    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-trusted-ca-bundle\") pod \"console-5c9f468564-9bc2g\" (UID: \"83acf15b-a80a-4a21-bcdb-08f6080ccb0f\") " pod="openshift-console/console-5c9f468564-9bc2g"
Jul 07 16:47:49.421429 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.421192    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9n5k\" (UniqueName: \"kubernetes.io/projected/c50b90e0-2467-43e9-ba20-83e720cab84d-kube-api-access-h9n5k\") pod \"downloads-685bfb5f64-xvlzm\" (UID: \"c50b90e0-2467-43e9-ba20-83e720cab84d\") " pod="openshift-console/downloads-685bfb5f64-xvlzm"
Jul 07 16:47:49.443833 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.443798    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c9f468564-9bc2g"]
Jul 07 16:47:49.466062 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.466007    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:49.532159 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.532114    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-service-ca\") pod \"console-5c9f468564-9bc2g\" (UID: \"83acf15b-a80a-4a21-bcdb-08f6080ccb0f\") " pod="openshift-console/console-5c9f468564-9bc2g"
Jul 07 16:47:49.532353 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.532187    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-console-serving-cert\") pod \"console-5c9f468564-9bc2g\" (UID: \"83acf15b-a80a-4a21-bcdb-08f6080ccb0f\") " pod="openshift-console/console-5c9f468564-9bc2g"
Jul 07 16:47:49.532353 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.532218    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-console-oauth-config\") pod \"console-5c9f468564-9bc2g\" (UID: \"83acf15b-a80a-4a21-bcdb-08f6080ccb0f\") " pod="openshift-console/console-5c9f468564-9bc2g"
Jul 07 16:47:49.532353 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.532238    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/25fef8f4-e7f3-4274-a226-e07d12bc3efd-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-6zkdb\" (UID: \"25fef8f4-e7f3-4274-a226-e07d12bc3efd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6zkdb"
Jul 07 16:47:49.532526 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.532357    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-trusted-ca-bundle\") pod \"console-5c9f468564-9bc2g\" (UID: \"83acf15b-a80a-4a21-bcdb-08f6080ccb0f\") " pod="openshift-console/console-5c9f468564-9bc2g"
Jul 07 16:47:49.532526 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.532395    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9n5k\" (UniqueName: \"kubernetes.io/projected/c50b90e0-2467-43e9-ba20-83e720cab84d-kube-api-access-h9n5k\") pod \"downloads-685bfb5f64-xvlzm\" (UID: \"c50b90e0-2467-43e9-ba20-83e720cab84d\") " pod="openshift-console/downloads-685bfb5f64-xvlzm"
Jul 07 16:47:49.532526 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.532451    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-oauth-serving-cert\") pod \"console-5c9f468564-9bc2g\" (UID: \"83acf15b-a80a-4a21-bcdb-08f6080ccb0f\") " pod="openshift-console/console-5c9f468564-9bc2g"
Jul 07 16:47:49.532526 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.532469    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-console-config\") pod \"console-5c9f468564-9bc2g\" (UID: \"83acf15b-a80a-4a21-bcdb-08f6080ccb0f\") " pod="openshift-console/console-5c9f468564-9bc2g"
Jul 07 16:47:49.532526 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.532486    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldktj\" (UniqueName: \"kubernetes.io/projected/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-kube-api-access-ldktj\") pod \"console-5c9f468564-9bc2g\" (UID: \"83acf15b-a80a-4a21-bcdb-08f6080ccb0f\") " pod="openshift-console/console-5c9f468564-9bc2g"
Jul 07 16:47:49.532894 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.532871    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-service-ca\") pod \"console-5c9f468564-9bc2g\" (UID: \"83acf15b-a80a-4a21-bcdb-08f6080ccb0f\") " pod="openshift-console/console-5c9f468564-9bc2g"
Jul 07 16:47:49.533574 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.533548    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-oauth-serving-cert\") pod \"console-5c9f468564-9bc2g\" (UID: \"83acf15b-a80a-4a21-bcdb-08f6080ccb0f\") " pod="openshift-console/console-5c9f468564-9bc2g"
Jul 07 16:47:49.533681 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.533634    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-trusted-ca-bundle\") pod \"console-5c9f468564-9bc2g\" (UID: \"83acf15b-a80a-4a21-bcdb-08f6080ccb0f\") " pod="openshift-console/console-5c9f468564-9bc2g"
Jul 07 16:47:49.533681 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.533639    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-console-config\") pod \"console-5c9f468564-9bc2g\" (UID: \"83acf15b-a80a-4a21-bcdb-08f6080ccb0f\") " pod="openshift-console/console-5c9f468564-9bc2g"
Jul 07 16:47:49.536082 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.536033    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/25fef8f4-e7f3-4274-a226-e07d12bc3efd-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-6zkdb\" (UID: \"25fef8f4-e7f3-4274-a226-e07d12bc3efd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6zkdb"
Jul 07 16:47:49.536185 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.536115    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-console-serving-cert\") pod \"console-5c9f468564-9bc2g\" (UID: \"83acf15b-a80a-4a21-bcdb-08f6080ccb0f\") " pod="openshift-console/console-5c9f468564-9bc2g"
Jul 07 16:47:49.537141 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.537124    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-console-oauth-config\") pod \"console-5c9f468564-9bc2g\" (UID: \"83acf15b-a80a-4a21-bcdb-08f6080ccb0f\") " pod="openshift-console/console-5c9f468564-9bc2g"
Jul 07 16:47:49.550824 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.550801    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldktj\" (UniqueName: \"kubernetes.io/projected/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-kube-api-access-ldktj\") pod \"console-5c9f468564-9bc2g\" (UID: \"83acf15b-a80a-4a21-bcdb-08f6080ccb0f\") " pod="openshift-console/console-5c9f468564-9bc2g"
Jul 07 16:47:49.551239 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.551221    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9n5k\" (UniqueName: \"kubernetes.io/projected/c50b90e0-2467-43e9-ba20-83e720cab84d-kube-api-access-h9n5k\") pod \"downloads-685bfb5f64-xvlzm\" (UID: \"c50b90e0-2467-43e9-ba20-83e720cab84d\") " pod="openshift-console/downloads-685bfb5f64-xvlzm"
Jul 07 16:47:49.580538 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.580455    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:49.632322 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.632289    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6zkdb"
Jul 07 16:47:49.663750 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.663438    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-xvlzm"
Jul 07 16:47:49.670558 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.670533    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c9f468564-9bc2g"
Jul 07 16:47:49.686748 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.686715    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:49.799490 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.799458    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:49.896562 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:49.896527    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:51.823118 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:51.822992    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c9f468564-9bc2g"]
Jul 07 16:47:51.827210 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:47:51.826768    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83acf15b_a80a_4a21_bcdb_08f6080ccb0f.slice/crio-06505609af6cb25c885738bc5040533d7c316f181f3875d08a9cee606d2eedbb WatchSource:0}: Error finding container 06505609af6cb25c885738bc5040533d7c316f181f3875d08a9cee606d2eedbb: Status 404 returned error can't find the container with id 06505609af6cb25c885738bc5040533d7c316f181f3875d08a9cee606d2eedbb
Jul 07 16:47:51.844160 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:51.843973    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-xvlzm"]
Jul 07 16:47:51.846271 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:47:51.846244    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc50b90e0_2467_43e9_ba20_83e720cab84d.slice/crio-f8e5c8e1cea69ec3edb4d4e135981fc04f01e6b67a3c1f591e86b1867ac83ec1 WatchSource:0}: Error finding container f8e5c8e1cea69ec3edb4d4e135981fc04f01e6b67a3c1f591e86b1867ac83ec1: Status 404 returned error can't find the container with id f8e5c8e1cea69ec3edb4d4e135981fc04f01e6b67a3c1f591e86b1867ac83ec1
Jul 07 16:47:51.865132 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:51.865109    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6zkdb"]
Jul 07 16:47:51.866980 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:47:51.866952    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25fef8f4_e7f3_4274_a226_e07d12bc3efd.slice/crio-aad0a5888b0457d60a3d2cb1b4824fd03f5fdc8d273501f2076a24300963d814 WatchSource:0}: Error finding container aad0a5888b0457d60a3d2cb1b4824fd03f5fdc8d273501f2076a24300963d814: Status 404 returned error can't find the container with id aad0a5888b0457d60a3d2cb1b4824fd03f5fdc8d273501f2076a24300963d814
Jul 07 16:47:52.363911 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:52.363868    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6zkdb" event={"ID":"25fef8f4-e7f3-4274-a226-e07d12bc3efd","Type":"ContainerStarted","Data":"aad0a5888b0457d60a3d2cb1b4824fd03f5fdc8d273501f2076a24300963d814"}
Jul 07 16:47:52.365674 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:52.365606    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jk247" event={"ID":"600fd20e-5eba-4886-9bda-68a5c995d9a8","Type":"ContainerStarted","Data":"4f38fa5ad2c85bd7c858d6c6192593b8c2ae855b8cedd6e4f7fc9f4e0f4a07c5"}
Jul 07 16:47:52.367152 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:52.367080    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-xvlzm" event={"ID":"c50b90e0-2467-43e9-ba20-83e720cab84d","Type":"ContainerStarted","Data":"f8e5c8e1cea69ec3edb4d4e135981fc04f01e6b67a3c1f591e86b1867ac83ec1"}
Jul 07 16:47:52.368909 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:52.368882    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c9f468564-9bc2g" event={"ID":"83acf15b-a80a-4a21-bcdb-08f6080ccb0f","Type":"ContainerStarted","Data":"06505609af6cb25c885738bc5040533d7c316f181f3875d08a9cee606d2eedbb"}
Jul 07 16:47:52.504956 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:52.504904    2584 patch_prober.go:28] interesting pod/image-registry-54b88fb4c5-zhjwb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 07 16:47:52.505149 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:52.504971    2584 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb" podUID="2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:47:53.373653 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:53.373612    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jk247" event={"ID":"600fd20e-5eba-4886-9bda-68a5c995d9a8","Type":"ContainerStarted","Data":"dff12004761a203d367c9f87f104a74b1e8fbedf21a629870ee88c0b9136f1b3"}
Jul 07 16:47:53.374946 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:53.374924    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6zkdb" event={"ID":"25fef8f4-e7f3-4274-a226-e07d12bc3efd","Type":"ContainerStarted","Data":"8e6fc0f062375a07c05b6bb4fcd1d4da1fd13aaad99af09ede0227e6de54e4f0"}
Jul 07 16:47:53.375171 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:53.375151    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6zkdb"
Jul 07 16:47:53.380430 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:53.380409    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6zkdb"
Jul 07 16:47:53.394180 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:53.394134    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6zkdb" podStartSLOduration=3.281679272 podStartE2EDuration="4.394117912s" totalImagesPullingTime="1.11243864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:47:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:47:53.393471562 +0000 UTC m=+121.994680268" watchObservedRunningTime="2026-07-07 16:47:53.394117912 +0000 UTC m=+121.995326615"
Jul 07 16:47:53.686444 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:53.686314    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:53.754846 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:53.754806    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:53.828930 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:53.828881    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:54.260295 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:54.260245    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:54.305603 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:54.305258    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:47:54.305603 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:54.305551    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-j7zx8"]
Jul 07 16:47:54.308532 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:54.308511    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j7zx8"
Jul 07 16:47:54.311731 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:54.311704    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 07 16:47:54.312060 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:54.312025    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-x8wsv\""
Jul 07 16:47:54.312262 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:54.312245    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 07 16:47:54.313398 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:54.313376    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 07 16:47:54.313671 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:54.313653    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 07 16:47:54.313671 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:54.313662    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 07 16:47:54.319853 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:54.319202    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-j7zx8"]
Jul 07 16:47:54.356172 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:54.356103    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:54.373635 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:54.373599    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b6b0ea8f-b17a-4ef6-8096-03e87c790d4c-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-j7zx8\" (UID: \"b6b0ea8f-b17a-4ef6-8096-03e87c790d4c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j7zx8"
Jul 07 16:47:54.373835 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:54.373654    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6b0ea8f-b17a-4ef6-8096-03e87c790d4c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-j7zx8\" (UID: \"b6b0ea8f-b17a-4ef6-8096-03e87c790d4c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j7zx8"
Jul 07 16:47:54.373835 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:54.373684    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6b0ea8f-b17a-4ef6-8096-03e87c790d4c-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-j7zx8\" (UID: \"b6b0ea8f-b17a-4ef6-8096-03e87c790d4c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j7zx8"
Jul 07 16:47:54.374341 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:54.374124    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npvkm\" (UniqueName: \"kubernetes.io/projected/b6b0ea8f-b17a-4ef6-8096-03e87c790d4c-kube-api-access-npvkm\") pod \"prometheus-operator-567f7b8b7c-j7zx8\" (UID: \"b6b0ea8f-b17a-4ef6-8096-03e87c790d4c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j7zx8"
Jul 07 16:47:54.427742 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:54.427705    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:54.476597 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:54.475509    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6b0ea8f-b17a-4ef6-8096-03e87c790d4c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-j7zx8\" (UID: \"b6b0ea8f-b17a-4ef6-8096-03e87c790d4c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j7zx8"
Jul 07 16:47:54.476597 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:54.475556    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6b0ea8f-b17a-4ef6-8096-03e87c790d4c-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-j7zx8\" (UID: \"b6b0ea8f-b17a-4ef6-8096-03e87c790d4c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j7zx8"
Jul 07 16:47:54.476597 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:54.475652    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npvkm\" (UniqueName: \"kubernetes.io/projected/b6b0ea8f-b17a-4ef6-8096-03e87c790d4c-kube-api-access-npvkm\") pod \"prometheus-operator-567f7b8b7c-j7zx8\" (UID: \"b6b0ea8f-b17a-4ef6-8096-03e87c790d4c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j7zx8"
Jul 07 16:47:54.476597 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:54.475701    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b6b0ea8f-b17a-4ef6-8096-03e87c790d4c-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-j7zx8\" (UID: \"b6b0ea8f-b17a-4ef6-8096-03e87c790d4c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j7zx8"
Jul 07 16:47:54.476946 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:54.476669    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6b0ea8f-b17a-4ef6-8096-03e87c790d4c-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-j7zx8\" (UID: \"b6b0ea8f-b17a-4ef6-8096-03e87c790d4c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j7zx8"
Jul 07 16:47:54.479352 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:54.479321    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6b0ea8f-b17a-4ef6-8096-03e87c790d4c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-j7zx8\" (UID: \"b6b0ea8f-b17a-4ef6-8096-03e87c790d4c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j7zx8"
Jul 07 16:47:54.480523 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:54.480499    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b6b0ea8f-b17a-4ef6-8096-03e87c790d4c-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-j7zx8\" (UID: \"b6b0ea8f-b17a-4ef6-8096-03e87c790d4c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j7zx8"
Jul 07 16:47:54.487544 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:54.487514    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npvkm\" (UniqueName: \"kubernetes.io/projected/b6b0ea8f-b17a-4ef6-8096-03e87c790d4c-kube-api-access-npvkm\") pod \"prometheus-operator-567f7b8b7c-j7zx8\" (UID: \"b6b0ea8f-b17a-4ef6-8096-03e87c790d4c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j7zx8"
Jul 07 16:47:54.623277 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:54.623239    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j7zx8"
Jul 07 16:47:54.840519 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:54.840457    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:54.941367 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:54.941275    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:55.295296 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:55.295186    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:55.372929 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:55.372849    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:57.032624 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:57.032593    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-j7zx8"]
Jul 07 16:47:57.036547 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:47:57.036493    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b0ea8f_b17a_4ef6_8096_03e87c790d4c.slice/crio-8a652de329bc43c3e278a1febea447564c3b8bef57a26652da285c015c150900 WatchSource:0}: Error finding container 8a652de329bc43c3e278a1febea447564c3b8bef57a26652da285c015c150900: Status 404 returned error can't find the container with id 8a652de329bc43c3e278a1febea447564c3b8bef57a26652da285c015c150900
Jul 07 16:47:57.393452 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:57.393355    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jk247" event={"ID":"600fd20e-5eba-4886-9bda-68a5c995d9a8","Type":"ContainerStarted","Data":"3ae108bfd3ce4017d76b272a34238f7d7d8683722b4a33433c970730d4c116ba"}
Jul 07 16:47:57.394870 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:57.394844    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c9f468564-9bc2g" event={"ID":"83acf15b-a80a-4a21-bcdb-08f6080ccb0f","Type":"ContainerStarted","Data":"dcaff062ac364fd1c25ce60de61e3e0f3a96419af3508ffa2949f60d66bbd3be"}
Jul 07 16:47:57.396509 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:57.396485    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j7zx8" event={"ID":"b6b0ea8f-b17a-4ef6-8096-03e87c790d4c","Type":"ContainerStarted","Data":"8a652de329bc43c3e278a1febea447564c3b8bef57a26652da285c015c150900"}
Jul 07 16:47:57.428066 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:57.427993    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jk247" podStartSLOduration=33.746865514 podStartE2EDuration="42.427978187s" totalImagesPullingTime="8.681112673s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:47:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:47:57.427526898 +0000 UTC m=+126.028735603" watchObservedRunningTime="2026-07-07 16:47:57.427978187 +0000 UTC m=+126.029186889"
Jul 07 16:47:57.465864 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:57.465811    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c9f468564-9bc2g" podStartSLOduration=3.418471357 podStartE2EDuration="8.46578968s" totalImagesPullingTime="5.047318323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:47:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:47:57.465195301 +0000 UTC m=+126.066404006" watchObservedRunningTime="2026-07-07 16:47:57.46578968 +0000 UTC m=+126.066998387"
Jul 07 16:47:57.903482 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:57.903439    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:57.996722 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:57.996683    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:58.369663 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:58.369632    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:58.402265 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:58.402240    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j7zx8" event={"ID":"b6b0ea8f-b17a-4ef6-8096-03e87c790d4c","Type":"ContainerStarted","Data":"2a272f6211492d6da06c902e307e93e243c363ce47abcfa0a166a0824e624af9"}
Jul 07 16:47:58.475242 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:58.475209    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:58.657553 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:58.657517    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:47:59.408357 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:59.408317    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j7zx8" event={"ID":"b6b0ea8f-b17a-4ef6-8096-03e87c790d4c","Type":"ContainerStarted","Data":"ba3245f2abf665b5e6d197d5d0bdebdca02db13c69f67808b40d64e1e8bd000a"}
Jul 07 16:47:59.435894 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:59.435834    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-j7zx8" podStartSLOduration=4.161606547 podStartE2EDuration="5.435812894s" totalImagesPullingTime="1.274206347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:47:59.435488383 +0000 UTC m=+128.036697086" watchObservedRunningTime="2026-07-07 16:47:59.435812894 +0000 UTC m=+128.037021601"
Jul 07 16:47:59.671452 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:59.671364    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c9f468564-9bc2g"
Jul 07 16:47:59.671452 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:59.671434    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c9f468564-9bc2g"
Jul 07 16:47:59.677397 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:47:59.677348    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c9f468564-9bc2g"
Jul 07 16:48:00.415867 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:00.415836    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c9f468564-9bc2g"
Jul 07 16:48:01.453779 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:01.453658    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:01.640170 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:01.640063    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f7e543c-8cdd-4f28-adb5-570b6004c796-metrics-certs\") pod \"network-metrics-daemon-9kj8c\" (UID: \"7f7e543c-8cdd-4f28-adb5-570b6004c796\") " pod="openshift-multus/network-metrics-daemon-9kj8c"
Jul 07 16:48:01.643068 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:01.643012    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f7e543c-8cdd-4f28-adb5-570b6004c796-metrics-certs\") pod \"network-metrics-daemon-9kj8c\" (UID: \"7f7e543c-8cdd-4f28-adb5-570b6004c796\") " pod="openshift-multus/network-metrics-daemon-9kj8c"
Jul 07 16:48:01.759692 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:01.759588    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:01.827822 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:01.827788    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-rw2p9"]
Jul 07 16:48:01.846030 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:01.845996    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rw2p9"
Jul 07 16:48:01.848364 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:01.848198    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-rw2p9"]
Jul 07 16:48:01.852081 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:01.852039    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 07 16:48:01.852249 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:01.852062    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 07 16:48:01.852591 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:01.852311    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-m5d7p\""
Jul 07 16:48:01.905005 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:01.904898    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:01.909007 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:01.908967    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-q6m8v"]
Jul 07 16:48:01.913121 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:01.912764    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q6m8v"
Jul 07 16:48:01.913747 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:01.913510    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rsbn5\""
Jul 07 16:48:01.916943 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:01.916922    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 07 16:48:01.917351 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:01.917171    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 07 16:48:01.917432 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:01.917407    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fzkwc\""
Jul 07 16:48:01.917606 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:01.917553    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 07 16:48:01.917873 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:01.917856    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9kj8c"
Jul 07 16:48:01.946739 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:01.946128    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69zln\" (UniqueName: \"kubernetes.io/projected/2449686a-76f3-46db-806b-953bc02530f3-kube-api-access-69zln\") pod \"openshift-state-metrics-54cf96dfcf-rw2p9\" (UID: \"2449686a-76f3-46db-806b-953bc02530f3\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rw2p9"
Jul 07 16:48:01.946739 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:01.946186    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee4b1222-3d0f-485e-a082-745d5a8f1cb2-sys\") pod \"node-exporter-q6m8v\" (UID: \"ee4b1222-3d0f-485e-a082-745d5a8f1cb2\") " pod="openshift-monitoring/node-exporter-q6m8v"
Jul 07 16:48:01.946739 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:01.946224    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ee4b1222-3d0f-485e-a082-745d5a8f1cb2-node-exporter-wtmp\") pod \"node-exporter-q6m8v\" (UID: \"ee4b1222-3d0f-485e-a082-745d5a8f1cb2\") " pod="openshift-monitoring/node-exporter-q6m8v"
Jul 07 16:48:01.946739 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:01.946254    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ee4b1222-3d0f-485e-a082-745d5a8f1cb2-node-exporter-tls\") pod \"node-exporter-q6m8v\" (UID: \"ee4b1222-3d0f-485e-a082-745d5a8f1cb2\") " pod="openshift-monitoring/node-exporter-q6m8v"
Jul 07 16:48:01.946739 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:01.946288    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2449686a-76f3-46db-806b-953bc02530f3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-rw2p9\" (UID: \"2449686a-76f3-46db-806b-953bc02530f3\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rw2p9"
Jul 07 16:48:01.946739 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:01.946314    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ee4b1222-3d0f-485e-a082-745d5a8f1cb2-root\") pod \"node-exporter-q6m8v\" (UID: \"ee4b1222-3d0f-485e-a082-745d5a8f1cb2\") " pod="openshift-monitoring/node-exporter-q6m8v"
Jul 07 16:48:01.946739 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:01.946344    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2449686a-76f3-46db-806b-953bc02530f3-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-rw2p9\" (UID: \"2449686a-76f3-46db-806b-953bc02530f3\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rw2p9"
Jul 07 16:48:01.946739 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:01.946390    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ee4b1222-3d0f-485e-a082-745d5a8f1cb2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q6m8v\" (UID: \"ee4b1222-3d0f-485e-a082-745d5a8f1cb2\") " pod="openshift-monitoring/node-exporter-q6m8v"
Jul 07 16:48:01.946739 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:01.946436    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee4b1222-3d0f-485e-a082-745d5a8f1cb2-metrics-client-ca\") pod \"node-exporter-q6m8v\" (UID: \"ee4b1222-3d0f-485e-a082-745d5a8f1cb2\") " pod="openshift-monitoring/node-exporter-q6m8v"
Jul 07 16:48:01.946739 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:01.946459    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ee4b1222-3d0f-485e-a082-745d5a8f1cb2-node-exporter-accelerators-collector-config\") pod \"node-exporter-q6m8v\" (UID: \"ee4b1222-3d0f-485e-a082-745d5a8f1cb2\") " pod="openshift-monitoring/node-exporter-q6m8v"
Jul 07 16:48:01.946739 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:01.946488    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2449686a-76f3-46db-806b-953bc02530f3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-rw2p9\" (UID: \"2449686a-76f3-46db-806b-953bc02530f3\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rw2p9"
Jul 07 16:48:01.946739 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:01.946528    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vqtt\" (UniqueName: \"kubernetes.io/projected/ee4b1222-3d0f-485e-a082-745d5a8f1cb2-kube-api-access-9vqtt\") pod \"node-exporter-q6m8v\" (UID: \"ee4b1222-3d0f-485e-a082-745d5a8f1cb2\") " pod="openshift-monitoring/node-exporter-q6m8v"
Jul 07 16:48:01.946739 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:01.946554    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ee4b1222-3d0f-485e-a082-745d5a8f1cb2-node-exporter-textfile\") pod \"node-exporter-q6m8v\" (UID: \"ee4b1222-3d0f-485e-a082-745d5a8f1cb2\") " pod="openshift-monitoring/node-exporter-q6m8v"
Jul 07 16:48:02.050826 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.047996    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69zln\" (UniqueName: \"kubernetes.io/projected/2449686a-76f3-46db-806b-953bc02530f3-kube-api-access-69zln\") pod \"openshift-state-metrics-54cf96dfcf-rw2p9\" (UID: \"2449686a-76f3-46db-806b-953bc02530f3\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rw2p9"
Jul 07 16:48:02.050826 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.048038    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee4b1222-3d0f-485e-a082-745d5a8f1cb2-sys\") pod \"node-exporter-q6m8v\" (UID: \"ee4b1222-3d0f-485e-a082-745d5a8f1cb2\") " pod="openshift-monitoring/node-exporter-q6m8v"
Jul 07 16:48:02.050826 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.048095    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ee4b1222-3d0f-485e-a082-745d5a8f1cb2-node-exporter-wtmp\") pod \"node-exporter-q6m8v\" (UID: \"ee4b1222-3d0f-485e-a082-745d5a8f1cb2\") " pod="openshift-monitoring/node-exporter-q6m8v"
Jul 07 16:48:02.050826 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.048126    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ee4b1222-3d0f-485e-a082-745d5a8f1cb2-node-exporter-tls\") pod \"node-exporter-q6m8v\" (UID: \"ee4b1222-3d0f-485e-a082-745d5a8f1cb2\") " pod="openshift-monitoring/node-exporter-q6m8v"
Jul 07 16:48:02.050826 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.048159    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2449686a-76f3-46db-806b-953bc02530f3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-rw2p9\" (UID: \"2449686a-76f3-46db-806b-953bc02530f3\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rw2p9"
Jul 07 16:48:02.050826 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.048187    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ee4b1222-3d0f-485e-a082-745d5a8f1cb2-root\") pod \"node-exporter-q6m8v\" (UID: \"ee4b1222-3d0f-485e-a082-745d5a8f1cb2\") " pod="openshift-monitoring/node-exporter-q6m8v"
Jul 07 16:48:02.050826 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.048215    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2449686a-76f3-46db-806b-953bc02530f3-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-rw2p9\" (UID: \"2449686a-76f3-46db-806b-953bc02530f3\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rw2p9"
Jul 07 16:48:02.050826 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.048257    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ee4b1222-3d0f-485e-a082-745d5a8f1cb2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q6m8v\" (UID: \"ee4b1222-3d0f-485e-a082-745d5a8f1cb2\") " pod="openshift-monitoring/node-exporter-q6m8v"
Jul 07 16:48:02.050826 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.048297    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee4b1222-3d0f-485e-a082-745d5a8f1cb2-metrics-client-ca\") pod \"node-exporter-q6m8v\" (UID: \"ee4b1222-3d0f-485e-a082-745d5a8f1cb2\") " pod="openshift-monitoring/node-exporter-q6m8v"
Jul 07 16:48:02.050826 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.048319    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ee4b1222-3d0f-485e-a082-745d5a8f1cb2-node-exporter-accelerators-collector-config\") pod \"node-exporter-q6m8v\" (UID: \"ee4b1222-3d0f-485e-a082-745d5a8f1cb2\") " pod="openshift-monitoring/node-exporter-q6m8v"
Jul 07 16:48:02.050826 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.048348    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2449686a-76f3-46db-806b-953bc02530f3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-rw2p9\" (UID: \"2449686a-76f3-46db-806b-953bc02530f3\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rw2p9"
Jul 07 16:48:02.050826 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.048383    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vqtt\" (UniqueName: \"kubernetes.io/projected/ee4b1222-3d0f-485e-a082-745d5a8f1cb2-kube-api-access-9vqtt\") pod \"node-exporter-q6m8v\" (UID: \"ee4b1222-3d0f-485e-a082-745d5a8f1cb2\") " pod="openshift-monitoring/node-exporter-q6m8v"
Jul 07 16:48:02.050826 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.048409    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ee4b1222-3d0f-485e-a082-745d5a8f1cb2-node-exporter-textfile\") pod \"node-exporter-q6m8v\" (UID: \"ee4b1222-3d0f-485e-a082-745d5a8f1cb2\") " pod="openshift-monitoring/node-exporter-q6m8v"
Jul 07 16:48:02.050826 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:48:02.048442    2584 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 07 16:48:02.050826 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:48:02.048542    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2449686a-76f3-46db-806b-953bc02530f3-openshift-state-metrics-tls podName:2449686a-76f3-46db-806b-953bc02530f3 nodeName:}" failed. No retries permitted until 2026-07-07 16:48:02.548517171 +0000 UTC m=+131.149725871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/2449686a-76f3-46db-806b-953bc02530f3-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-rw2p9" (UID: "2449686a-76f3-46db-806b-953bc02530f3") : secret "openshift-state-metrics-tls" not found
Jul 07 16:48:02.050826 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.048784    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ee4b1222-3d0f-485e-a082-745d5a8f1cb2-node-exporter-textfile\") pod \"node-exporter-q6m8v\" (UID: \"ee4b1222-3d0f-485e-a082-745d5a8f1cb2\") " pod="openshift-monitoring/node-exporter-q6m8v"
Jul 07 16:48:02.051787 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.048856    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ee4b1222-3d0f-485e-a082-745d5a8f1cb2-root\") pod \"node-exporter-q6m8v\" (UID: \"ee4b1222-3d0f-485e-a082-745d5a8f1cb2\") " pod="openshift-monitoring/node-exporter-q6m8v"
Jul 07 16:48:02.051787 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.049071    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee4b1222-3d0f-485e-a082-745d5a8f1cb2-sys\") pod \"node-exporter-q6m8v\" (UID: \"ee4b1222-3d0f-485e-a082-745d5a8f1cb2\") " pod="openshift-monitoring/node-exporter-q6m8v"
Jul 07 16:48:02.051787 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.049213    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ee4b1222-3d0f-485e-a082-745d5a8f1cb2-node-exporter-wtmp\") pod \"node-exporter-q6m8v\" (UID: \"ee4b1222-3d0f-485e-a082-745d5a8f1cb2\") " pod="openshift-monitoring/node-exporter-q6m8v"
Jul 07 16:48:02.051787 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:48:02.049280    2584 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 07 16:48:02.051787 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:48:02.049334    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee4b1222-3d0f-485e-a082-745d5a8f1cb2-node-exporter-tls podName:ee4b1222-3d0f-485e-a082-745d5a8f1cb2 nodeName:}" failed. No retries permitted until 2026-07-07 16:48:02.549316455 +0000 UTC m=+131.150525143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/ee4b1222-3d0f-485e-a082-745d5a8f1cb2-node-exporter-tls") pod "node-exporter-q6m8v" (UID: "ee4b1222-3d0f-485e-a082-745d5a8f1cb2") : secret "node-exporter-tls" not found
Jul 07 16:48:02.051787 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.049620    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2449686a-76f3-46db-806b-953bc02530f3-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-rw2p9\" (UID: \"2449686a-76f3-46db-806b-953bc02530f3\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rw2p9"
Jul 07 16:48:02.051787 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.049687    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ee4b1222-3d0f-485e-a082-745d5a8f1cb2-node-exporter-accelerators-collector-config\") pod \"node-exporter-q6m8v\" (UID: \"ee4b1222-3d0f-485e-a082-745d5a8f1cb2\") " pod="openshift-monitoring/node-exporter-q6m8v"
Jul 07 16:48:02.051787 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.050319    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee4b1222-3d0f-485e-a082-745d5a8f1cb2-metrics-client-ca\") pod \"node-exporter-q6m8v\" (UID: \"ee4b1222-3d0f-485e-a082-745d5a8f1cb2\") " pod="openshift-monitoring/node-exporter-q6m8v"
Jul 07 16:48:02.051787 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.050765    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:02.062873 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.061311    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ee4b1222-3d0f-485e-a082-745d5a8f1cb2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q6m8v\" (UID: \"ee4b1222-3d0f-485e-a082-745d5a8f1cb2\") " pod="openshift-monitoring/node-exporter-q6m8v"
Jul 07 16:48:02.064441 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.064407    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2449686a-76f3-46db-806b-953bc02530f3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-rw2p9\" (UID: \"2449686a-76f3-46db-806b-953bc02530f3\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rw2p9"
Jul 07 16:48:02.069317 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.069261    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vqtt\" (UniqueName: \"kubernetes.io/projected/ee4b1222-3d0f-485e-a082-745d5a8f1cb2-kube-api-access-9vqtt\") pod \"node-exporter-q6m8v\" (UID: \"ee4b1222-3d0f-485e-a082-745d5a8f1cb2\") " pod="openshift-monitoring/node-exporter-q6m8v"
Jul 07 16:48:02.094597 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.094562    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69zln\" (UniqueName: \"kubernetes.io/projected/2449686a-76f3-46db-806b-953bc02530f3-kube-api-access-69zln\") pod \"openshift-state-metrics-54cf96dfcf-rw2p9\" (UID: \"2449686a-76f3-46db-806b-953bc02530f3\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rw2p9"
Jul 07 16:48:02.117278 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.117246    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9kj8c"]
Jul 07 16:48:02.123874 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:48:02.123843    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f7e543c_8cdd_4f28_adb5_570b6004c796.slice/crio-999875f6f26622eee80158e8fd00d60a788e3022d6f04ac425bb129263c53ef4 WatchSource:0}: Error finding container 999875f6f26622eee80158e8fd00d60a788e3022d6f04ac425bb129263c53ef4: Status 404 returned error can't find the container with id 999875f6f26622eee80158e8fd00d60a788e3022d6f04ac425bb129263c53ef4
Jul 07 16:48:02.313687 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.313587    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:02.421196 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.421156    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9kj8c" event={"ID":"7f7e543c-8cdd-4f28-adb5-570b6004c796","Type":"ContainerStarted","Data":"999875f6f26622eee80158e8fd00d60a788e3022d6f04ac425bb129263c53ef4"}
Jul 07 16:48:02.459938 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.459889    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:02.553481 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.553439    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ee4b1222-3d0f-485e-a082-745d5a8f1cb2-node-exporter-tls\") pod \"node-exporter-q6m8v\" (UID: \"ee4b1222-3d0f-485e-a082-745d5a8f1cb2\") " pod="openshift-monitoring/node-exporter-q6m8v"
Jul 07 16:48:02.553669 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.553492    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2449686a-76f3-46db-806b-953bc02530f3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-rw2p9\" (UID: \"2449686a-76f3-46db-806b-953bc02530f3\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rw2p9"
Jul 07 16:48:02.553669 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:48:02.553595    2584 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 07 16:48:02.553669 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:48:02.553656    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee4b1222-3d0f-485e-a082-745d5a8f1cb2-node-exporter-tls podName:ee4b1222-3d0f-485e-a082-745d5a8f1cb2 nodeName:}" failed. No retries permitted until 2026-07-07 16:48:03.553641411 +0000 UTC m=+132.154850093 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/ee4b1222-3d0f-485e-a082-745d5a8f1cb2-node-exporter-tls") pod "node-exporter-q6m8v" (UID: "ee4b1222-3d0f-485e-a082-745d5a8f1cb2") : secret "node-exporter-tls" not found
Jul 07 16:48:02.556888 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.556828    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2449686a-76f3-46db-806b-953bc02530f3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-rw2p9\" (UID: \"2449686a-76f3-46db-806b-953bc02530f3\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rw2p9"
Jul 07 16:48:02.586996 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.586867    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:02.686534 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.686491    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:02.762362 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.762320    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rw2p9"
Jul 07 16:48:02.878405 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.878367    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:02.952169 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.952135    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-rw2p9"]
Jul 07 16:48:02.977174 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:02.977095    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:03.013010 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.012975    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 16:48:03.018409 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.018382    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.023160 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.022773    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 07 16:48:03.023498 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.023479    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 07 16:48:03.024619 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.024600    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 07 16:48:03.024846 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.024789    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 07 16:48:03.026008 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.025651    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 07 16:48:03.026008 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.025822    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9ndpp\""
Jul 07 16:48:03.026008 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.025888    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 07 16:48:03.026233 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.026078    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 07 16:48:03.026233 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.026172    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 07 16:48:03.026327 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.026235    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 07 16:48:03.045672 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.045581    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 16:48:03.059358 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.059316    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/58834c6a-cd00-428f-8a2b-37b529c9aadf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.059550 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.059364    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/58834c6a-cd00-428f-8a2b-37b529c9aadf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.059550 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.059449    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/58834c6a-cd00-428f-8a2b-37b529c9aadf-config-out\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.059550 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.059480    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59cdh\" (UniqueName: \"kubernetes.io/projected/58834c6a-cd00-428f-8a2b-37b529c9aadf-kube-api-access-59cdh\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.059722 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.059553    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/58834c6a-cd00-428f-8a2b-37b529c9aadf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.059722 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.059581    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/58834c6a-cd00-428f-8a2b-37b529c9aadf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.059722 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.059683    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/58834c6a-cd00-428f-8a2b-37b529c9aadf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.059870 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.059722    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/58834c6a-cd00-428f-8a2b-37b529c9aadf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.059870 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.059759    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58834c6a-cd00-428f-8a2b-37b529c9aadf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.059870 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.059793    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/58834c6a-cd00-428f-8a2b-37b529c9aadf-config-volume\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.059870 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.059815    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/58834c6a-cd00-428f-8a2b-37b529c9aadf-web-config\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.060093 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.059899    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58834c6a-cd00-428f-8a2b-37b529c9aadf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.060093 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.059945    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/58834c6a-cd00-428f-8a2b-37b529c9aadf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.082630 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.082593    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:03.089686 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:48:03.089649    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2449686a_76f3_46db_806b_953bc02530f3.slice/crio-0276c350dcadaf0431a4f60ee6e9e3092bb9ee06ad6d9900f743b252b50dd00a WatchSource:0}: Error finding container 0276c350dcadaf0431a4f60ee6e9e3092bb9ee06ad6d9900f743b252b50dd00a: Status 404 returned error can't find the container with id 0276c350dcadaf0431a4f60ee6e9e3092bb9ee06ad6d9900f743b252b50dd00a
Jul 07 16:48:03.161281 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.161259    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/58834c6a-cd00-428f-8a2b-37b529c9aadf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.161455 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.161295    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/58834c6a-cd00-428f-8a2b-37b529c9aadf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.161455 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.161318    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58834c6a-cd00-428f-8a2b-37b529c9aadf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.161455 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.161348    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/58834c6a-cd00-428f-8a2b-37b529c9aadf-config-volume\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.161455 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.161371    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/58834c6a-cd00-428f-8a2b-37b529c9aadf-web-config\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.161661 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:48:03.161473    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/58834c6a-cd00-428f-8a2b-37b529c9aadf-alertmanager-trusted-ca-bundle podName:58834c6a-cd00-428f-8a2b-37b529c9aadf nodeName:}" failed. No retries permitted until 2026-07-07 16:48:03.661449898 +0000 UTC m=+132.262658581 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/58834c6a-cd00-428f-8a2b-37b529c9aadf-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "58834c6a-cd00-428f-8a2b-37b529c9aadf") : configmap references non-existent config key: ca-bundle.crt
Jul 07 16:48:03.161661 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.161511    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58834c6a-cd00-428f-8a2b-37b529c9aadf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.161661 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.161551    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/58834c6a-cd00-428f-8a2b-37b529c9aadf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.161661 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.161588    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/58834c6a-cd00-428f-8a2b-37b529c9aadf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.161661 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.161615    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/58834c6a-cd00-428f-8a2b-37b529c9aadf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.162446 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.162407    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/58834c6a-cd00-428f-8a2b-37b529c9aadf-config-out\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.163376 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.162612    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58834c6a-cd00-428f-8a2b-37b529c9aadf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.163376 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.161649    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/58834c6a-cd00-428f-8a2b-37b529c9aadf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.163376 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.163270    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59cdh\" (UniqueName: \"kubernetes.io/projected/58834c6a-cd00-428f-8a2b-37b529c9aadf-kube-api-access-59cdh\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.163612 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.163406    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/58834c6a-cd00-428f-8a2b-37b529c9aadf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.163681 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.163615    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/58834c6a-cd00-428f-8a2b-37b529c9aadf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.170423 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.168080    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/58834c6a-cd00-428f-8a2b-37b529c9aadf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.170423 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.169484    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/58834c6a-cd00-428f-8a2b-37b529c9aadf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.172404 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.172337    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/58834c6a-cd00-428f-8a2b-37b529c9aadf-config-volume\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.172784 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.172761    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/58834c6a-cd00-428f-8a2b-37b529c9aadf-config-out\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.172921 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.172863    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/58834c6a-cd00-428f-8a2b-37b529c9aadf-web-config\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.174733 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.173067    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/58834c6a-cd00-428f-8a2b-37b529c9aadf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.174733 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.174517    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/58834c6a-cd00-428f-8a2b-37b529c9aadf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.175359 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.175133    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/58834c6a-cd00-428f-8a2b-37b529c9aadf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.175359 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.175135    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/58834c6a-cd00-428f-8a2b-37b529c9aadf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.179980 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.179682    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59cdh\" (UniqueName: \"kubernetes.io/projected/58834c6a-cd00-428f-8a2b-37b529c9aadf-kube-api-access-59cdh\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.185536 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.185505    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:03.427580 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.427332    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rw2p9" event={"ID":"2449686a-76f3-46db-806b-953bc02530f3","Type":"ContainerStarted","Data":"aace65aa324c17a280188c3fe8226281b75c7c4732a718f331d660e76bbd4921"}
Jul 07 16:48:03.427580 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.427444    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rw2p9" event={"ID":"2449686a-76f3-46db-806b-953bc02530f3","Type":"ContainerStarted","Data":"66168eb1b6e0fcd34e7caddabc1703a9fa12deb2a0b512e48c00f16a5a26e401"}
Jul 07 16:48:03.427580 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.427477    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rw2p9" event={"ID":"2449686a-76f3-46db-806b-953bc02530f3","Type":"ContainerStarted","Data":"0276c350dcadaf0431a4f60ee6e9e3092bb9ee06ad6d9900f743b252b50dd00a"}
Jul 07 16:48:03.429516 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.429412    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9kj8c" event={"ID":"7f7e543c-8cdd-4f28-adb5-570b6004c796","Type":"ContainerStarted","Data":"1b75987eb9b324f328ee8cb75cde1df14a955a1c1b01741e2fff895f91d58e3a"}
Jul 07 16:48:03.429516 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.429466    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9kj8c" event={"ID":"7f7e543c-8cdd-4f28-adb5-570b6004c796","Type":"ContainerStarted","Data":"ec8271548fa8940dba4a951b9e1b6d7e8654ffe12f8d3324b5514210f9db02fe"}
Jul 07 16:48:03.454025 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.453967    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9kj8c" podStartSLOduration=129.442484432 podStartE2EDuration="2m10.453948604s" totalImagesPullingTime="1.011464172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:45:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:03.452691469 +0000 UTC m=+132.053900173" watchObservedRunningTime="2026-07-07 16:48:03.453948604 +0000 UTC m=+132.055157307"
Jul 07 16:48:03.568332 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.568294    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ee4b1222-3d0f-485e-a082-745d5a8f1cb2-node-exporter-tls\") pod \"node-exporter-q6m8v\" (UID: \"ee4b1222-3d0f-485e-a082-745d5a8f1cb2\") " pod="openshift-monitoring/node-exporter-q6m8v"
Jul 07 16:48:03.571220 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.571190    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ee4b1222-3d0f-485e-a082-745d5a8f1cb2-node-exporter-tls\") pod \"node-exporter-q6m8v\" (UID: \"ee4b1222-3d0f-485e-a082-745d5a8f1cb2\") " pod="openshift-monitoring/node-exporter-q6m8v"
Jul 07 16:48:03.593959 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.593923    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:03.669410 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.669378    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58834c6a-cd00-428f-8a2b-37b529c9aadf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.670357 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.670323    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58834c6a-cd00-428f-8a2b-37b529c9aadf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"58834c6a-cd00-428f-8a2b-37b529c9aadf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:03.710434 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.710349    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:03.737417 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.737383    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q6m8v"
Jul 07 16:48:03.933348 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:03.933307    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 07 16:48:04.874002 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:04.873969    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:04.921282 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:04.921239    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6"]
Jul 07 16:48:04.925408 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:04.925385    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6"
Jul 07 16:48:04.934826 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:04.934730    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 07 16:48:04.934894 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:04.934822    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 07 16:48:04.934958 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:04.934939    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 07 16:48:04.935027 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:04.934976    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 07 16:48:04.935027 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:04.934949    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-4klrz\""
Jul 07 16:48:04.935196 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:04.935034    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5ed37i0r7h0q4\""
Jul 07 16:48:04.935196 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:04.935037    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 07 16:48:04.943206 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:04.943180    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6"]
Jul 07 16:48:04.980478 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:04.980440    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8f33dbb1-01ec-4e73-9cae-9a20e299c0b5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55f89bcf5d-dp6c6\" (UID: \"8f33dbb1-01ec-4e73-9cae-9a20e299c0b5\") " pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6"
Jul 07 16:48:04.980669 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:04.980503    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f33dbb1-01ec-4e73-9cae-9a20e299c0b5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55f89bcf5d-dp6c6\" (UID: \"8f33dbb1-01ec-4e73-9cae-9a20e299c0b5\") " pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6"
Jul 07 16:48:04.980669 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:04.980554    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8f33dbb1-01ec-4e73-9cae-9a20e299c0b5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55f89bcf5d-dp6c6\" (UID: \"8f33dbb1-01ec-4e73-9cae-9a20e299c0b5\") " pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6"
Jul 07 16:48:04.980669 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:04.980579    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f33dbb1-01ec-4e73-9cae-9a20e299c0b5-metrics-client-ca\") pod \"thanos-querier-55f89bcf5d-dp6c6\" (UID: \"8f33dbb1-01ec-4e73-9cae-9a20e299c0b5\") " pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6"
Jul 07 16:48:04.980669 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:04.980647    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8f33dbb1-01ec-4e73-9cae-9a20e299c0b5-secret-thanos-querier-tls\") pod \"thanos-querier-55f89bcf5d-dp6c6\" (UID: \"8f33dbb1-01ec-4e73-9cae-9a20e299c0b5\") " pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6"
Jul 07 16:48:04.980873 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:04.980691    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75pl2\" (UniqueName: \"kubernetes.io/projected/8f33dbb1-01ec-4e73-9cae-9a20e299c0b5-kube-api-access-75pl2\") pod \"thanos-querier-55f89bcf5d-dp6c6\" (UID: \"8f33dbb1-01ec-4e73-9cae-9a20e299c0b5\") " pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6"
Jul 07 16:48:04.980873 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:04.980729    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8f33dbb1-01ec-4e73-9cae-9a20e299c0b5-secret-grpc-tls\") pod \"thanos-querier-55f89bcf5d-dp6c6\" (UID: \"8f33dbb1-01ec-4e73-9cae-9a20e299c0b5\") " pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6"
Jul 07 16:48:04.980873 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:04.980768    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f33dbb1-01ec-4e73-9cae-9a20e299c0b5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55f89bcf5d-dp6c6\" (UID: \"8f33dbb1-01ec-4e73-9cae-9a20e299c0b5\") " pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6"
Jul 07 16:48:05.004542 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:05.004493    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:05.082162 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:05.082123    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f33dbb1-01ec-4e73-9cae-9a20e299c0b5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55f89bcf5d-dp6c6\" (UID: \"8f33dbb1-01ec-4e73-9cae-9a20e299c0b5\") " pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6"
Jul 07 16:48:05.082363 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:05.082195    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8f33dbb1-01ec-4e73-9cae-9a20e299c0b5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55f89bcf5d-dp6c6\" (UID: \"8f33dbb1-01ec-4e73-9cae-9a20e299c0b5\") " pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6"
Jul 07 16:48:05.082363 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:05.082223    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f33dbb1-01ec-4e73-9cae-9a20e299c0b5-metrics-client-ca\") pod \"thanos-querier-55f89bcf5d-dp6c6\" (UID: \"8f33dbb1-01ec-4e73-9cae-9a20e299c0b5\") " pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6"
Jul 07 16:48:05.082363 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:05.082330    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8f33dbb1-01ec-4e73-9cae-9a20e299c0b5-secret-thanos-querier-tls\") pod \"thanos-querier-55f89bcf5d-dp6c6\" (UID: \"8f33dbb1-01ec-4e73-9cae-9a20e299c0b5\") " pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6"
Jul 07 16:48:05.082363 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:05.082361    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75pl2\" (UniqueName: \"kubernetes.io/projected/8f33dbb1-01ec-4e73-9cae-9a20e299c0b5-kube-api-access-75pl2\") pod \"thanos-querier-55f89bcf5d-dp6c6\" (UID: \"8f33dbb1-01ec-4e73-9cae-9a20e299c0b5\") " pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6"
Jul 07 16:48:05.082567 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:05.082383    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8f33dbb1-01ec-4e73-9cae-9a20e299c0b5-secret-grpc-tls\") pod \"thanos-querier-55f89bcf5d-dp6c6\" (UID: \"8f33dbb1-01ec-4e73-9cae-9a20e299c0b5\") " pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6"
Jul 07 16:48:05.082567 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:05.082422    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f33dbb1-01ec-4e73-9cae-9a20e299c0b5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55f89bcf5d-dp6c6\" (UID: \"8f33dbb1-01ec-4e73-9cae-9a20e299c0b5\") " pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6"
Jul 07 16:48:05.082567 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:05.082466    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8f33dbb1-01ec-4e73-9cae-9a20e299c0b5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55f89bcf5d-dp6c6\" (UID: \"8f33dbb1-01ec-4e73-9cae-9a20e299c0b5\") " pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6"
Jul 07 16:48:05.083030 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:05.082980    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f33dbb1-01ec-4e73-9cae-9a20e299c0b5-metrics-client-ca\") pod \"thanos-querier-55f89bcf5d-dp6c6\" (UID: \"8f33dbb1-01ec-4e73-9cae-9a20e299c0b5\") " pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6"
Jul 07 16:48:05.085561 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:05.085534    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8f33dbb1-01ec-4e73-9cae-9a20e299c0b5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55f89bcf5d-dp6c6\" (UID: \"8f33dbb1-01ec-4e73-9cae-9a20e299c0b5\") " pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6"
Jul 07 16:48:05.085890 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:05.085866    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8f33dbb1-01ec-4e73-9cae-9a20e299c0b5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55f89bcf5d-dp6c6\" (UID: \"8f33dbb1-01ec-4e73-9cae-9a20e299c0b5\") " pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6"
Jul 07 16:48:05.086260 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:05.086239    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f33dbb1-01ec-4e73-9cae-9a20e299c0b5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55f89bcf5d-dp6c6\" (UID: \"8f33dbb1-01ec-4e73-9cae-9a20e299c0b5\") " pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6"
Jul 07 16:48:05.086353 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:05.086297    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8f33dbb1-01ec-4e73-9cae-9a20e299c0b5-secret-thanos-querier-tls\") pod \"thanos-querier-55f89bcf5d-dp6c6\" (UID: \"8f33dbb1-01ec-4e73-9cae-9a20e299c0b5\") " pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6"
Jul 07 16:48:05.087259 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:05.086987    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8f33dbb1-01ec-4e73-9cae-9a20e299c0b5-secret-grpc-tls\") pod \"thanos-querier-55f89bcf5d-dp6c6\" (UID: \"8f33dbb1-01ec-4e73-9cae-9a20e299c0b5\") " pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6"
Jul 07 16:48:05.087645 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:05.087608    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f33dbb1-01ec-4e73-9cae-9a20e299c0b5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55f89bcf5d-dp6c6\" (UID: \"8f33dbb1-01ec-4e73-9cae-9a20e299c0b5\") " pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6"
Jul 07 16:48:05.093383 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:05.093360    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75pl2\" (UniqueName: \"kubernetes.io/projected/8f33dbb1-01ec-4e73-9cae-9a20e299c0b5-kube-api-access-75pl2\") pod \"thanos-querier-55f89bcf5d-dp6c6\" (UID: \"8f33dbb1-01ec-4e73-9cae-9a20e299c0b5\") " pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6"
Jul 07 16:48:05.119235 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:05.119182    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:05.238032 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:05.237953    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6"
Jul 07 16:48:05.275035 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:05.274994    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:06.505698 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:06.505662    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:06.506564 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:06.506539    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-xzvh8"]
Jul 07 16:48:06.513992 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:06.511234    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-xzvh8"
Jul 07 16:48:06.516197 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:06.516163    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 07 16:48:06.516197 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:06.516181    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-x5jvs\""
Jul 07 16:48:06.523531 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:06.523493    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-xzvh8"]
Jul 07 16:48:06.594611 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:06.594572    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mxbv\" (UniqueName: \"kubernetes.io/projected/80c6c3b9-e9c7-4305-96cf-cb9124673a77-kube-api-access-5mxbv\") pod \"monitoring-plugin-d5dcff7cf-xzvh8\" (UID: \"80c6c3b9-e9c7-4305-96cf-cb9124673a77\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-xzvh8"
Jul 07 16:48:06.594796 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:06.594632    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/80c6c3b9-e9c7-4305-96cf-cb9124673a77-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-xzvh8\" (UID: \"80c6c3b9-e9c7-4305-96cf-cb9124673a77\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-xzvh8"
Jul 07 16:48:06.601383 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:06.601351    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:06.695576 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:06.695538    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mxbv\" (UniqueName: \"kubernetes.io/projected/80c6c3b9-e9c7-4305-96cf-cb9124673a77-kube-api-access-5mxbv\") pod \"monitoring-plugin-d5dcff7cf-xzvh8\" (UID: \"80c6c3b9-e9c7-4305-96cf-cb9124673a77\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-xzvh8"
Jul 07 16:48:06.695781 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:06.695596    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/80c6c3b9-e9c7-4305-96cf-cb9124673a77-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-xzvh8\" (UID: \"80c6c3b9-e9c7-4305-96cf-cb9124673a77\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-xzvh8"
Jul 07 16:48:06.695844 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:48:06.695780    2584 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 07 16:48:06.695897 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:48:06.695868    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80c6c3b9-e9c7-4305-96cf-cb9124673a77-monitoring-plugin-cert podName:80c6c3b9-e9c7-4305-96cf-cb9124673a77 nodeName:}" failed. No retries permitted until 2026-07-07 16:48:07.195846807 +0000 UTC m=+135.797055495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/80c6c3b9-e9c7-4305-96cf-cb9124673a77-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-xzvh8" (UID: "80c6c3b9-e9c7-4305-96cf-cb9124673a77") : secret "monitoring-plugin-cert" not found
Jul 07 16:48:06.706138 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:06.706101    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mxbv\" (UniqueName: \"kubernetes.io/projected/80c6c3b9-e9c7-4305-96cf-cb9124673a77-kube-api-access-5mxbv\") pod \"monitoring-plugin-d5dcff7cf-xzvh8\" (UID: \"80c6c3b9-e9c7-4305-96cf-cb9124673a77\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-xzvh8"
Jul 07 16:48:06.969172 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:06.969119    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:07.168438 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:07.168390    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:07.200496 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:07.200453    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/80c6c3b9-e9c7-4305-96cf-cb9124673a77-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-xzvh8\" (UID: \"80c6c3b9-e9c7-4305-96cf-cb9124673a77\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-xzvh8"
Jul 07 16:48:07.203288 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:07.203259    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/80c6c3b9-e9c7-4305-96cf-cb9124673a77-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-xzvh8\" (UID: \"80c6c3b9-e9c7-4305-96cf-cb9124673a77\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-xzvh8"
Jul 07 16:48:07.268861 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:07.268769    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:07.424867 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:07.424825    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-xzvh8"
Jul 07 16:48:07.458169 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:07.458132    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:07.585148 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:07.585031    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:08.205433 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.205397    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 16:48:08.208198 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.208155    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:08.210013 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.209993    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.230114 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.230084    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 07 16:48:08.231118 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.231096    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 07 16:48:08.234171 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.234133    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 07 16:48:08.234171 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.234152    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 07 16:48:08.234435 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.234408    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-56ld5\""
Jul 07 16:48:08.235726 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.235559    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 07 16:48:08.239548 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.239523    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 07 16:48:08.239548 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.239542    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 07 16:48:08.239724 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.239555    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 07 16:48:08.240415 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.240390    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 07 16:48:08.240672 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.240655    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 07 16:48:08.240844 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.240827    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 07 16:48:08.241444 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.241164    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8ilhn64kgvdtp\""
Jul 07 16:48:08.241444 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.241254    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 07 16:48:08.241913 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.241891    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 07 16:48:08.243953 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.243932    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 07 16:48:08.244163 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.244035    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 07 16:48:08.254108 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.254007    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 16:48:08.310662 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.310627    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/96441ff7-552d-4991-be04-8efb1673fbf3-config-out\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.310839 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.310675    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/96441ff7-552d-4991-be04-8efb1673fbf3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.310839 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.310694    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/96441ff7-552d-4991-be04-8efb1673fbf3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.310839 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.310757    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96441ff7-552d-4991-be04-8efb1673fbf3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.310839 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.310803    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96441ff7-552d-4991-be04-8efb1673fbf3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.310839 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.310836    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/96441ff7-552d-4991-be04-8efb1673fbf3-web-config\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.311102 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.310870    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/96441ff7-552d-4991-be04-8efb1673fbf3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.311102 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.310908    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/96441ff7-552d-4991-be04-8efb1673fbf3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.311102 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.310935    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96441ff7-552d-4991-be04-8efb1673fbf3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.311102 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.310959    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/96441ff7-552d-4991-be04-8efb1673fbf3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.311102 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.310983    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/96441ff7-552d-4991-be04-8efb1673fbf3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.311102 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.311008    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/96441ff7-552d-4991-be04-8efb1673fbf3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.311102 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.311035    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8xqg\" (UniqueName: \"kubernetes.io/projected/96441ff7-552d-4991-be04-8efb1673fbf3-kube-api-access-m8xqg\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.311417 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.311121    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96441ff7-552d-4991-be04-8efb1673fbf3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.311417 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.311149    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/96441ff7-552d-4991-be04-8efb1673fbf3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.311417 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.311221    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/96441ff7-552d-4991-be04-8efb1673fbf3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.311417 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.311290    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/96441ff7-552d-4991-be04-8efb1673fbf3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.311417 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.311333    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/96441ff7-552d-4991-be04-8efb1673fbf3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.311417 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.311371    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/96441ff7-552d-4991-be04-8efb1673fbf3-config\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.311417 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.311396    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/96441ff7-552d-4991-be04-8efb1673fbf3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.414261 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.412324    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/96441ff7-552d-4991-be04-8efb1673fbf3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.414261 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.412390    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/96441ff7-552d-4991-be04-8efb1673fbf3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.414261 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.412416    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96441ff7-552d-4991-be04-8efb1673fbf3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.414261 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.412439    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/96441ff7-552d-4991-be04-8efb1673fbf3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.414261 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.412461    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/96441ff7-552d-4991-be04-8efb1673fbf3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.414261 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.412485    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/96441ff7-552d-4991-be04-8efb1673fbf3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.414261 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.413546    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/96441ff7-552d-4991-be04-8efb1673fbf3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.414797 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.414390    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8xqg\" (UniqueName: \"kubernetes.io/projected/96441ff7-552d-4991-be04-8efb1673fbf3-kube-api-access-m8xqg\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.414797 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.414634    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96441ff7-552d-4991-be04-8efb1673fbf3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.415162 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.415130    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96441ff7-552d-4991-be04-8efb1673fbf3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.415552 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.415181    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/96441ff7-552d-4991-be04-8efb1673fbf3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.415552 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.415219    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/96441ff7-552d-4991-be04-8efb1673fbf3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.415552 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.415277    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/96441ff7-552d-4991-be04-8efb1673fbf3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.415552 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.415301    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/96441ff7-552d-4991-be04-8efb1673fbf3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.415552 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.415334    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/96441ff7-552d-4991-be04-8efb1673fbf3-config\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.415552 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.415357    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/96441ff7-552d-4991-be04-8efb1673fbf3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.415552 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.415419    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/96441ff7-552d-4991-be04-8efb1673fbf3-config-out\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.415552 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.415460    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/96441ff7-552d-4991-be04-8efb1673fbf3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.415552 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.415484    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/96441ff7-552d-4991-be04-8efb1673fbf3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.415552 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.415509    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96441ff7-552d-4991-be04-8efb1673fbf3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.415552 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.415535    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96441ff7-552d-4991-be04-8efb1673fbf3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.416161 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.415559    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/96441ff7-552d-4991-be04-8efb1673fbf3-web-config\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.416554 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.416523    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96441ff7-552d-4991-be04-8efb1673fbf3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.417366 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.417345    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/96441ff7-552d-4991-be04-8efb1673fbf3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.421971 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.417698    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96441ff7-552d-4991-be04-8efb1673fbf3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.421971 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.417901    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/96441ff7-552d-4991-be04-8efb1673fbf3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.421971 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.418628    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96441ff7-552d-4991-be04-8efb1673fbf3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.421971 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.419987    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/96441ff7-552d-4991-be04-8efb1673fbf3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.421971 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.420009    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/96441ff7-552d-4991-be04-8efb1673fbf3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.421971 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.420234    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/96441ff7-552d-4991-be04-8efb1673fbf3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.421971 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.420448    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/96441ff7-552d-4991-be04-8efb1673fbf3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.421971 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.421308    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/96441ff7-552d-4991-be04-8efb1673fbf3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.421971 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.421930    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/96441ff7-552d-4991-be04-8efb1673fbf3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.422476 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.422376    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/96441ff7-552d-4991-be04-8efb1673fbf3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.422884 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.422837    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/96441ff7-552d-4991-be04-8efb1673fbf3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.422995 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.422903    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/96441ff7-552d-4991-be04-8efb1673fbf3-config\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.423698 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.423663    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/96441ff7-552d-4991-be04-8efb1673fbf3-web-config\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.425907 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.425877    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/96441ff7-552d-4991-be04-8efb1673fbf3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.426325 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.426226    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/96441ff7-552d-4991-be04-8efb1673fbf3-config-out\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.427066 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.427029    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8xqg\" (UniqueName: \"kubernetes.io/projected/96441ff7-552d-4991-be04-8efb1673fbf3-kube-api-access-m8xqg\") pod \"prometheus-k8s-0\" (UID: \"96441ff7-552d-4991-be04-8efb1673fbf3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:08.522366 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:08.522270    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:10.933431 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:10.933327    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 16:48:10.938703 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:48:10.938674    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58834c6a_cd00_428f_8a2b_37b529c9aadf.slice/crio-9c9d4e31e368c60f2657f2a6813eadb16d4a14ed1761a2541dbfeddd11a19795 WatchSource:0}: Error finding container 9c9d4e31e368c60f2657f2a6813eadb16d4a14ed1761a2541dbfeddd11a19795: Status 404 returned error can't find the container with id 9c9d4e31e368c60f2657f2a6813eadb16d4a14ed1761a2541dbfeddd11a19795
Jul 07 16:48:11.002387 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:11.002311    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 16:48:11.005585 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:48:11.005554    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96441ff7_552d_4991_be04_8efb1673fbf3.slice/crio-4434ae04c560f37ac985d99443be16f58dc90bdc66e5eeb266388d109c770cda WatchSource:0}: Error finding container 4434ae04c560f37ac985d99443be16f58dc90bdc66e5eeb266388d109c770cda: Status 404 returned error can't find the container with id 4434ae04c560f37ac985d99443be16f58dc90bdc66e5eeb266388d109c770cda
Jul 07 16:48:11.157452 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:11.157375    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-xzvh8"]
Jul 07 16:48:11.163649 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:11.163483    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6"]
Jul 07 16:48:11.167749 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:48:11.167703    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f33dbb1_01ec_4e73_9cae_9a20e299c0b5.slice/crio-3ff7941d3c7ff4857435b3c6351b87607a0bf156e461714482b8c4ef96b12237 WatchSource:0}: Error finding container 3ff7941d3c7ff4857435b3c6351b87607a0bf156e461714482b8c4ef96b12237: Status 404 returned error can't find the container with id 3ff7941d3c7ff4857435b3c6351b87607a0bf156e461714482b8c4ef96b12237
Jul 07 16:48:11.466260 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:11.466221    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-xzvh8" event={"ID":"80c6c3b9-e9c7-4305-96cf-cb9124673a77","Type":"ContainerStarted","Data":"5ef1773f9ad2dc6095e1aacdf1e0ff9c72718ac0f6bb088ba28794501acff38a"}
Jul 07 16:48:11.468226 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:11.468138    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6" event={"ID":"8f33dbb1-01ec-4e73-9cae-9a20e299c0b5","Type":"ContainerStarted","Data":"3ff7941d3c7ff4857435b3c6351b87607a0bf156e461714482b8c4ef96b12237"}
Jul 07 16:48:11.471330 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:11.471254    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q6m8v" event={"ID":"ee4b1222-3d0f-485e-a082-745d5a8f1cb2","Type":"ContainerStarted","Data":"298c89b52820ef93090eb3f82cd6dfd34ef1929c9aa57306d83d23578b5e1ccc"}
Jul 07 16:48:11.473666 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:11.473572    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58834c6a-cd00-428f-8a2b-37b529c9aadf","Type":"ContainerStarted","Data":"9c9d4e31e368c60f2657f2a6813eadb16d4a14ed1761a2541dbfeddd11a19795"}
Jul 07 16:48:11.478574 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:11.478436    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-xvlzm" event={"ID":"c50b90e0-2467-43e9-ba20-83e720cab84d","Type":"ContainerStarted","Data":"df020bcb0520369822fb9d85ee341de66a0cbd20f662f41bdbd80eef451435dc"}
Jul 07 16:48:11.478574 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:11.478489    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-xvlzm"
Jul 07 16:48:11.479859 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:11.479778    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"96441ff7-552d-4991-be04-8efb1673fbf3","Type":"ContainerStarted","Data":"4434ae04c560f37ac985d99443be16f58dc90bdc66e5eeb266388d109c770cda"}
Jul 07 16:48:11.489722 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:11.489696    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-xvlzm"
Jul 07 16:48:11.528634 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:11.528570    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-xvlzm" podStartSLOduration=3.517057774 podStartE2EDuration="22.528549402s" totalImagesPullingTime="19.011491628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:47:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:11.502734821 +0000 UTC m=+140.103943536" watchObservedRunningTime="2026-07-07 16:48:11.528549402 +0000 UTC m=+140.129758110"
Jul 07 16:48:13.091026 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:13.090940    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:13.181549 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:13.181508    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:13.495125 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:13.494466    2584 generic.go:356] "Generic (PLEG): container finished" podID="ee4b1222-3d0f-485e-a082-745d5a8f1cb2" containerID="11e1c4288d0b9b6db366cd69666dfff5e957de4d33eba01bd0f447990112db54" exitCode=0
Jul 07 16:48:13.495125 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:13.494583    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q6m8v" event={"ID":"ee4b1222-3d0f-485e-a082-745d5a8f1cb2","Type":"ContainerDied","Data":"11e1c4288d0b9b6db366cd69666dfff5e957de4d33eba01bd0f447990112db54"}
Jul 07 16:48:13.499918 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:13.499875    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rw2p9" event={"ID":"2449686a-76f3-46db-806b-953bc02530f3","Type":"ContainerStarted","Data":"b9fdc2fcf374b9894d6a21f7fa4706568ee664569ae5fb7a8c3656b04da8704a"}
Jul 07 16:48:13.554778 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:13.554133    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rw2p9" podStartSLOduration=3.858401547 podStartE2EDuration="12.554112405s" totalImagesPullingTime="8.695710858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:48:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:13.553385572 +0000 UTC m=+142.154594278" watchObservedRunningTime="2026-07-07 16:48:13.554112405 +0000 UTC m=+142.155321111"
Jul 07 16:48:13.701856 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:13.701806    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:13.802826 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:13.802625    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:13.902083 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:13.902019    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:14.508922 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:14.508821    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q6m8v" event={"ID":"ee4b1222-3d0f-485e-a082-745d5a8f1cb2","Type":"ContainerStarted","Data":"bc9e5a162c10390056a9c94e21907550b76e533fa474dde20e7dfdc6c3481028"}
Jul 07 16:48:14.509455 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:14.508943    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q6m8v" event={"ID":"ee4b1222-3d0f-485e-a082-745d5a8f1cb2","Type":"ContainerStarted","Data":"a212333cfdaa2b874d5b36b7f26f5dcc763e72de1f89d3809b853714dbd4047f"}
Jul 07 16:48:14.512605 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:14.512560    2584 generic.go:356] "Generic (PLEG): container finished" podID="58834c6a-cd00-428f-8a2b-37b529c9aadf" containerID="fd6c5777f44cf7cbf498345f8d0758a0b7a1e398bf5dee301ec1e04af85ab36b" exitCode=0
Jul 07 16:48:14.512819 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:14.512791    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58834c6a-cd00-428f-8a2b-37b529c9aadf","Type":"ContainerDied","Data":"fd6c5777f44cf7cbf498345f8d0758a0b7a1e398bf5dee301ec1e04af85ab36b"}
Jul 07 16:48:14.516283 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:14.516251    2584 generic.go:356] "Generic (PLEG): container finished" podID="96441ff7-552d-4991-be04-8efb1673fbf3" containerID="6f43a09d517ffaab0f90a897102ba7bb998b001c3eaf443ce4d19147ac0766f5" exitCode=0
Jul 07 16:48:14.516456 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:14.516353    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"96441ff7-552d-4991-be04-8efb1673fbf3","Type":"ContainerDied","Data":"6f43a09d517ffaab0f90a897102ba7bb998b001c3eaf443ce4d19147ac0766f5"}
Jul 07 16:48:14.549924 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:14.549895    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-54b88fb4c5-zhjwb"]
Jul 07 16:48:14.555846 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:14.555665    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-q6m8v" podStartSLOduration=12.206646336 podStartE2EDuration="13.555641869s" totalImagesPullingTime="1.229195263s" totalInitContainerRuntime="119.80027ms" isStatefulPod=true podCreationTimestamp="2026-07-07 16:48:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:14.553329699 +0000 UTC m=+143.154538406" watchObservedRunningTime="2026-07-07 16:48:14.555641869 +0000 UTC m=+143.156850574"
Jul 07 16:48:14.688881 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:14.688841    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:14.825628 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:14.825554    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:15.528261 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:15.528224    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-xzvh8" event={"ID":"80c6c3b9-e9c7-4305-96cf-cb9124673a77","Type":"ContainerStarted","Data":"a12e74ffa4958aaa34c5cf51f54b267f52c14fae10deec8c7dc961be09fdb911"}
Jul 07 16:48:15.529130 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:15.529108    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-xzvh8"
Jul 07 16:48:15.539081 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:15.536423    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-xzvh8"
Jul 07 16:48:15.560160 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:15.560093    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-xzvh8" podStartSLOduration=5.725261638 podStartE2EDuration="9.560066628s" totalImagesPullingTime="3.83480499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:48:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:15.558647793 +0000 UTC m=+144.159856500" watchObservedRunningTime="2026-07-07 16:48:15.560066628 +0000 UTC m=+144.161275330"
Jul 07 16:48:16.213439 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:16.213238    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:16.309927 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:16.309839    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:16.548270 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:16.548030    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6" event={"ID":"8f33dbb1-01ec-4e73-9cae-9a20e299c0b5","Type":"ContainerStarted","Data":"2ae97a3b399f5ff0d867484400d33b9b331b73f3bbf0c2c999bbfe70ae1046be"}
Jul 07 16:48:16.548270 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:16.548187    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6" event={"ID":"8f33dbb1-01ec-4e73-9cae-9a20e299c0b5","Type":"ContainerStarted","Data":"7863f64a57db22f09eafc36e4fea67a6122669a43dc9fdad32fd60ba4825f564"}
Jul 07 16:48:16.548270 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:16.548225    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6" event={"ID":"8f33dbb1-01ec-4e73-9cae-9a20e299c0b5","Type":"ContainerStarted","Data":"e4a26e8759a8eeef88e01a7db272bc5d97b95b9439f1fee98eb623c647295cd8"}
Jul 07 16:48:17.475721 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:17.475680    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:17.577218 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:17.577030    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:20.572344 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:20.572258    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6" event={"ID":"8f33dbb1-01ec-4e73-9cae-9a20e299c0b5","Type":"ContainerStarted","Data":"3ad5678fd4426cdcbe417ec210c58742d218f81f8c4fcd7b0321f48001414d16"}
Jul 07 16:48:20.572868 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:20.572348    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6" event={"ID":"8f33dbb1-01ec-4e73-9cae-9a20e299c0b5","Type":"ContainerStarted","Data":"cc471ee99de1790aff6602df9e880077ca116a55b8de21a2f36b66599b729c0c"}
Jul 07 16:48:20.572868 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:20.572394    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6" event={"ID":"8f33dbb1-01ec-4e73-9cae-9a20e299c0b5","Type":"ContainerStarted","Data":"6ea1bbb04e835b86986df8db029527f0b1ca211fd3fd08591221308b7e2c2334"}
Jul 07 16:48:20.572868 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:20.572441    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6"
Jul 07 16:48:20.576000 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:20.575972    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58834c6a-cd00-428f-8a2b-37b529c9aadf","Type":"ContainerStarted","Data":"ef3aadac71082a34025405451354a93d8246d88a5b76db5865596f6a265ee0e1"}
Jul 07 16:48:20.576134 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:20.576037    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58834c6a-cd00-428f-8a2b-37b529c9aadf","Type":"ContainerStarted","Data":"948b4534ccc532d35d95372153064992e9e0b251185a8dcb8a8871ae2d0ae3a9"}
Jul 07 16:48:20.576134 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:20.576104    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58834c6a-cd00-428f-8a2b-37b529c9aadf","Type":"ContainerStarted","Data":"b2c737b40817ef2d92cc3e8dae8ecfe6d3a437cb5ca3760a436832d754d0ad8c"}
Jul 07 16:48:20.576134 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:20.576129    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58834c6a-cd00-428f-8a2b-37b529c9aadf","Type":"ContainerStarted","Data":"d29e43d50c66bf81b3ef39605144863d9e44b9a3d8035b8c1ad19cb8c2c39af0"}
Jul 07 16:48:20.576288 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:20.576165    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58834c6a-cd00-428f-8a2b-37b529c9aadf","Type":"ContainerStarted","Data":"067cef217399ae07d845cd326566ddb28d74b8db6f27f3d62aeb96961ceb72a7"}
Jul 07 16:48:20.576288 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:20.576206    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58834c6a-cd00-428f-8a2b-37b529c9aadf","Type":"ContainerStarted","Data":"53edd65088c2df2e452e14d9d45d4946b9190bc3646f38d168b224d0f81ad12e"}
Jul 07 16:48:20.579507 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:20.579487    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6"
Jul 07 16:48:20.579675 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:20.579659    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"96441ff7-552d-4991-be04-8efb1673fbf3","Type":"ContainerStarted","Data":"1635e07ba6ada0b30c26637ed057c689a1757d0b1db919ca852fb9726ee10148"}
Jul 07 16:48:20.579729 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:20.579703    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"96441ff7-552d-4991-be04-8efb1673fbf3","Type":"ContainerStarted","Data":"a77b9ea01fa295e3bb585860bd346d4daf28561cac4f9056061d81103ef03baf"}
Jul 07 16:48:20.579765 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:20.579734    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"96441ff7-552d-4991-be04-8efb1673fbf3","Type":"ContainerStarted","Data":"90a35499c0663069f933f006a12d7bbdd42426b4a74bd6ee0fe6abea230a2c27"}
Jul 07 16:48:20.579797 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:20.579762    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"96441ff7-552d-4991-be04-8efb1673fbf3","Type":"ContainerStarted","Data":"72ad132fd7bf5bc883b15bd1f2466b4bb4e5324927b27077e04306b50b4155b7"}
Jul 07 16:48:20.579828 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:20.579793    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"96441ff7-552d-4991-be04-8efb1673fbf3","Type":"ContainerStarted","Data":"c8a50a5e92086c229da438fef053fc68e5392480aa6ee542edd1ad3928f428d3"}
Jul 07 16:48:20.579828 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:20.579825    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"96441ff7-552d-4991-be04-8efb1673fbf3","Type":"ContainerStarted","Data":"87dcbc77e9f36b148c0bab18fec7e24097e09674c67ac12231862cdc824c7480"}
Jul 07 16:48:20.610826 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:20.610766    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-55f89bcf5d-dp6c6" podStartSLOduration=8.668252608 podStartE2EDuration="16.610744627s" totalImagesPullingTime="7.942492019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:48:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:20.608506894 +0000 UTC m=+149.209715599" watchObservedRunningTime="2026-07-07 16:48:20.610744627 +0000 UTC m=+149.211953332"
Jul 07 16:48:20.654345 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:20.654291    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.711001907 podStartE2EDuration="12.654273064s" totalImagesPullingTime="7.913363307s" totalInitContainerRuntime="29.90785ms" isStatefulPod=false podCreationTimestamp="2026-07-07 16:48:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:20.652112875 +0000 UTC m=+149.253321664" watchObservedRunningTime="2026-07-07 16:48:20.654273064 +0000 UTC m=+149.255481767"
Jul 07 16:48:20.721978 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:20.721915    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=10.718870249 podStartE2EDuration="18.721815857s" totalImagesPullingTime="7.973780056s" totalInitContainerRuntime="29.165552ms" isStatefulPod=false podCreationTimestamp="2026-07-07 16:48:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:48:20.720503021 +0000 UTC m=+149.321711763" watchObservedRunningTime="2026-07-07 16:48:20.721815857 +0000 UTC m=+149.323024562"
Jul 07 16:48:23.522773 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:23.522732    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:48:23.647399 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:23.647365    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c9f468564-9bc2g"]
Jul 07 16:48:23.785712 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:23.785626    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:23.892502 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:23.892462    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:28.550066 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:28.550006    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:28.657100 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:28.657023    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:28.738215 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:28.738183    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:30.622616 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:30.622580    2584 generic.go:356] "Generic (PLEG): container finished" podID="7b9d7134-446c-4c14-a1fb-92202cdde701" containerID="847f86386a357e2635e855505f4084107cb7d92bfe355b681dc5bafdaae8abdc" exitCode=0
Jul 07 16:48:30.623035 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:30.622623    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-v9xbq" event={"ID":"7b9d7134-446c-4c14-a1fb-92202cdde701","Type":"ContainerDied","Data":"847f86386a357e2635e855505f4084107cb7d92bfe355b681dc5bafdaae8abdc"}
Jul 07 16:48:30.623035 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:30.622911    2584 scope.go:117] "RemoveContainer" containerID="847f86386a357e2635e855505f4084107cb7d92bfe355b681dc5bafdaae8abdc"
Jul 07 16:48:31.627921 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:31.627885    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-v9xbq" event={"ID":"7b9d7134-446c-4c14-a1fb-92202cdde701","Type":"ContainerStarted","Data":"79b72d9ccfeec3fe07331ac903fa48e8edca2e53e1c474933527312b84ebcc72"}
Jul 07 16:48:32.507747 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:32.507717    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:32.586235 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:32.586206    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:39.580829 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:39.580763    2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb" podUID="2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2" containerName="registry" containerID="cri-o://5928cbf409274726e135f3f1b4364d3a261eb8c217186800723c102db8df5059" gracePeriod=30
Jul 07 16:48:39.850171 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:39.850140    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:48:39.946226 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:39.946196    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-registry-tls\") pod \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") "
Jul 07 16:48:39.946226 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:39.946233    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-ca-trust-extracted\") pod \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") "
Jul 07 16:48:39.946479 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:39.946268    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-registry-certificates\") pod \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") "
Jul 07 16:48:39.946479 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:39.946308    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-bound-sa-token\") pod \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") "
Jul 07 16:48:39.946479 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:39.946347    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-image-registry-private-configuration\") pod \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") "
Jul 07 16:48:39.946479 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:39.946406    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-installation-pull-secrets\") pod \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") "
Jul 07 16:48:39.946479 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:39.946460    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-trusted-ca\") pod \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") "
Jul 07 16:48:39.946729 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:39.946488    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvt6c\" (UniqueName: \"kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-kube-api-access-xvt6c\") pod \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\" (UID: \"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2\") "
Jul 07 16:48:39.946729 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:39.946671    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2" (UID: "2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:48:39.946907 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:39.946886    2584 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-registry-certificates\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 16:48:39.948211 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:39.948178    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2" (UID: "2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:48:39.949100 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:39.949071    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2" (UID: "2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:48:39.949344 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:39.949316    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2" (UID: "2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:48:39.949430 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:39.949384    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2" (UID: "2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:48:39.949430 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:39.949409    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2" (UID: "2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:48:39.949510 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:39.949434    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-kube-api-access-xvt6c" (OuterVolumeSpecName: "kube-api-access-xvt6c") pod "2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2" (UID: "2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2"). InnerVolumeSpecName "kube-api-access-xvt6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:48:39.955164 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:39.955142    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2" (UID: "2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:48:40.048076 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:40.048008    2584 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-bound-sa-token\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 16:48:40.048076 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:40.048072    2584 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-image-registry-private-configuration\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 16:48:40.048267 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:40.048089    2584 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-installation-pull-secrets\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 16:48:40.048267 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:40.048102    2584 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-trusted-ca\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 16:48:40.048267 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:40.048114    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvt6c\" (UniqueName: \"kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-kube-api-access-xvt6c\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 16:48:40.048267 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:40.048124    2584 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-registry-tls\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 16:48:40.048267 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:40.048136    2584 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2-ca-trust-extracted\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 16:48:40.675098 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:40.675038    2584 generic.go:356] "Generic (PLEG): container finished" podID="2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2" containerID="5928cbf409274726e135f3f1b4364d3a261eb8c217186800723c102db8df5059" exitCode=0
Jul 07 16:48:40.675545 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:40.675111    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb"
Jul 07 16:48:40.675545 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:40.675113    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb" event={"ID":"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2","Type":"ContainerDied","Data":"5928cbf409274726e135f3f1b4364d3a261eb8c217186800723c102db8df5059"}
Jul 07 16:48:40.675545 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:40.675230    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54b88fb4c5-zhjwb" event={"ID":"2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2","Type":"ContainerDied","Data":"b35bda9feacc2bd7a4f8ea6e5fff0bca21f9fa43d38e3d994790d3ce0130ffe8"}
Jul 07 16:48:40.675545 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:40.675274    2584 scope.go:117] "RemoveContainer" containerID="5928cbf409274726e135f3f1b4364d3a261eb8c217186800723c102db8df5059"
Jul 07 16:48:40.684509 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:40.684489    2584 scope.go:117] "RemoveContainer" containerID="5928cbf409274726e135f3f1b4364d3a261eb8c217186800723c102db8df5059"
Jul 07 16:48:40.684813 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:48:40.684792    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5928cbf409274726e135f3f1b4364d3a261eb8c217186800723c102db8df5059\": container with ID starting with 5928cbf409274726e135f3f1b4364d3a261eb8c217186800723c102db8df5059 not found: ID does not exist" containerID="5928cbf409274726e135f3f1b4364d3a261eb8c217186800723c102db8df5059"
Jul 07 16:48:40.684871 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:40.684822    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5928cbf409274726e135f3f1b4364d3a261eb8c217186800723c102db8df5059"} err="failed to get container status \"5928cbf409274726e135f3f1b4364d3a261eb8c217186800723c102db8df5059\": rpc error: code = NotFound desc = could not find container \"5928cbf409274726e135f3f1b4364d3a261eb8c217186800723c102db8df5059\": container with ID starting with 5928cbf409274726e135f3f1b4364d3a261eb8c217186800723c102db8df5059 not found: ID does not exist"
Jul 07 16:48:40.698901 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:40.698873    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-54b88fb4c5-zhjwb"]
Jul 07 16:48:40.707511 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:40.707484    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-54b88fb4c5-zhjwb"]
Jul 07 16:48:41.893506 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:41.893472    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2" path="/var/lib/kubelet/pods/2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2/volumes"
Jul 07 16:48:42.685848 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:42.685724    2584 generic.go:356] "Generic (PLEG): container finished" podID="f45b3a44-6bd5-4cee-912e-5edc4d056972" containerID="73bca516dce8d0764a95f986014c811e5f03b457c5e03b6dbd3891a83987f7ba" exitCode=0
Jul 07 16:48:42.685848 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:42.685799    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-dcrfk" event={"ID":"f45b3a44-6bd5-4cee-912e-5edc4d056972","Type":"ContainerDied","Data":"73bca516dce8d0764a95f986014c811e5f03b457c5e03b6dbd3891a83987f7ba"}
Jul 07 16:48:42.686167 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:42.686153    2584 scope.go:117] "RemoveContainer" containerID="73bca516dce8d0764a95f986014c811e5f03b457c5e03b6dbd3891a83987f7ba"
Jul 07 16:48:43.698649 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:43.698587    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-dcrfk" event={"ID":"f45b3a44-6bd5-4cee-912e-5edc4d056972","Type":"ContainerStarted","Data":"f4db151b5e7192bd310ff00e4edb1777adc31f3fe0a1f435e3c09512453e1d27"}
Jul 07 16:48:47.549112 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:47.549077    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:47.751017 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:47.750977    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:48:48.669846 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:48.669636    2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5c9f468564-9bc2g" podUID="83acf15b-a80a-4a21-bcdb-08f6080ccb0f" containerName="console" containerID="cri-o://dcaff062ac364fd1c25ce60de61e3e0f3a96419af3508ffa2949f60d66bbd3be" gracePeriod=15
Jul 07 16:48:48.998411 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:48.998381    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c9f468564-9bc2g"
Jul 07 16:48:49.130283 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:49.130245    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-oauth-serving-cert\") pod \"83acf15b-a80a-4a21-bcdb-08f6080ccb0f\" (UID: \"83acf15b-a80a-4a21-bcdb-08f6080ccb0f\") "
Jul 07 16:48:49.130580 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:49.130561    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-trusted-ca-bundle\") pod \"83acf15b-a80a-4a21-bcdb-08f6080ccb0f\" (UID: \"83acf15b-a80a-4a21-bcdb-08f6080ccb0f\") "
Jul 07 16:48:49.130760 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:49.130744    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldktj\" (UniqueName: \"kubernetes.io/projected/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-kube-api-access-ldktj\") pod \"83acf15b-a80a-4a21-bcdb-08f6080ccb0f\" (UID: \"83acf15b-a80a-4a21-bcdb-08f6080ccb0f\") "
Jul 07 16:48:49.130875 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:49.130860    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-service-ca\") pod \"83acf15b-a80a-4a21-bcdb-08f6080ccb0f\" (UID: \"83acf15b-a80a-4a21-bcdb-08f6080ccb0f\") "
Jul 07 16:48:49.130978 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:49.130965    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-console-config\") pod \"83acf15b-a80a-4a21-bcdb-08f6080ccb0f\" (UID: \"83acf15b-a80a-4a21-bcdb-08f6080ccb0f\") "
Jul 07 16:48:49.131126 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:49.131110    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-console-serving-cert\") pod \"83acf15b-a80a-4a21-bcdb-08f6080ccb0f\" (UID: \"83acf15b-a80a-4a21-bcdb-08f6080ccb0f\") "
Jul 07 16:48:49.131265 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:49.131248    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-console-oauth-config\") pod \"83acf15b-a80a-4a21-bcdb-08f6080ccb0f\" (UID: \"83acf15b-a80a-4a21-bcdb-08f6080ccb0f\") "
Jul 07 16:48:49.133223 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:49.132469    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-console-config" (OuterVolumeSpecName: "console-config") pod "83acf15b-a80a-4a21-bcdb-08f6080ccb0f" (UID: "83acf15b-a80a-4a21-bcdb-08f6080ccb0f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:48:49.133223 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:49.132721    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-service-ca" (OuterVolumeSpecName: "service-ca") pod "83acf15b-a80a-4a21-bcdb-08f6080ccb0f" (UID: "83acf15b-a80a-4a21-bcdb-08f6080ccb0f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:48:49.133223 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:49.132959    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "83acf15b-a80a-4a21-bcdb-08f6080ccb0f" (UID: "83acf15b-a80a-4a21-bcdb-08f6080ccb0f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:48:49.134137 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:49.134083    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "83acf15b-a80a-4a21-bcdb-08f6080ccb0f" (UID: "83acf15b-a80a-4a21-bcdb-08f6080ccb0f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:48:49.139993 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:49.139954    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-kube-api-access-ldktj" (OuterVolumeSpecName: "kube-api-access-ldktj") pod "83acf15b-a80a-4a21-bcdb-08f6080ccb0f" (UID: "83acf15b-a80a-4a21-bcdb-08f6080ccb0f"). InnerVolumeSpecName "kube-api-access-ldktj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:48:49.140162 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:49.140019    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "83acf15b-a80a-4a21-bcdb-08f6080ccb0f" (UID: "83acf15b-a80a-4a21-bcdb-08f6080ccb0f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:48:49.140162 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:49.140114    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "83acf15b-a80a-4a21-bcdb-08f6080ccb0f" (UID: "83acf15b-a80a-4a21-bcdb-08f6080ccb0f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:48:49.233063 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:49.232957    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldktj\" (UniqueName: \"kubernetes.io/projected/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-kube-api-access-ldktj\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 16:48:49.233063 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:49.232988    2584 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-service-ca\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 16:48:49.233063 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:49.233000    2584 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-console-config\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 16:48:49.233063 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:49.233012    2584 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-console-serving-cert\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 16:48:49.233063 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:49.233025    2584 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-console-oauth-config\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 16:48:49.233063 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:49.233036    2584 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-oauth-serving-cert\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 16:48:49.233357 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:49.233070    2584 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83acf15b-a80a-4a21-bcdb-08f6080ccb0f-trusted-ca-bundle\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 16:48:49.725129 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:49.725015    2584 generic.go:356] "Generic (PLEG): container finished" podID="83acf15b-a80a-4a21-bcdb-08f6080ccb0f" containerID="dcaff062ac364fd1c25ce60de61e3e0f3a96419af3508ffa2949f60d66bbd3be" exitCode=0
Jul 07 16:48:49.725129 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:49.725090    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c9f468564-9bc2g"
Jul 07 16:48:49.725129 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:49.725090    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c9f468564-9bc2g" event={"ID":"83acf15b-a80a-4a21-bcdb-08f6080ccb0f","Type":"ContainerDied","Data":"dcaff062ac364fd1c25ce60de61e3e0f3a96419af3508ffa2949f60d66bbd3be"}
Jul 07 16:48:49.725743 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:49.725153    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c9f468564-9bc2g" event={"ID":"83acf15b-a80a-4a21-bcdb-08f6080ccb0f","Type":"ContainerDied","Data":"06505609af6cb25c885738bc5040533d7c316f181f3875d08a9cee606d2eedbb"}
Jul 07 16:48:49.725743 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:49.725181    2584 scope.go:117] "RemoveContainer" containerID="dcaff062ac364fd1c25ce60de61e3e0f3a96419af3508ffa2949f60d66bbd3be"
Jul 07 16:48:49.748575 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:49.748067    2584 scope.go:117] "RemoveContainer" containerID="dcaff062ac364fd1c25ce60de61e3e0f3a96419af3508ffa2949f60d66bbd3be"
Jul 07 16:48:49.748575 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:48:49.748456    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcaff062ac364fd1c25ce60de61e3e0f3a96419af3508ffa2949f60d66bbd3be\": container with ID starting with dcaff062ac364fd1c25ce60de61e3e0f3a96419af3508ffa2949f60d66bbd3be not found: ID does not exist" containerID="dcaff062ac364fd1c25ce60de61e3e0f3a96419af3508ffa2949f60d66bbd3be"
Jul 07 16:48:49.748575 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:49.748491    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcaff062ac364fd1c25ce60de61e3e0f3a96419af3508ffa2949f60d66bbd3be"} err="failed to get container status \"dcaff062ac364fd1c25ce60de61e3e0f3a96419af3508ffa2949f60d66bbd3be\": rpc error: code = NotFound desc = could not find container \"dcaff062ac364fd1c25ce60de61e3e0f3a96419af3508ffa2949f60d66bbd3be\": container with ID starting with dcaff062ac364fd1c25ce60de61e3e0f3a96419af3508ffa2949f60d66bbd3be not found: ID does not exist"
Jul 07 16:48:49.758503 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:49.758470    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c9f468564-9bc2g"]
Jul 07 16:48:49.761451 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:49.761423    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c9f468564-9bc2g"]
Jul 07 16:48:49.899307 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:49.899275    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83acf15b-a80a-4a21-bcdb-08f6080ccb0f" path="/var/lib/kubelet/pods/83acf15b-a80a-4a21-bcdb-08f6080ccb0f/volumes"
Jul 07 16:48:59.235118 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:48:59.235079    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:49:08.523338 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:49:08.523300    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:08.545358 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:49:08.545330    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:08.821860 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:49:08.821779    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 16:49:18.071311 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:49:18.071269    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:49:18.158766 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:49:18.158724    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:49:30.141794 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:49:30.141756    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:49:30.278848 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:49:30.278803    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:49:32.554695 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:49:32.554655    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:49:32.651837 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:49:32.651736    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:49:36.666258 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:49:36.666221    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:49:36.764141 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:49:36.764094    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:49:36.864631 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:49:36.864595    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:49:37.407657 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:49:37.407617    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:49:37.493854 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:49:37.493816    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:49:47.190862 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:49:47.190804    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:49:47.304502 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:49:47.304464    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:49:47.499922 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:49:47.499823    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:49:47.621959 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:49:47.621922    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:50:06.767735 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:50:06.767697    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:50:06.861712 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:50:06.861674    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:50:29.895796 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:50:29.895706    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:50:47.894266 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:50:47.894230    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:50:49.281946 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:50:49.281909    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:50:49.387792 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:50:49.387754    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:50:49.690199 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:50:49.690161    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:50:49.811284 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:50:49.811254    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:50:49.908209 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:50:49.908178    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:50:51.809575 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:50:51.809550    2584 kubelet.go:1648] "Image garbage collection succeeded"
Jul 07 16:51:16.804169 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:51:16.804125    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:16.949330 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:51:16.949298    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:24.127936 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:51:24.127893    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:24.547962 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:51:24.547867    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:24.700823 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:51:24.700786    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:25.115831 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:51:25.115794    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:25.376755 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:51:25.376666    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:25.659254 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:51:25.659157    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:25.815753 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:51:25.815722    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:26.081605 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:51:26.081511    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:26.221252 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:51:26.221206    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:26.512164 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:51:26.512128    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:26.671845 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:51:26.671808    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:26.868066 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:51:26.868017    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:27.215425 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:51:27.215329    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:44.390482 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:51:44.390403    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:51:45.695187 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:51:45.695133    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:52:26.871332 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:52:26.871287    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:52:27.148336 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:52:27.148236    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:52:36.416224 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:52:36.416181    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:52:36.525645 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:52:36.525612    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:52:36.630600 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:52:36.630563    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:52:36.844688 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:52:36.844654    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:52:37.000868 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:52:37.000832    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:52:37.702978 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:52:37.702945    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:52:37.799326 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:52:37.799292    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:52:37.895901 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:52:37.895866    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:53:37.552721 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:53:37.552686    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:53:37.692266 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:53:37.692230    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:53:47.982836 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:53:47.982798    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:53:48.178341 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:53:48.178304    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:53:48.424520 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:53:48.424487    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:53:49.125770 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:53:49.125734    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:53:49.230948 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:53:49.230917    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:53:58.293026 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:53:58.292967    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:54:04.685490 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:54:04.685452    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-gj2jw"]
Jul 07 16:54:04.686007 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:54:04.685785    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2" containerName="registry"
Jul 07 16:54:04.686007 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:54:04.685796    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2" containerName="registry"
Jul 07 16:54:04.686007 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:54:04.685814    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83acf15b-a80a-4a21-bcdb-08f6080ccb0f" containerName="console"
Jul 07 16:54:04.686007 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:54:04.685819    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="83acf15b-a80a-4a21-bcdb-08f6080ccb0f" containerName="console"
Jul 07 16:54:04.686007 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:54:04.685870    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="83acf15b-a80a-4a21-bcdb-08f6080ccb0f" containerName="console"
Jul 07 16:54:04.686007 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:54:04.685880    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cbfbc07-f19d-49f8-8cb3-a1ea8e712eb2" containerName="registry"
Jul 07 16:54:04.687783 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:54:04.687768    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-gj2jw"
Jul 07 16:54:04.693073 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:54:04.693033    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 07 16:54:04.693220 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:54:04.693177    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 07 16:54:04.693220 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:54:04.693202    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 07 16:54:04.694530 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:54:04.694513    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-lfczg\""
Jul 07 16:54:04.703392 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:54:04.703366    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-gj2jw"]
Jul 07 16:54:04.716433 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:54:04.716407    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wdhq\" (UniqueName: \"kubernetes.io/projected/f5c865f5-0889-4158-a322-d7cb04474e3b-kube-api-access-5wdhq\") pod \"s3-init-gj2jw\" (UID: \"f5c865f5-0889-4158-a322-d7cb04474e3b\") " pod="kserve/s3-init-gj2jw"
Jul 07 16:54:04.716584 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:54:04.716504    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5c865f5-0889-4158-a322-d7cb04474e3b-model-cache\") pod \"s3-init-gj2jw\" (UID: \"f5c865f5-0889-4158-a322-d7cb04474e3b\") " pod="kserve/s3-init-gj2jw"
Jul 07 16:54:04.817512 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:54:04.817465    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wdhq\" (UniqueName: \"kubernetes.io/projected/f5c865f5-0889-4158-a322-d7cb04474e3b-kube-api-access-5wdhq\") pod \"s3-init-gj2jw\" (UID: \"f5c865f5-0889-4158-a322-d7cb04474e3b\") " pod="kserve/s3-init-gj2jw"
Jul 07 16:54:04.817716 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:54:04.817591    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5c865f5-0889-4158-a322-d7cb04474e3b-model-cache\") pod \"s3-init-gj2jw\" (UID: \"f5c865f5-0889-4158-a322-d7cb04474e3b\") " pod="kserve/s3-init-gj2jw"
Jul 07 16:54:04.817971 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:54:04.817949    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5c865f5-0889-4158-a322-d7cb04474e3b-model-cache\") pod \"s3-init-gj2jw\" (UID: \"f5c865f5-0889-4158-a322-d7cb04474e3b\") " pod="kserve/s3-init-gj2jw"
Jul 07 16:54:04.828975 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:54:04.828945    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wdhq\" (UniqueName: \"kubernetes.io/projected/f5c865f5-0889-4158-a322-d7cb04474e3b-kube-api-access-5wdhq\") pod \"s3-init-gj2jw\" (UID: \"f5c865f5-0889-4158-a322-d7cb04474e3b\") " pod="kserve/s3-init-gj2jw"
Jul 07 16:54:05.005670 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:54:05.005587    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-gj2jw"
Jul 07 16:54:05.145360 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:54:05.145328    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-gj2jw"]
Jul 07 16:54:05.148769 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:54:05.148742    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5c865f5_0889_4158_a322_d7cb04474e3b.slice/crio-2db8e7ba1c4ce84d6a4c32582d7ca81dac39196d17ce1eed39eb47b4e399b749 WatchSource:0}: Error finding container 2db8e7ba1c4ce84d6a4c32582d7ca81dac39196d17ce1eed39eb47b4e399b749: Status 404 returned error can't find the container with id 2db8e7ba1c4ce84d6a4c32582d7ca81dac39196d17ce1eed39eb47b4e399b749
Jul 07 16:54:05.151199 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:54:05.151184    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 16:54:06.008521 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:54:06.008475    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gj2jw" event={"ID":"f5c865f5-0889-4158-a322-d7cb04474e3b","Type":"ContainerStarted","Data":"2db8e7ba1c4ce84d6a4c32582d7ca81dac39196d17ce1eed39eb47b4e399b749"}
Jul 07 16:54:09.023678 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:54:09.023589    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gj2jw" event={"ID":"f5c865f5-0889-4158-a322-d7cb04474e3b","Type":"ContainerStarted","Data":"2e370a1d9fb800f13a70e9bd3199619619abd7af0f2001d43787a4c8ccc615c9"}
Jul 07 16:54:11.034147 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:54:11.034110    2584 generic.go:356] "Generic (PLEG): container finished" podID="f5c865f5-0889-4158-a322-d7cb04474e3b" containerID="2e370a1d9fb800f13a70e9bd3199619619abd7af0f2001d43787a4c8ccc615c9" exitCode=0
Jul 07 16:54:11.034555 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:54:11.034164    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gj2jw" event={"ID":"f5c865f5-0889-4158-a322-d7cb04474e3b","Type":"ContainerDied","Data":"2e370a1d9fb800f13a70e9bd3199619619abd7af0f2001d43787a4c8ccc615c9"}
Jul 07 16:54:16.059064 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:54:16.059012    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gj2jw" event={"ID":"f5c865f5-0889-4158-a322-d7cb04474e3b","Type":"ContainerStarted","Data":"12d312fec78c23a30f0adebba3f492c2048c1412bf913709a9f7f16aa3e9a6b4"}
Jul 07 16:54:43.715990 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:54:43.715907    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:55:21.342136 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:21.342093    2584 generic.go:356] "Generic (PLEG): container finished" podID="f5c865f5-0889-4158-a322-d7cb04474e3b" containerID="12d312fec78c23a30f0adebba3f492c2048c1412bf913709a9f7f16aa3e9a6b4" exitCode=0
Jul 07 16:55:21.342565 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:21.342161    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gj2jw" event={"ID":"f5c865f5-0889-4158-a322-d7cb04474e3b","Type":"ContainerDied","Data":"12d312fec78c23a30f0adebba3f492c2048c1412bf913709a9f7f16aa3e9a6b4"}
Jul 07 16:55:22.470267 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:22.470242    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-gj2jw"
Jul 07 16:55:22.597478 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:22.597385    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5c865f5-0889-4158-a322-d7cb04474e3b-model-cache\") pod \"f5c865f5-0889-4158-a322-d7cb04474e3b\" (UID: \"f5c865f5-0889-4158-a322-d7cb04474e3b\") "
Jul 07 16:55:22.597478 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:22.597426    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wdhq\" (UniqueName: \"kubernetes.io/projected/f5c865f5-0889-4158-a322-d7cb04474e3b-kube-api-access-5wdhq\") pod \"f5c865f5-0889-4158-a322-d7cb04474e3b\" (UID: \"f5c865f5-0889-4158-a322-d7cb04474e3b\") "
Jul 07 16:55:22.599846 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:22.599816    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5c865f5-0889-4158-a322-d7cb04474e3b-kube-api-access-5wdhq" (OuterVolumeSpecName: "kube-api-access-5wdhq") pod "f5c865f5-0889-4158-a322-d7cb04474e3b" (UID: "f5c865f5-0889-4158-a322-d7cb04474e3b"). InnerVolumeSpecName "kube-api-access-5wdhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:55:22.616778 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:22.616734    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5c865f5-0889-4158-a322-d7cb04474e3b-model-cache" (OuterVolumeSpecName: "model-cache") pod "f5c865f5-0889-4158-a322-d7cb04474e3b" (UID: "f5c865f5-0889-4158-a322-d7cb04474e3b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:55:22.699125 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:22.699090    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5c865f5-0889-4158-a322-d7cb04474e3b-model-cache\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 16:55:22.699125 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:22.699122    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wdhq\" (UniqueName: \"kubernetes.io/projected/f5c865f5-0889-4158-a322-d7cb04474e3b-kube-api-access-5wdhq\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 16:55:23.351853 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:23.351821    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-gj2jw"
Jul 07 16:55:23.352026 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:23.351853    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gj2jw" event={"ID":"f5c865f5-0889-4158-a322-d7cb04474e3b","Type":"ContainerDied","Data":"2db8e7ba1c4ce84d6a4c32582d7ca81dac39196d17ce1eed39eb47b4e399b749"}
Jul 07 16:55:23.352026 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:23.351890    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2db8e7ba1c4ce84d6a4c32582d7ca81dac39196d17ce1eed39eb47b4e399b749"
Jul 07 16:55:24.173072 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:24.173000    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:55:30.472191 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:30.472156    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-s2j2b"]
Jul 07 16:55:30.472576 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:30.472509    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5c865f5-0889-4158-a322-d7cb04474e3b" containerName="download-hf-model"
Jul 07 16:55:30.472576 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:30.472520    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5c865f5-0889-4158-a322-d7cb04474e3b" containerName="download-hf-model"
Jul 07 16:55:30.472576 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:30.472538    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5c865f5-0889-4158-a322-d7cb04474e3b" containerName="s3-init"
Jul 07 16:55:30.472576 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:30.472544    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5c865f5-0889-4158-a322-d7cb04474e3b" containerName="s3-init"
Jul 07 16:55:30.472722 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:30.472591    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5c865f5-0889-4158-a322-d7cb04474e3b" containerName="s3-init"
Jul 07 16:55:30.475920 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:30.475904    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-s2j2b"
Jul 07 16:55:30.479164 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:30.479137    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 07 16:55:30.479310 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:30.479166    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 07 16:55:30.479310 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:30.479138    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 07 16:55:30.480254 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:30.480239    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-lfczg\""
Jul 07 16:55:30.488097 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:30.488073    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-s2j2b"]
Jul 07 16:55:30.561627 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:30.561593    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eda6df85-b2f8-4a77-8641-7df86bce9d6a-model-cache\") pod \"s3-tls-init-custom-s2j2b\" (UID: \"eda6df85-b2f8-4a77-8641-7df86bce9d6a\") " pod="kserve/s3-tls-init-custom-s2j2b"
Jul 07 16:55:30.561804 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:30.561690    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq7px\" (UniqueName: \"kubernetes.io/projected/eda6df85-b2f8-4a77-8641-7df86bce9d6a-kube-api-access-tq7px\") pod \"s3-tls-init-custom-s2j2b\" (UID: \"eda6df85-b2f8-4a77-8641-7df86bce9d6a\") " pod="kserve/s3-tls-init-custom-s2j2b"
Jul 07 16:55:30.662926 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:30.662879    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq7px\" (UniqueName: \"kubernetes.io/projected/eda6df85-b2f8-4a77-8641-7df86bce9d6a-kube-api-access-tq7px\") pod \"s3-tls-init-custom-s2j2b\" (UID: \"eda6df85-b2f8-4a77-8641-7df86bce9d6a\") " pod="kserve/s3-tls-init-custom-s2j2b"
Jul 07 16:55:30.663151 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:30.662959    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eda6df85-b2f8-4a77-8641-7df86bce9d6a-model-cache\") pod \"s3-tls-init-custom-s2j2b\" (UID: \"eda6df85-b2f8-4a77-8641-7df86bce9d6a\") " pod="kserve/s3-tls-init-custom-s2j2b"
Jul 07 16:55:30.663351 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:30.663333    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eda6df85-b2f8-4a77-8641-7df86bce9d6a-model-cache\") pod \"s3-tls-init-custom-s2j2b\" (UID: \"eda6df85-b2f8-4a77-8641-7df86bce9d6a\") " pod="kserve/s3-tls-init-custom-s2j2b"
Jul 07 16:55:30.674117 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:30.674081    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq7px\" (UniqueName: \"kubernetes.io/projected/eda6df85-b2f8-4a77-8641-7df86bce9d6a-kube-api-access-tq7px\") pod \"s3-tls-init-custom-s2j2b\" (UID: \"eda6df85-b2f8-4a77-8641-7df86bce9d6a\") " pod="kserve/s3-tls-init-custom-s2j2b"
Jul 07 16:55:30.793725 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:30.793638    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-s2j2b"
Jul 07 16:55:30.943237 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:30.943201    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-s2j2b"]
Jul 07 16:55:30.946858 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:55:30.946828    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeda6df85_b2f8_4a77_8641_7df86bce9d6a.slice/crio-e783fda41977ef6d6fb61aa7cc0c93cbddaef86dee8512e2d677dafb3ce20c0e WatchSource:0}: Error finding container e783fda41977ef6d6fb61aa7cc0c93cbddaef86dee8512e2d677dafb3ce20c0e: Status 404 returned error can't find the container with id e783fda41977ef6d6fb61aa7cc0c93cbddaef86dee8512e2d677dafb3ce20c0e
Jul 07 16:55:31.387555 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:31.387518    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-s2j2b" event={"ID":"eda6df85-b2f8-4a77-8641-7df86bce9d6a","Type":"ContainerStarted","Data":"36fcfb343986dff215f8329bbe9b6731cf4a54f0b5ebd4e9afb31ed7b88fd190"}
Jul 07 16:55:31.387717 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:31.387576    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-s2j2b" event={"ID":"eda6df85-b2f8-4a77-8641-7df86bce9d6a","Type":"ContainerStarted","Data":"e783fda41977ef6d6fb61aa7cc0c93cbddaef86dee8512e2d677dafb3ce20c0e"}
Jul 07 16:55:36.409811 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:36.409778    2584 generic.go:356] "Generic (PLEG): container finished" podID="eda6df85-b2f8-4a77-8641-7df86bce9d6a" containerID="36fcfb343986dff215f8329bbe9b6731cf4a54f0b5ebd4e9afb31ed7b88fd190" exitCode=0
Jul 07 16:55:36.410197 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:36.409821    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-s2j2b" event={"ID":"eda6df85-b2f8-4a77-8641-7df86bce9d6a","Type":"ContainerDied","Data":"36fcfb343986dff215f8329bbe9b6731cf4a54f0b5ebd4e9afb31ed7b88fd190"}
Jul 07 16:55:37.415697 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:37.415661    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-s2j2b" event={"ID":"eda6df85-b2f8-4a77-8641-7df86bce9d6a","Type":"ContainerStarted","Data":"1caaff6ee954ce5036b01fe57385c0286de6e9dc0b279bcde1bd08c6e7f58a4d"}
Jul 07 16:55:37.437599 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:37.437545    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-s2j2b" podStartSLOduration=2.184294001 podStartE2EDuration="7.437526922s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.253232921s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:55:37.436971441 +0000 UTC m=+586.038180147" watchObservedRunningTime="2026-07-07 16:55:37.437526922 +0000 UTC m=+586.038735627"
Jul 07 16:55:44.448070 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:44.448017    2584 generic.go:356] "Generic (PLEG): container finished" podID="eda6df85-b2f8-4a77-8641-7df86bce9d6a" containerID="1caaff6ee954ce5036b01fe57385c0286de6e9dc0b279bcde1bd08c6e7f58a4d" exitCode=0
Jul 07 16:55:44.448472 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:44.448089    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-s2j2b" event={"ID":"eda6df85-b2f8-4a77-8641-7df86bce9d6a","Type":"ContainerDied","Data":"1caaff6ee954ce5036b01fe57385c0286de6e9dc0b279bcde1bd08c6e7f58a4d"}
Jul 07 16:55:45.626590 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:45.626563    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-s2j2b"
Jul 07 16:55:45.698359 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:45.698315    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tq7px\" (UniqueName: \"kubernetes.io/projected/eda6df85-b2f8-4a77-8641-7df86bce9d6a-kube-api-access-tq7px\") pod \"eda6df85-b2f8-4a77-8641-7df86bce9d6a\" (UID: \"eda6df85-b2f8-4a77-8641-7df86bce9d6a\") "
Jul 07 16:55:45.698526 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:45.698392    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eda6df85-b2f8-4a77-8641-7df86bce9d6a-model-cache\") pod \"eda6df85-b2f8-4a77-8641-7df86bce9d6a\" (UID: \"eda6df85-b2f8-4a77-8641-7df86bce9d6a\") "
Jul 07 16:55:45.700624 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:45.700590    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eda6df85-b2f8-4a77-8641-7df86bce9d6a-kube-api-access-tq7px" (OuterVolumeSpecName: "kube-api-access-tq7px") pod "eda6df85-b2f8-4a77-8641-7df86bce9d6a" (UID: "eda6df85-b2f8-4a77-8641-7df86bce9d6a"). InnerVolumeSpecName "kube-api-access-tq7px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:55:45.717543 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:45.717504    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eda6df85-b2f8-4a77-8641-7df86bce9d6a-model-cache" (OuterVolumeSpecName: "model-cache") pod "eda6df85-b2f8-4a77-8641-7df86bce9d6a" (UID: "eda6df85-b2f8-4a77-8641-7df86bce9d6a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:55:45.799031 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:45.798984    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eda6df85-b2f8-4a77-8641-7df86bce9d6a-model-cache\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 16:55:45.799031 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:45.799020    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq7px\" (UniqueName: \"kubernetes.io/projected/eda6df85-b2f8-4a77-8641-7df86bce9d6a-kube-api-access-tq7px\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 16:55:46.458064 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:46.458026    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-s2j2b"
Jul 07 16:55:46.458064 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:46.458060    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-s2j2b" event={"ID":"eda6df85-b2f8-4a77-8641-7df86bce9d6a","Type":"ContainerDied","Data":"e783fda41977ef6d6fb61aa7cc0c93cbddaef86dee8512e2d677dafb3ce20c0e"}
Jul 07 16:55:46.458268 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:46.458093    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e783fda41977ef6d6fb61aa7cc0c93cbddaef86dee8512e2d677dafb3ce20c0e"
Jul 07 16:55:47.136455 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:47.136415    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:55:47.495296 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:47.495205    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:55:49.824423 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:49.824383    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-rpzc4"]
Jul 07 16:55:49.824895 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:49.824862    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eda6df85-b2f8-4a77-8641-7df86bce9d6a" containerName="s3-tls-init-custom"
Jul 07 16:55:49.824895 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:49.824881    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="eda6df85-b2f8-4a77-8641-7df86bce9d6a" containerName="s3-tls-init-custom"
Jul 07 16:55:49.825022 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:49.824933    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eda6df85-b2f8-4a77-8641-7df86bce9d6a" containerName="download-hf-model"
Jul 07 16:55:49.825022 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:49.824944    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="eda6df85-b2f8-4a77-8641-7df86bce9d6a" containerName="download-hf-model"
Jul 07 16:55:49.825150 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:49.825030    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="eda6df85-b2f8-4a77-8641-7df86bce9d6a" containerName="s3-tls-init-custom"
Jul 07 16:55:49.828207 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:49.828185    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-rpzc4"
Jul 07 16:55:49.831781 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:49.831761    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 07 16:55:49.831901 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:49.831851    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-lfczg\""
Jul 07 16:55:49.832773 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:49.832755    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 07 16:55:49.832878 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:49.832800    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 07 16:55:49.840934 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:49.840895    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-rpzc4"]
Jul 07 16:55:49.935896 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:49.935861    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrvk8\" (UniqueName: \"kubernetes.io/projected/c35abd2a-853f-4e82-8faf-635a125534a4-kube-api-access-qrvk8\") pod \"s3-tls-init-serving-rpzc4\" (UID: \"c35abd2a-853f-4e82-8faf-635a125534a4\") " pod="kserve/s3-tls-init-serving-rpzc4"
Jul 07 16:55:49.936081 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:49.935909    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c35abd2a-853f-4e82-8faf-635a125534a4-model-cache\") pod \"s3-tls-init-serving-rpzc4\" (UID: \"c35abd2a-853f-4e82-8faf-635a125534a4\") " pod="kserve/s3-tls-init-serving-rpzc4"
Jul 07 16:55:50.037300 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:50.037269    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrvk8\" (UniqueName: \"kubernetes.io/projected/c35abd2a-853f-4e82-8faf-635a125534a4-kube-api-access-qrvk8\") pod \"s3-tls-init-serving-rpzc4\" (UID: \"c35abd2a-853f-4e82-8faf-635a125534a4\") " pod="kserve/s3-tls-init-serving-rpzc4"
Jul 07 16:55:50.037443 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:50.037318    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c35abd2a-853f-4e82-8faf-635a125534a4-model-cache\") pod \"s3-tls-init-serving-rpzc4\" (UID: \"c35abd2a-853f-4e82-8faf-635a125534a4\") " pod="kserve/s3-tls-init-serving-rpzc4"
Jul 07 16:55:50.037651 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:50.037636    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c35abd2a-853f-4e82-8faf-635a125534a4-model-cache\") pod \"s3-tls-init-serving-rpzc4\" (UID: \"c35abd2a-853f-4e82-8faf-635a125534a4\") " pod="kserve/s3-tls-init-serving-rpzc4"
Jul 07 16:55:50.047938 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:50.047916    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrvk8\" (UniqueName: \"kubernetes.io/projected/c35abd2a-853f-4e82-8faf-635a125534a4-kube-api-access-qrvk8\") pod \"s3-tls-init-serving-rpzc4\" (UID: \"c35abd2a-853f-4e82-8faf-635a125534a4\") " pod="kserve/s3-tls-init-serving-rpzc4"
Jul 07 16:55:50.155066 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:50.155018    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-rpzc4"
Jul 07 16:55:50.298162 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:50.298135    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-rpzc4"]
Jul 07 16:55:50.300664 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:55:50.300640    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc35abd2a_853f_4e82_8faf_635a125534a4.slice/crio-3d479f6d7198e32c1d07c3cf0081688503c82de5857fa59449bd741a1f7394e3 WatchSource:0}: Error finding container 3d479f6d7198e32c1d07c3cf0081688503c82de5857fa59449bd741a1f7394e3: Status 404 returned error can't find the container with id 3d479f6d7198e32c1d07c3cf0081688503c82de5857fa59449bd741a1f7394e3
Jul 07 16:55:50.483464 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:50.483371    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-rpzc4" event={"ID":"c35abd2a-853f-4e82-8faf-635a125534a4","Type":"ContainerStarted","Data":"8ec9af5e1bddf8544b029b9f3ed9e677d52ae37275f1f86ab464b19d602141dc"}
Jul 07 16:55:50.483464 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:50.483415    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-rpzc4" event={"ID":"c35abd2a-853f-4e82-8faf-635a125534a4","Type":"ContainerStarted","Data":"3d479f6d7198e32c1d07c3cf0081688503c82de5857fa59449bd741a1f7394e3"}
Jul 07 16:55:53.496915 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:53.496882    2584 generic.go:356] "Generic (PLEG): container finished" podID="c35abd2a-853f-4e82-8faf-635a125534a4" containerID="8ec9af5e1bddf8544b029b9f3ed9e677d52ae37275f1f86ab464b19d602141dc" exitCode=0
Jul 07 16:55:53.497346 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:53.496960    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-rpzc4" event={"ID":"c35abd2a-853f-4e82-8faf-635a125534a4","Type":"ContainerDied","Data":"8ec9af5e1bddf8544b029b9f3ed9e677d52ae37275f1f86ab464b19d602141dc"}
Jul 07 16:55:53.507258 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:53.507235    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 07 16:55:54.502637 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:54.502602    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-rpzc4" event={"ID":"c35abd2a-853f-4e82-8faf-635a125534a4","Type":"ContainerStarted","Data":"ba2729b34f097703557befe80cfaf6ddafd8fc9dfc0671e0bf9b4e73151e044c"}
Jul 07 16:55:54.524312 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:54.524251    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-rpzc4" podStartSLOduration=2.593763279 podStartE2EDuration="5.524232883s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.930469604s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:55:54.52335245 +0000 UTC m=+603.124561178" watchObservedRunningTime="2026-07-07 16:55:54.524232883 +0000 UTC m=+603.125441622"
Jul 07 16:55:59.526724 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:59.526685    2584 generic.go:356] "Generic (PLEG): container finished" podID="c35abd2a-853f-4e82-8faf-635a125534a4" containerID="ba2729b34f097703557befe80cfaf6ddafd8fc9dfc0671e0bf9b4e73151e044c" exitCode=0
Jul 07 16:55:59.527109 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:55:59.526758    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-rpzc4" event={"ID":"c35abd2a-853f-4e82-8faf-635a125534a4","Type":"ContainerDied","Data":"ba2729b34f097703557befe80cfaf6ddafd8fc9dfc0671e0bf9b4e73151e044c"}
Jul 07 16:56:00.672766 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:00.672741    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-rpzc4"
Jul 07 16:56:00.731960 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:00.731910    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrvk8\" (UniqueName: \"kubernetes.io/projected/c35abd2a-853f-4e82-8faf-635a125534a4-kube-api-access-qrvk8\") pod \"c35abd2a-853f-4e82-8faf-635a125534a4\" (UID: \"c35abd2a-853f-4e82-8faf-635a125534a4\") "
Jul 07 16:56:00.732150 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:00.732064    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c35abd2a-853f-4e82-8faf-635a125534a4-model-cache\") pod \"c35abd2a-853f-4e82-8faf-635a125534a4\" (UID: \"c35abd2a-853f-4e82-8faf-635a125534a4\") "
Jul 07 16:56:00.734238 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:00.734205    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c35abd2a-853f-4e82-8faf-635a125534a4-kube-api-access-qrvk8" (OuterVolumeSpecName: "kube-api-access-qrvk8") pod "c35abd2a-853f-4e82-8faf-635a125534a4" (UID: "c35abd2a-853f-4e82-8faf-635a125534a4"). InnerVolumeSpecName "kube-api-access-qrvk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:56:00.755059 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:00.755000    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c35abd2a-853f-4e82-8faf-635a125534a4-model-cache" (OuterVolumeSpecName: "model-cache") pod "c35abd2a-853f-4e82-8faf-635a125534a4" (UID: "c35abd2a-853f-4e82-8faf-635a125534a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:56:00.832570 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:00.832491    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrvk8\" (UniqueName: \"kubernetes.io/projected/c35abd2a-853f-4e82-8faf-635a125534a4-kube-api-access-qrvk8\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 16:56:00.832570 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:00.832518    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c35abd2a-853f-4e82-8faf-635a125534a4-model-cache\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 16:56:01.536000 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:01.535972    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-rpzc4"
Jul 07 16:56:01.536000 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:01.535996    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-rpzc4" event={"ID":"c35abd2a-853f-4e82-8faf-635a125534a4","Type":"ContainerDied","Data":"3d479f6d7198e32c1d07c3cf0081688503c82de5857fa59449bd741a1f7394e3"}
Jul 07 16:56:01.536223 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:01.536029    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d479f6d7198e32c1d07c3cf0081688503c82de5857fa59449bd741a1f7394e3"
Jul 07 16:56:01.939153 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:01.939118    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:56:02.039058 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:02.038999    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:56:13.848256 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:13.848147    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:56:13.955832 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:13.955792    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f"]
Jul 07 16:56:13.956233 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:13.956215    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c35abd2a-853f-4e82-8faf-635a125534a4" containerName="s3-tls-init-serving"
Jul 07 16:56:13.956333 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:13.956234    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c35abd2a-853f-4e82-8faf-635a125534a4" containerName="s3-tls-init-serving"
Jul 07 16:56:13.956333 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:13.956252    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c35abd2a-853f-4e82-8faf-635a125534a4" containerName="download-hf-model"
Jul 07 16:56:13.956333 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:13.956261    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c35abd2a-853f-4e82-8faf-635a125534a4" containerName="download-hf-model"
Jul 07 16:56:13.956488 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:13.956375    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c35abd2a-853f-4e82-8faf-635a125534a4" containerName="s3-tls-init-serving"
Jul 07 16:56:13.959509 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:13.959487    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f"
Jul 07 16:56:13.962532 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:13.962508    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 07 16:56:13.962669 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:13.962513    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 16:56:13.962669 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:13.962518    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 16:56:13.962669 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:13.962524    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 16:56:13.962669 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:13.962536    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 16:56:13.962977 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:13.962962    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 07 16:56:13.963762 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:13.963749    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7526s\""
Jul 07 16:56:13.982655 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:13.982625    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f"]
Jul 07 16:56:14.039452 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:14.039411    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48f6db63-3224-48ce-9cef-e140d5600ee7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f\" (UID: \"48f6db63-3224-48ce-9cef-e140d5600ee7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f"
Jul 07 16:56:14.039621 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:14.039470    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48f6db63-3224-48ce-9cef-e140d5600ee7-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f\" (UID: \"48f6db63-3224-48ce-9cef-e140d5600ee7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f"
Jul 07 16:56:14.039621 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:14.039498    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48f6db63-3224-48ce-9cef-e140d5600ee7-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f\" (UID: \"48f6db63-3224-48ce-9cef-e140d5600ee7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f"
Jul 07 16:56:14.039621 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:14.039517    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g2rr\" (UniqueName: \"kubernetes.io/projected/48f6db63-3224-48ce-9cef-e140d5600ee7-kube-api-access-8g2rr\") pod \"isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f\" (UID: \"48f6db63-3224-48ce-9cef-e140d5600ee7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f"
Jul 07 16:56:14.039621 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:14.039587    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48f6db63-3224-48ce-9cef-e140d5600ee7-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f\" (UID: \"48f6db63-3224-48ce-9cef-e140d5600ee7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f"
Jul 07 16:56:14.125565 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:14.125471    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:56:14.140534 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:14.140500    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48f6db63-3224-48ce-9cef-e140d5600ee7-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f\" (UID: \"48f6db63-3224-48ce-9cef-e140d5600ee7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f"
Jul 07 16:56:14.140645 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:14.140547    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48f6db63-3224-48ce-9cef-e140d5600ee7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f\" (UID: \"48f6db63-3224-48ce-9cef-e140d5600ee7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f"
Jul 07 16:56:14.140645 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:14.140588    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48f6db63-3224-48ce-9cef-e140d5600ee7-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f\" (UID: \"48f6db63-3224-48ce-9cef-e140d5600ee7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f"
Jul 07 16:56:14.140645 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:14.140606    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48f6db63-3224-48ce-9cef-e140d5600ee7-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f\" (UID: \"48f6db63-3224-48ce-9cef-e140d5600ee7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f"
Jul 07 16:56:14.140645 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:14.140622    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g2rr\" (UniqueName: \"kubernetes.io/projected/48f6db63-3224-48ce-9cef-e140d5600ee7-kube-api-access-8g2rr\") pod \"isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f\" (UID: \"48f6db63-3224-48ce-9cef-e140d5600ee7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f"
Jul 07 16:56:14.141104 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:14.141078    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48f6db63-3224-48ce-9cef-e140d5600ee7-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f\" (UID: \"48f6db63-3224-48ce-9cef-e140d5600ee7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f"
Jul 07 16:56:14.141284 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:14.141268    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48f6db63-3224-48ce-9cef-e140d5600ee7-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f\" (UID: \"48f6db63-3224-48ce-9cef-e140d5600ee7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f"
Jul 07 16:56:14.141335 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:14.141313    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48f6db63-3224-48ce-9cef-e140d5600ee7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f\" (UID: \"48f6db63-3224-48ce-9cef-e140d5600ee7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f"
Jul 07 16:56:14.143180 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:14.143163    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48f6db63-3224-48ce-9cef-e140d5600ee7-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f\" (UID: \"48f6db63-3224-48ce-9cef-e140d5600ee7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f"
Jul 07 16:56:14.153838 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:14.153809    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g2rr\" (UniqueName: \"kubernetes.io/projected/48f6db63-3224-48ce-9cef-e140d5600ee7-kube-api-access-8g2rr\") pod \"isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f\" (UID: \"48f6db63-3224-48ce-9cef-e140d5600ee7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f"
Jul 07 16:56:14.272802 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:14.272759    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f"
Jul 07 16:56:14.428983 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:14.428862    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f"]
Jul 07 16:56:14.431915 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:56:14.431886    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f6db63_3224_48ce_9cef_e140d5600ee7.slice/crio-8bb211b129225e838e894a92a3f4e8dc634fa5a75176de749c6b50fab85995fc WatchSource:0}: Error finding container 8bb211b129225e838e894a92a3f4e8dc634fa5a75176de749c6b50fab85995fc: Status 404 returned error can't find the container with id 8bb211b129225e838e894a92a3f4e8dc634fa5a75176de749c6b50fab85995fc
Jul 07 16:56:14.589722 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:14.589683    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" event={"ID":"48f6db63-3224-48ce-9cef-e140d5600ee7","Type":"ContainerStarted","Data":"578ee47d51c025c7c24ff9efa6475cb0852ad99ca6e2c1d8cbf84164c6cf21d7"}
Jul 07 16:56:14.589920 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:14.589749    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" event={"ID":"48f6db63-3224-48ce-9cef-e140d5600ee7","Type":"ContainerStarted","Data":"8bb211b129225e838e894a92a3f4e8dc634fa5a75176de749c6b50fab85995fc"}
Jul 07 16:56:18.609167 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:18.609133    2584 generic.go:356] "Generic (PLEG): container finished" podID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerID="578ee47d51c025c7c24ff9efa6475cb0852ad99ca6e2c1d8cbf84164c6cf21d7" exitCode=0
Jul 07 16:56:18.609695 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:18.609212    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" event={"ID":"48f6db63-3224-48ce-9cef-e140d5600ee7","Type":"ContainerDied","Data":"578ee47d51c025c7c24ff9efa6475cb0852ad99ca6e2c1d8cbf84164c6cf21d7"}
Jul 07 16:56:32.692578 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:32.692494    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" event={"ID":"48f6db63-3224-48ce-9cef-e140d5600ee7","Type":"ContainerStarted","Data":"da18d2efd831c10583cdd9bb280a89f26d67f40fb7818aab5cd2b169d736cb2f"}
Jul 07 16:56:34.703677 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:34.703640    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" event={"ID":"48f6db63-3224-48ce-9cef-e140d5600ee7","Type":"ContainerStarted","Data":"264b39c44aa9cfe7f3cbb64c88e58be22919ab04130b2508022b6c0f4fc9be64"}
Jul 07 16:56:37.719287 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:37.719255    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" event={"ID":"48f6db63-3224-48ce-9cef-e140d5600ee7","Type":"ContainerStarted","Data":"578b7dcc69d927f717a235b2855f06b45ae2b0eccc474a0f3300d4d66fa8a6de"}
Jul 07 16:56:37.719698 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:37.719530    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f"
Jul 07 16:56:37.719698 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:37.719682    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f"
Jul 07 16:56:37.720926 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:37.720893    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 07 16:56:37.746718 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:37.746667    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" podStartSLOduration=2.055444527 podStartE2EDuration="24.746648654s" totalImagesPullingTime="18.780701117s" totalInitContainerRuntime="3.91050301s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:56:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:56:37.746164434 +0000 UTC m=+646.347373137" watchObservedRunningTime="2026-07-07 16:56:37.746648654 +0000 UTC m=+646.347857357"
Jul 07 16:56:38.724336 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:38.724305    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f"
Jul 07 16:56:38.724739 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:38.724305    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 07 16:56:38.725502 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:38.725475    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:56:39.728320 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:39.728276    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 07 16:56:39.728720 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:39.728692    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:56:39.732095 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:39.732078    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f"
Jul 07 16:56:40.732820 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:40.732779    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 07 16:56:40.733291 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:40.733028    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:56:50.733140 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:50.733098    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 07 16:56:50.733623 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:56:50.733515    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:57:00.733209 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:57:00.733165    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 07 16:57:00.733704 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:57:00.733677    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:57:10.733654 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:57:10.733603    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 07 16:57:10.734145 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:57:10.733954    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:57:20.733024 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:57:20.732981    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 07 16:57:20.733560 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:57:20.733538    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:57:30.733353 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:57:30.733301    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 07 16:57:30.733850 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:57:30.733747    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:57:40.732833 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:57:40.732779    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 07 16:57:40.733386 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:57:40.733290    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:57:50.733309 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:57:50.733230    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f"
Jul 07 16:57:50.733823 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:57:50.733558    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f"
Jul 07 16:57:59.242088 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:57:59.242037    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:57:59.290313 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:57:59.290280    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f"]
Jul 07 16:57:59.290576 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:57:59.290544    2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="kserve-container" containerID="cri-o://da18d2efd831c10583cdd9bb280a89f26d67f40fb7818aab5cd2b169d736cb2f" gracePeriod=30
Jul 07 16:57:59.290720 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:57:59.290570    2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="agent" containerID="cri-o://578b7dcc69d927f717a235b2855f06b45ae2b0eccc474a0f3300d4d66fa8a6de" gracePeriod=30
Jul 07 16:57:59.290720 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:57:59.290598    2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="kube-rbac-proxy" containerID="cri-o://264b39c44aa9cfe7f3cbb64c88e58be22919ab04130b2508022b6c0f4fc9be64" gracePeriod=30
Jul 07 16:57:59.313287 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:57:59.313256    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp"]
Jul 07 16:57:59.316225 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:57:59.316207    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp"
Jul 07 16:57:59.330957 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:57:59.330921    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"isvc-sklearn-batcher-custom-predictor-serving-cert\" is forbidden: User \"system:node:ip-10-0-133-184.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-133-184.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\"" type="*v1.Secret"
Jul 07 16:57:59.331361 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:57:59.331339    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" is forbidden: User \"system:node:ip-10-0-133-184.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-133-184.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\"" type="*v1.ConfigMap"
Jul 07 16:57:59.334819 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:57:59.334791    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp"]
Jul 07 16:57:59.472145 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:57:59.472109    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3c5c722-5800-4b51-a234-f31c61786bba-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp\" (UID: \"d3c5c722-5800-4b51-a234-f31c61786bba\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp"
Jul 07 16:57:59.472326 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:57:59.472159    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3c5c722-5800-4b51-a234-f31c61786bba-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp\" (UID: \"d3c5c722-5800-4b51-a234-f31c61786bba\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp"
Jul 07 16:57:59.472326 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:57:59.472195    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3c5c722-5800-4b51-a234-f31c61786bba-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp\" (UID: \"d3c5c722-5800-4b51-a234-f31c61786bba\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp"
Jul 07 16:57:59.472326 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:57:59.472271    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj6nk\" (UniqueName: \"kubernetes.io/projected/d3c5c722-5800-4b51-a234-f31c61786bba-kube-api-access-zj6nk\") pod \"isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp\" (UID: \"d3c5c722-5800-4b51-a234-f31c61786bba\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp"
Jul 07 16:57:59.472326 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:57:59.472304    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3c5c722-5800-4b51-a234-f31c61786bba-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp\" (UID: \"d3c5c722-5800-4b51-a234-f31c61786bba\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp"
Jul 07 16:57:59.573067 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:57:59.572946    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3c5c722-5800-4b51-a234-f31c61786bba-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp\" (UID: \"d3c5c722-5800-4b51-a234-f31c61786bba\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp"
Jul 07 16:57:59.573067 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:57:59.573031    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj6nk\" (UniqueName: \"kubernetes.io/projected/d3c5c722-5800-4b51-a234-f31c61786bba-kube-api-access-zj6nk\") pod \"isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp\" (UID: \"d3c5c722-5800-4b51-a234-f31c61786bba\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp"
Jul 07 16:57:59.573292 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:57:59.573090    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3c5c722-5800-4b51-a234-f31c61786bba-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp\" (UID: \"d3c5c722-5800-4b51-a234-f31c61786bba\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp"
Jul 07 16:57:59.573292 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:57:59.573188    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3c5c722-5800-4b51-a234-f31c61786bba-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp\" (UID: \"d3c5c722-5800-4b51-a234-f31c61786bba\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp"
Jul 07 16:57:59.573292 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:57:59.573226    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3c5c722-5800-4b51-a234-f31c61786bba-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp\" (UID: \"d3c5c722-5800-4b51-a234-f31c61786bba\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp"
Jul 07 16:57:59.573483 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:57:59.573462    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3c5c722-5800-4b51-a234-f31c61786bba-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp\" (UID: \"d3c5c722-5800-4b51-a234-f31c61786bba\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp"
Jul 07 16:57:59.573761 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:57:59.573744    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3c5c722-5800-4b51-a234-f31c61786bba-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp\" (UID: \"d3c5c722-5800-4b51-a234-f31c61786bba\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp"
Jul 07 16:57:59.584476 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:57:59.584453    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj6nk\" (UniqueName: \"kubernetes.io/projected/d3c5c722-5800-4b51-a234-f31c61786bba-kube-api-access-zj6nk\") pod \"isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp\" (UID: \"d3c5c722-5800-4b51-a234-f31c61786bba\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp"
Jul 07 16:57:59.729303 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:57:59.729267    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Jul 07 16:58:00.089189 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:00.089154    2584 generic.go:356] "Generic (PLEG): container finished" podID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerID="264b39c44aa9cfe7f3cbb64c88e58be22919ab04130b2508022b6c0f4fc9be64" exitCode=2
Jul 07 16:58:00.089378 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:00.089234    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" event={"ID":"48f6db63-3224-48ce-9cef-e140d5600ee7","Type":"ContainerDied","Data":"264b39c44aa9cfe7f3cbb64c88e58be22919ab04130b2508022b6c0f4fc9be64"}
Jul 07 16:58:00.176836 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:00.176805    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 07 16:58:00.184605 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:00.184577    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3c5c722-5800-4b51-a234-f31c61786bba-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp\" (UID: \"d3c5c722-5800-4b51-a234-f31c61786bba\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp"
Jul 07 16:58:00.392780 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:00.392737    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:58:00.573287 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:58:00.573257    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-serving-cert: failed to sync secret cache: timed out waiting for the condition
Jul 07 16:58:00.573487 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:58:00.573343    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3c5c722-5800-4b51-a234-f31c61786bba-proxy-tls podName:d3c5c722-5800-4b51-a234-f31c61786bba nodeName:}" failed. No retries permitted until 2026-07-07 16:58:01.073321839 +0000 UTC m=+729.674530536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d3c5c722-5800-4b51-a234-f31c61786bba-proxy-tls") pod "isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" (UID: "d3c5c722-5800-4b51-a234-f31c61786bba") : failed to sync secret cache: timed out waiting for the condition
Jul 07 16:58:00.597785 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:00.597752    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 07 16:58:00.733446 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:00.733341    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 07 16:58:00.733696 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:00.733668    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:58:01.087462 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:01.087361    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3c5c722-5800-4b51-a234-f31c61786bba-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp\" (UID: \"d3c5c722-5800-4b51-a234-f31c61786bba\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp"
Jul 07 16:58:01.090107 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:01.090076    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3c5c722-5800-4b51-a234-f31c61786bba-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp\" (UID: \"d3c5c722-5800-4b51-a234-f31c61786bba\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp"
Jul 07 16:58:01.126084 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:01.126029    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp"
Jul 07 16:58:01.279966 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:01.279937    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp"]
Jul 07 16:58:01.282595 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:58:01.282566    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c5c722_5800_4b51_a234_f31c61786bba.slice/crio-55a7bb9a117df1514b30755ca079e4e654af0fb563f4f36f2265cded82e9dbd0 WatchSource:0}: Error finding container 55a7bb9a117df1514b30755ca079e4e654af0fb563f4f36f2265cded82e9dbd0: Status 404 returned error can't find the container with id 55a7bb9a117df1514b30755ca079e4e654af0fb563f4f36f2265cded82e9dbd0
Jul 07 16:58:02.100847 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:02.100809    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" event={"ID":"d3c5c722-5800-4b51-a234-f31c61786bba","Type":"ContainerStarted","Data":"963c5d89ae46a7152ce7aadb30659fe7275330c7289a57ad6136b0352044e9dc"}
Jul 07 16:58:02.101272 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:02.100878    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" event={"ID":"d3c5c722-5800-4b51-a234-f31c61786bba","Type":"ContainerStarted","Data":"55a7bb9a117df1514b30755ca079e4e654af0fb563f4f36f2265cded82e9dbd0"}
Jul 07 16:58:04.115449 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:04.115417    2584 generic.go:356] "Generic (PLEG): container finished" podID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerID="da18d2efd831c10583cdd9bb280a89f26d67f40fb7818aab5cd2b169d736cb2f" exitCode=0
Jul 07 16:58:04.115779 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:04.115463    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" event={"ID":"48f6db63-3224-48ce-9cef-e140d5600ee7","Type":"ContainerDied","Data":"da18d2efd831c10583cdd9bb280a89f26d67f40fb7818aab5cd2b169d736cb2f"}
Jul 07 16:58:04.728597 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:04.728549    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Jul 07 16:58:05.121795 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:05.121761    2584 generic.go:356] "Generic (PLEG): container finished" podID="d3c5c722-5800-4b51-a234-f31c61786bba" containerID="963c5d89ae46a7152ce7aadb30659fe7275330c7289a57ad6136b0352044e9dc" exitCode=0
Jul 07 16:58:05.122217 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:05.121837    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" event={"ID":"d3c5c722-5800-4b51-a234-f31c61786bba","Type":"ContainerDied","Data":"963c5d89ae46a7152ce7aadb30659fe7275330c7289a57ad6136b0352044e9dc"}
Jul 07 16:58:06.129198 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:06.129160    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" event={"ID":"d3c5c722-5800-4b51-a234-f31c61786bba","Type":"ContainerStarted","Data":"7c81f0698872767025ec46af63f66a591ef7623a23a7ffbb58846de6c28dc274"}
Jul 07 16:58:06.129573 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:06.129224    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" event={"ID":"d3c5c722-5800-4b51-a234-f31c61786bba","Type":"ContainerStarted","Data":"e637f2264f43a5fe628a64e282a2b5382617c5ccb21844a2c500eac89764140b"}
Jul 07 16:58:06.129573 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:06.129250    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" event={"ID":"d3c5c722-5800-4b51-a234-f31c61786bba","Type":"ContainerStarted","Data":"643c70dc6e948e9d34f93bf2132bb3e64078e63677e7346de55d90032243574c"}
Jul 07 16:58:06.129573 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:06.129551    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp"
Jul 07 16:58:06.129726 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:06.129708    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp"
Jul 07 16:58:06.130983 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:06.130955    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused"
Jul 07 16:58:06.160226 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:06.160159    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podStartSLOduration=3.6288396880000002 podStartE2EDuration="7.160138303s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.531298615s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:58:06.15882697 +0000 UTC m=+734.760035674" watchObservedRunningTime="2026-07-07 16:58:06.160138303 +0000 UTC m=+734.761346989"
Jul 07 16:58:07.134611 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:07.134581    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp"
Jul 07 16:58:07.135120 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:07.134713    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused"
Jul 07 16:58:07.135738 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:07.135710    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:58:08.139390 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:08.139341    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused"
Jul 07 16:58:08.139849 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:08.139791    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:58:09.728769 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:09.728733    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Jul 07 16:58:09.729205 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:09.728873    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f"
Jul 07 16:58:10.733409 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:10.733357    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 07 16:58:10.733795 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:10.733754    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:58:13.144533 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:13.144505    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp"
Jul 07 16:58:13.145213 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:13.145176    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused"
Jul 07 16:58:13.145359 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:13.145334    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:58:14.729255 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:14.729216    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Jul 07 16:58:19.728471 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:19.728425    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Jul 07 16:58:20.733685 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:20.733638    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 07 16:58:20.734154 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:20.733799    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f"
Jul 07 16:58:20.734154 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:20.733989    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:58:20.734154 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:20.734136    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f"
Jul 07 16:58:23.145847 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:23.145799    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused"
Jul 07 16:58:23.146280 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:23.146257    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:58:24.729106 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:24.729031    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Jul 07 16:58:29.446756 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:29.446730    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f"
Jul 07 16:58:29.530475 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:29.530441    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8g2rr\" (UniqueName: \"kubernetes.io/projected/48f6db63-3224-48ce-9cef-e140d5600ee7-kube-api-access-8g2rr\") pod \"48f6db63-3224-48ce-9cef-e140d5600ee7\" (UID: \"48f6db63-3224-48ce-9cef-e140d5600ee7\") "
Jul 07 16:58:29.530678 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:29.530491    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48f6db63-3224-48ce-9cef-e140d5600ee7-cabundle-cert\") pod \"48f6db63-3224-48ce-9cef-e140d5600ee7\" (UID: \"48f6db63-3224-48ce-9cef-e140d5600ee7\") "
Jul 07 16:58:29.530678 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:29.530574    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48f6db63-3224-48ce-9cef-e140d5600ee7-proxy-tls\") pod \"48f6db63-3224-48ce-9cef-e140d5600ee7\" (UID: \"48f6db63-3224-48ce-9cef-e140d5600ee7\") "
Jul 07 16:58:29.530678 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:29.530606    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48f6db63-3224-48ce-9cef-e140d5600ee7-kube-rbac-proxy-sar-config\") pod \"48f6db63-3224-48ce-9cef-e140d5600ee7\" (UID: \"48f6db63-3224-48ce-9cef-e140d5600ee7\") "
Jul 07 16:58:29.530678 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:29.530652    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48f6db63-3224-48ce-9cef-e140d5600ee7-kserve-provision-location\") pod \"48f6db63-3224-48ce-9cef-e140d5600ee7\" (UID: \"48f6db63-3224-48ce-9cef-e140d5600ee7\") "
Jul 07 16:58:29.530998 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:29.530960    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48f6db63-3224-48ce-9cef-e140d5600ee7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "48f6db63-3224-48ce-9cef-e140d5600ee7" (UID: "48f6db63-3224-48ce-9cef-e140d5600ee7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:58:29.531131 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:29.531001    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48f6db63-3224-48ce-9cef-e140d5600ee7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "48f6db63-3224-48ce-9cef-e140d5600ee7" (UID: "48f6db63-3224-48ce-9cef-e140d5600ee7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 16:58:29.531131 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:29.531082    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f6db63-3224-48ce-9cef-e140d5600ee7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "48f6db63-3224-48ce-9cef-e140d5600ee7" (UID: "48f6db63-3224-48ce-9cef-e140d5600ee7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 16:58:29.532974 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:29.532943    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48f6db63-3224-48ce-9cef-e140d5600ee7-kube-api-access-8g2rr" (OuterVolumeSpecName: "kube-api-access-8g2rr") pod "48f6db63-3224-48ce-9cef-e140d5600ee7" (UID: "48f6db63-3224-48ce-9cef-e140d5600ee7"). InnerVolumeSpecName "kube-api-access-8g2rr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 16:58:29.532974 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:29.532966    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48f6db63-3224-48ce-9cef-e140d5600ee7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "48f6db63-3224-48ce-9cef-e140d5600ee7" (UID: "48f6db63-3224-48ce-9cef-e140d5600ee7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 16:58:29.631367 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:29.631327    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g2rr\" (UniqueName: \"kubernetes.io/projected/48f6db63-3224-48ce-9cef-e140d5600ee7-kube-api-access-8g2rr\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 16:58:29.631367 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:29.631357    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48f6db63-3224-48ce-9cef-e140d5600ee7-cabundle-cert\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 16:58:29.631367 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:29.631367    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48f6db63-3224-48ce-9cef-e140d5600ee7-proxy-tls\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 16:58:29.631367 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:29.631374    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48f6db63-3224-48ce-9cef-e140d5600ee7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 16:58:29.631643 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:29.631383    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48f6db63-3224-48ce-9cef-e140d5600ee7-kserve-provision-location\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 16:58:30.248765 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:30.248670    2584 generic.go:356] "Generic (PLEG): container finished" podID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerID="578b7dcc69d927f717a235b2855f06b45ae2b0eccc474a0f3300d4d66fa8a6de" exitCode=0
Jul 07 16:58:30.248765 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:30.248740    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f"
Jul 07 16:58:30.248765 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:30.248747    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" event={"ID":"48f6db63-3224-48ce-9cef-e140d5600ee7","Type":"ContainerDied","Data":"578b7dcc69d927f717a235b2855f06b45ae2b0eccc474a0f3300d4d66fa8a6de"}
Jul 07 16:58:30.249075 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:30.248833    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f" event={"ID":"48f6db63-3224-48ce-9cef-e140d5600ee7","Type":"ContainerDied","Data":"8bb211b129225e838e894a92a3f4e8dc634fa5a75176de749c6b50fab85995fc"}
Jul 07 16:58:30.249075 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:30.248876    2584 scope.go:117] "RemoveContainer" containerID="578b7dcc69d927f717a235b2855f06b45ae2b0eccc474a0f3300d4d66fa8a6de"
Jul 07 16:58:30.257353 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:30.257330    2584 scope.go:117] "RemoveContainer" containerID="264b39c44aa9cfe7f3cbb64c88e58be22919ab04130b2508022b6c0f4fc9be64"
Jul 07 16:58:30.265404 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:30.265381    2584 scope.go:117] "RemoveContainer" containerID="da18d2efd831c10583cdd9bb280a89f26d67f40fb7818aab5cd2b169d736cb2f"
Jul 07 16:58:30.273236 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:30.273214    2584 scope.go:117] "RemoveContainer" containerID="578ee47d51c025c7c24ff9efa6475cb0852ad99ca6e2c1d8cbf84164c6cf21d7"
Jul 07 16:58:30.280602 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:30.280573    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f"]
Jul 07 16:58:30.281695 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:30.281677    2584 scope.go:117] "RemoveContainer" containerID="578b7dcc69d927f717a235b2855f06b45ae2b0eccc474a0f3300d4d66fa8a6de"
Jul 07 16:58:30.281991 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:58:30.281970    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"578b7dcc69d927f717a235b2855f06b45ae2b0eccc474a0f3300d4d66fa8a6de\": container with ID starting with 578b7dcc69d927f717a235b2855f06b45ae2b0eccc474a0f3300d4d66fa8a6de not found: ID does not exist" containerID="578b7dcc69d927f717a235b2855f06b45ae2b0eccc474a0f3300d4d66fa8a6de"
Jul 07 16:58:30.282036 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:30.282003    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"578b7dcc69d927f717a235b2855f06b45ae2b0eccc474a0f3300d4d66fa8a6de"} err="failed to get container status \"578b7dcc69d927f717a235b2855f06b45ae2b0eccc474a0f3300d4d66fa8a6de\": rpc error: code = NotFound desc = could not find container \"578b7dcc69d927f717a235b2855f06b45ae2b0eccc474a0f3300d4d66fa8a6de\": container with ID starting with 578b7dcc69d927f717a235b2855f06b45ae2b0eccc474a0f3300d4d66fa8a6de not found: ID does not exist"
Jul 07 16:58:30.282036 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:30.282023    2584 scope.go:117] "RemoveContainer" containerID="264b39c44aa9cfe7f3cbb64c88e58be22919ab04130b2508022b6c0f4fc9be64"
Jul 07 16:58:30.282306 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:58:30.282290    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"264b39c44aa9cfe7f3cbb64c88e58be22919ab04130b2508022b6c0f4fc9be64\": container with ID starting with 264b39c44aa9cfe7f3cbb64c88e58be22919ab04130b2508022b6c0f4fc9be64 not found: ID does not exist" containerID="264b39c44aa9cfe7f3cbb64c88e58be22919ab04130b2508022b6c0f4fc9be64"
Jul 07 16:58:30.282360 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:30.282311    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"264b39c44aa9cfe7f3cbb64c88e58be22919ab04130b2508022b6c0f4fc9be64"} err="failed to get container status \"264b39c44aa9cfe7f3cbb64c88e58be22919ab04130b2508022b6c0f4fc9be64\": rpc error: code = NotFound desc = could not find container \"264b39c44aa9cfe7f3cbb64c88e58be22919ab04130b2508022b6c0f4fc9be64\": container with ID starting with 264b39c44aa9cfe7f3cbb64c88e58be22919ab04130b2508022b6c0f4fc9be64 not found: ID does not exist"
Jul 07 16:58:30.282360 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:30.282325    2584 scope.go:117] "RemoveContainer" containerID="da18d2efd831c10583cdd9bb280a89f26d67f40fb7818aab5cd2b169d736cb2f"
Jul 07 16:58:30.282563 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:58:30.282548    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da18d2efd831c10583cdd9bb280a89f26d67f40fb7818aab5cd2b169d736cb2f\": container with ID starting with da18d2efd831c10583cdd9bb280a89f26d67f40fb7818aab5cd2b169d736cb2f not found: ID does not exist" containerID="da18d2efd831c10583cdd9bb280a89f26d67f40fb7818aab5cd2b169d736cb2f"
Jul 07 16:58:30.282602 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:30.282567    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da18d2efd831c10583cdd9bb280a89f26d67f40fb7818aab5cd2b169d736cb2f"} err="failed to get container status \"da18d2efd831c10583cdd9bb280a89f26d67f40fb7818aab5cd2b169d736cb2f\": rpc error: code = NotFound desc = could not find container \"da18d2efd831c10583cdd9bb280a89f26d67f40fb7818aab5cd2b169d736cb2f\": container with ID starting with da18d2efd831c10583cdd9bb280a89f26d67f40fb7818aab5cd2b169d736cb2f not found: ID does not exist"
Jul 07 16:58:30.282602 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:30.282580    2584 scope.go:117] "RemoveContainer" containerID="578ee47d51c025c7c24ff9efa6475cb0852ad99ca6e2c1d8cbf84164c6cf21d7"
Jul 07 16:58:30.282835 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:58:30.282817    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"578ee47d51c025c7c24ff9efa6475cb0852ad99ca6e2c1d8cbf84164c6cf21d7\": container with ID starting with 578ee47d51c025c7c24ff9efa6475cb0852ad99ca6e2c1d8cbf84164c6cf21d7 not found: ID does not exist" containerID="578ee47d51c025c7c24ff9efa6475cb0852ad99ca6e2c1d8cbf84164c6cf21d7"
Jul 07 16:58:30.282886 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:30.282844    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"578ee47d51c025c7c24ff9efa6475cb0852ad99ca6e2c1d8cbf84164c6cf21d7"} err="failed to get container status \"578ee47d51c025c7c24ff9efa6475cb0852ad99ca6e2c1d8cbf84164c6cf21d7\": rpc error: code = NotFound desc = could not find container \"578ee47d51c025c7c24ff9efa6475cb0852ad99ca6e2c1d8cbf84164c6cf21d7\": container with ID starting with 578ee47d51c025c7c24ff9efa6475cb0852ad99ca6e2c1d8cbf84164c6cf21d7 not found: ID does not exist"
Jul 07 16:58:30.286725 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:30.286701    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-75c9fd9d88-v8g7f"]
Jul 07 16:58:31.894206 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:31.894166    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" path="/var/lib/kubelet/pods/48f6db63-3224-48ce-9cef-e140d5600ee7/volumes"
Jul 07 16:58:33.145757 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:33.145709    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused"
Jul 07 16:58:33.146188 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:33.146156    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:58:43.145747 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:43.145700    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused"
Jul 07 16:58:43.146329 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:43.146216    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:58:53.145890 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:53.145848    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused"
Jul 07 16:58:53.146297 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:58:53.146237    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:59:03.145615 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:03.145569    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused"
Jul 07 16:59:03.146026 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:03.146000    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:59:13.145833 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:13.145739    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused"
Jul 07 16:59:13.146242 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:13.146153    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:59:14.888666 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:14.888618    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused"
Jul 07 16:59:14.889282 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:14.889107    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:59:24.889203 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:24.889168    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp"
Jul 07 16:59:24.889675 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:24.889258    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp"
Jul 07 16:59:34.538500 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:34.538467    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp"]
Jul 07 16:59:34.538924 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:34.538739    2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="kserve-container" containerID="cri-o://643c70dc6e948e9d34f93bf2132bb3e64078e63677e7346de55d90032243574c" gracePeriod=30
Jul 07 16:59:34.538924 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:34.538768    2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="agent" containerID="cri-o://7c81f0698872767025ec46af63f66a591ef7623a23a7ffbb58846de6c28dc274" gracePeriod=30
Jul 07 16:59:34.538924 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:34.538839    2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="kube-rbac-proxy" containerID="cri-o://e637f2264f43a5fe628a64e282a2b5382617c5ccb21844a2c500eac89764140b" gracePeriod=30
Jul 07 16:59:34.581760 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:34.581702    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:59:34.888662 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:34.888621    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused"
Jul 07 16:59:34.888881 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:34.888858    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:59:34.959764 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:34.959725    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:59:35.550616 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:35.550582    2584 generic.go:356] "Generic (PLEG): container finished" podID="d3c5c722-5800-4b51-a234-f31c61786bba" containerID="e637f2264f43a5fe628a64e282a2b5382617c5ccb21844a2c500eac89764140b" exitCode=2
Jul 07 16:59:35.551019 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:35.550658    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" event={"ID":"d3c5c722-5800-4b51-a234-f31c61786bba","Type":"ContainerDied","Data":"e637f2264f43a5fe628a64e282a2b5382617c5ccb21844a2c500eac89764140b"}
Jul 07 16:59:38.139832 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:38.139786    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Jul 07 16:59:39.569959 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:39.569877    2584 generic.go:356] "Generic (PLEG): container finished" podID="d3c5c722-5800-4b51-a234-f31c61786bba" containerID="643c70dc6e948e9d34f93bf2132bb3e64078e63677e7346de55d90032243574c" exitCode=0
Jul 07 16:59:39.569959 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:39.569923    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" event={"ID":"d3c5c722-5800-4b51-a234-f31c61786bba","Type":"ContainerDied","Data":"643c70dc6e948e9d34f93bf2132bb3e64078e63677e7346de55d90032243574c"}
Jul 07 16:59:43.140041 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:43.139995    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Jul 07 16:59:44.889232 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:44.889186    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused"
Jul 07 16:59:44.889627 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:44.889556    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:59:48.139691 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:48.139647    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Jul 07 16:59:48.140246 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:48.139804    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp"
Jul 07 16:59:53.140040 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:53.139995    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Jul 07 16:59:54.559507 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.559467    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:59:54.617928 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.617891    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z"]
Jul 07 16:59:54.618428 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.618406    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="kserve-container"
Jul 07 16:59:54.618428 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.618424    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="kserve-container"
Jul 07 16:59:54.618428 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.618435    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="agent"
Jul 07 16:59:54.618620 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.618441    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="agent"
Jul 07 16:59:54.618620 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.618461    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="storage-initializer"
Jul 07 16:59:54.618620 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.618467    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="storage-initializer"
Jul 07 16:59:54.618620 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.618478    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="kube-rbac-proxy"
Jul 07 16:59:54.618620 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.618484    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="kube-rbac-proxy"
Jul 07 16:59:54.618620 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.618534    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="agent"
Jul 07 16:59:54.618620 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.618544    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="kube-rbac-proxy"
Jul 07 16:59:54.618620 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.618550    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="48f6db63-3224-48ce-9cef-e140d5600ee7" containerName="kserve-container"
Jul 07 16:59:54.620741 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.620724    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z"
Jul 07 16:59:54.623649 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.623626    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 07 16:59:54.624120 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.624103    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 07 16:59:54.637673 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.637634    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z"]
Jul 07 16:59:54.756470 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.756429    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj4fn\" (UniqueName: \"kubernetes.io/projected/eb319868-44d2-46e9-b425-fc0134930aa9-kube-api-access-qj4fn\") pod \"isvc-logger-predictor-7979f68479-zqv5z\" (UID: \"eb319868-44d2-46e9-b425-fc0134930aa9\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z"
Jul 07 16:59:54.756635 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.756486    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb319868-44d2-46e9-b425-fc0134930aa9-proxy-tls\") pod \"isvc-logger-predictor-7979f68479-zqv5z\" (UID: \"eb319868-44d2-46e9-b425-fc0134930aa9\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z"
Jul 07 16:59:54.756635 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.756519    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb319868-44d2-46e9-b425-fc0134930aa9-cabundle-cert\") pod \"isvc-logger-predictor-7979f68479-zqv5z\" (UID: \"eb319868-44d2-46e9-b425-fc0134930aa9\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z"
Jul 07 16:59:54.756635 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.756557    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb319868-44d2-46e9-b425-fc0134930aa9-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-7979f68479-zqv5z\" (UID: \"eb319868-44d2-46e9-b425-fc0134930aa9\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z"
Jul 07 16:59:54.756635 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.756621    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb319868-44d2-46e9-b425-fc0134930aa9-kserve-provision-location\") pod \"isvc-logger-predictor-7979f68479-zqv5z\" (UID: \"eb319868-44d2-46e9-b425-fc0134930aa9\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z"
Jul 07 16:59:54.857709 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.857674    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj4fn\" (UniqueName: \"kubernetes.io/projected/eb319868-44d2-46e9-b425-fc0134930aa9-kube-api-access-qj4fn\") pod \"isvc-logger-predictor-7979f68479-zqv5z\" (UID: \"eb319868-44d2-46e9-b425-fc0134930aa9\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z"
Jul 07 16:59:54.857877 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.857726    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb319868-44d2-46e9-b425-fc0134930aa9-proxy-tls\") pod \"isvc-logger-predictor-7979f68479-zqv5z\" (UID: \"eb319868-44d2-46e9-b425-fc0134930aa9\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z"
Jul 07 16:59:54.857877 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.857747    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb319868-44d2-46e9-b425-fc0134930aa9-cabundle-cert\") pod \"isvc-logger-predictor-7979f68479-zqv5z\" (UID: \"eb319868-44d2-46e9-b425-fc0134930aa9\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z"
Jul 07 16:59:54.857877 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.857779    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb319868-44d2-46e9-b425-fc0134930aa9-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-7979f68479-zqv5z\" (UID: \"eb319868-44d2-46e9-b425-fc0134930aa9\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z"
Jul 07 16:59:54.857877 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.857811    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb319868-44d2-46e9-b425-fc0134930aa9-kserve-provision-location\") pod \"isvc-logger-predictor-7979f68479-zqv5z\" (UID: \"eb319868-44d2-46e9-b425-fc0134930aa9\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z"
Jul 07 16:59:54.857877 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:59:54.857851    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Jul 07 16:59:54.858160 ip-10-0-133-184 kubenswrapper[2584]: E0707 16:59:54.857928    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb319868-44d2-46e9-b425-fc0134930aa9-proxy-tls podName:eb319868-44d2-46e9-b425-fc0134930aa9 nodeName:}" failed. No retries permitted until 2026-07-07 16:59:55.357907469 +0000 UTC m=+843.959116151 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/eb319868-44d2-46e9-b425-fc0134930aa9-proxy-tls") pod "isvc-logger-predictor-7979f68479-zqv5z" (UID: "eb319868-44d2-46e9-b425-fc0134930aa9") : secret "isvc-logger-predictor-serving-cert" not found
Jul 07 16:59:54.858321 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.858299    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb319868-44d2-46e9-b425-fc0134930aa9-kserve-provision-location\") pod \"isvc-logger-predictor-7979f68479-zqv5z\" (UID: \"eb319868-44d2-46e9-b425-fc0134930aa9\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z"
Jul 07 16:59:54.858448 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.858432    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb319868-44d2-46e9-b425-fc0134930aa9-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-7979f68479-zqv5z\" (UID: \"eb319868-44d2-46e9-b425-fc0134930aa9\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z"
Jul 07 16:59:54.858497 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.858479    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb319868-44d2-46e9-b425-fc0134930aa9-cabundle-cert\") pod \"isvc-logger-predictor-7979f68479-zqv5z\" (UID: \"eb319868-44d2-46e9-b425-fc0134930aa9\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z"
Jul 07 16:59:54.872215 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.872184    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj4fn\" (UniqueName: \"kubernetes.io/projected/eb319868-44d2-46e9-b425-fc0134930aa9-kube-api-access-qj4fn\") pod \"isvc-logger-predictor-7979f68479-zqv5z\" (UID: \"eb319868-44d2-46e9-b425-fc0134930aa9\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z"
Jul 07 16:59:54.888833 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.888790    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:5000: connect: connection refused"
Jul 07 16:59:54.888978 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.888931    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp"
Jul 07 16:59:54.889167 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.889150    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 16:59:54.889245 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.889219    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp"
Jul 07 16:59:54.955972 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:54.955934    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 16:59:55.362826 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:55.362792    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb319868-44d2-46e9-b425-fc0134930aa9-proxy-tls\") pod \"isvc-logger-predictor-7979f68479-zqv5z\" (UID: \"eb319868-44d2-46e9-b425-fc0134930aa9\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z"
Jul 07 16:59:55.365462 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:55.365428    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb319868-44d2-46e9-b425-fc0134930aa9-proxy-tls\") pod \"isvc-logger-predictor-7979f68479-zqv5z\" (UID: \"eb319868-44d2-46e9-b425-fc0134930aa9\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z"
Jul 07 16:59:55.531941 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:55.531910    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z"
Jul 07 16:59:55.676421 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:55.676397    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z"]
Jul 07 16:59:55.678899 ip-10-0-133-184 kubenswrapper[2584]: W0707 16:59:55.678865    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb319868_44d2_46e9_b425_fc0134930aa9.slice/crio-049c81f0a2f2cee5e9a496bf6de22aae8a748a63dc8d008b768119c5f59c6f3b WatchSource:0}: Error finding container 049c81f0a2f2cee5e9a496bf6de22aae8a748a63dc8d008b768119c5f59c6f3b: Status 404 returned error can't find the container with id 049c81f0a2f2cee5e9a496bf6de22aae8a748a63dc8d008b768119c5f59c6f3b
Jul 07 16:59:56.648716 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:56.648678    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" event={"ID":"eb319868-44d2-46e9-b425-fc0134930aa9","Type":"ContainerStarted","Data":"94b0287db2fec149a2778bc398d4f297b5835c15cd2e69bfaf4e36b6bd1d85c8"}
Jul 07 16:59:56.648908 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:56.648738    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" event={"ID":"eb319868-44d2-46e9-b425-fc0134930aa9","Type":"ContainerStarted","Data":"049c81f0a2f2cee5e9a496bf6de22aae8a748a63dc8d008b768119c5f59c6f3b"}
Jul 07 16:59:58.139578 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:58.139536    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Jul 07 16:59:59.664269 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:59.664181    2584 generic.go:356] "Generic (PLEG): container finished" podID="eb319868-44d2-46e9-b425-fc0134930aa9" containerID="94b0287db2fec149a2778bc398d4f297b5835c15cd2e69bfaf4e36b6bd1d85c8" exitCode=0
Jul 07 16:59:59.664269 ip-10-0-133-184 kubenswrapper[2584]: I0707 16:59:59.664235    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" event={"ID":"eb319868-44d2-46e9-b425-fc0134930aa9","Type":"ContainerDied","Data":"94b0287db2fec149a2778bc398d4f297b5835c15cd2e69bfaf4e36b6bd1d85c8"}
Jul 07 17:00:00.671726 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:00.671691    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" event={"ID":"eb319868-44d2-46e9-b425-fc0134930aa9","Type":"ContainerStarted","Data":"8e6c65b4bfcec5eb6adbd9919162a3159a40b8cee6f52456c149565cfb9bd4e3"}
Jul 07 17:00:00.672183 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:00.671764    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" event={"ID":"eb319868-44d2-46e9-b425-fc0134930aa9","Type":"ContainerStarted","Data":"46f687ef562569dd1fd272bb4695169cdf272b6f490cd606c4057bed9aeecb6d"}
Jul 07 17:00:00.672183 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:00.671791    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" event={"ID":"eb319868-44d2-46e9-b425-fc0134930aa9","Type":"ContainerStarted","Data":"bd9fb6f4a99ae53c7221a976efe8774ed5eb238dec7ee76e7d186110333dbc6c"}
Jul 07 17:00:00.672183 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:00.672097    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z"
Jul 07 17:00:00.672371 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:00.672244    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z"
Jul 07 17:00:00.673399 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:00.673371    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 07 17:00:00.705715 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:00.705664    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" podStartSLOduration=3.046912488 podStartE2EDuration="6.705645938s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.65873345s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:00:00.702426342 +0000 UTC m=+849.303635046" watchObservedRunningTime="2026-07-07 17:00:00.705645938 +0000 UTC m=+849.306854709"
Jul 07 17:00:01.676587 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:01.676556    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z"
Jul 07 17:00:01.677014 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:01.676725    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 07 17:00:01.677767 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:01.677743    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:00:02.681135 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:02.681093    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 07 17:00:02.681639 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:02.681616    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:00:03.139559 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:03.139513    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Jul 07 17:00:04.691623 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:04.691598    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp"
Jul 07 17:00:04.692416 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:04.692392    2584 generic.go:356] "Generic (PLEG): container finished" podID="d3c5c722-5800-4b51-a234-f31c61786bba" containerID="7c81f0698872767025ec46af63f66a591ef7623a23a7ffbb58846de6c28dc274" exitCode=0
Jul 07 17:00:04.692504 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:04.692460    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" event={"ID":"d3c5c722-5800-4b51-a234-f31c61786bba","Type":"ContainerDied","Data":"7c81f0698872767025ec46af63f66a591ef7623a23a7ffbb58846de6c28dc274"}
Jul 07 17:00:04.692570 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:04.692559    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp" event={"ID":"d3c5c722-5800-4b51-a234-f31c61786bba","Type":"ContainerDied","Data":"55a7bb9a117df1514b30755ca079e4e654af0fb563f4f36f2265cded82e9dbd0"}
Jul 07 17:00:04.692607 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:04.692593    2584 scope.go:117] "RemoveContainer" containerID="7c81f0698872767025ec46af63f66a591ef7623a23a7ffbb58846de6c28dc274"
Jul 07 17:00:04.701038 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:04.701020    2584 scope.go:117] "RemoveContainer" containerID="e637f2264f43a5fe628a64e282a2b5382617c5ccb21844a2c500eac89764140b"
Jul 07 17:00:04.710226 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:04.710205    2584 scope.go:117] "RemoveContainer" containerID="643c70dc6e948e9d34f93bf2132bb3e64078e63677e7346de55d90032243574c"
Jul 07 17:00:04.717902 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:04.717882    2584 scope.go:117] "RemoveContainer" containerID="963c5d89ae46a7152ce7aadb30659fe7275330c7289a57ad6136b0352044e9dc"
Jul 07 17:00:04.728859 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:04.728833    2584 scope.go:117] "RemoveContainer" containerID="7c81f0698872767025ec46af63f66a591ef7623a23a7ffbb58846de6c28dc274"
Jul 07 17:00:04.729229 ip-10-0-133-184 kubenswrapper[2584]: E0707 17:00:04.729204    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c81f0698872767025ec46af63f66a591ef7623a23a7ffbb58846de6c28dc274\": container with ID starting with 7c81f0698872767025ec46af63f66a591ef7623a23a7ffbb58846de6c28dc274 not found: ID does not exist" containerID="7c81f0698872767025ec46af63f66a591ef7623a23a7ffbb58846de6c28dc274"
Jul 07 17:00:04.729303 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:04.729238    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c81f0698872767025ec46af63f66a591ef7623a23a7ffbb58846de6c28dc274"} err="failed to get container status \"7c81f0698872767025ec46af63f66a591ef7623a23a7ffbb58846de6c28dc274\": rpc error: code = NotFound desc = could not find container \"7c81f0698872767025ec46af63f66a591ef7623a23a7ffbb58846de6c28dc274\": container with ID starting with 7c81f0698872767025ec46af63f66a591ef7623a23a7ffbb58846de6c28dc274 not found: ID does not exist"
Jul 07 17:00:04.729303 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:04.729258    2584 scope.go:117] "RemoveContainer" containerID="e637f2264f43a5fe628a64e282a2b5382617c5ccb21844a2c500eac89764140b"
Jul 07 17:00:04.729532 ip-10-0-133-184 kubenswrapper[2584]: E0707 17:00:04.729497    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e637f2264f43a5fe628a64e282a2b5382617c5ccb21844a2c500eac89764140b\": container with ID starting with e637f2264f43a5fe628a64e282a2b5382617c5ccb21844a2c500eac89764140b not found: ID does not exist" containerID="e637f2264f43a5fe628a64e282a2b5382617c5ccb21844a2c500eac89764140b"
Jul 07 17:00:04.729578 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:04.729533    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e637f2264f43a5fe628a64e282a2b5382617c5ccb21844a2c500eac89764140b"} err="failed to get container status \"e637f2264f43a5fe628a64e282a2b5382617c5ccb21844a2c500eac89764140b\": rpc error: code = NotFound desc = could not find container \"e637f2264f43a5fe628a64e282a2b5382617c5ccb21844a2c500eac89764140b\": container with ID starting with e637f2264f43a5fe628a64e282a2b5382617c5ccb21844a2c500eac89764140b not found: ID does not exist"
Jul 07 17:00:04.729578 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:04.729557    2584 scope.go:117] "RemoveContainer" containerID="643c70dc6e948e9d34f93bf2132bb3e64078e63677e7346de55d90032243574c"
Jul 07 17:00:04.729862 ip-10-0-133-184 kubenswrapper[2584]: E0707 17:00:04.729847    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"643c70dc6e948e9d34f93bf2132bb3e64078e63677e7346de55d90032243574c\": container with ID starting with 643c70dc6e948e9d34f93bf2132bb3e64078e63677e7346de55d90032243574c not found: ID does not exist" containerID="643c70dc6e948e9d34f93bf2132bb3e64078e63677e7346de55d90032243574c"
Jul 07 17:00:04.729913 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:04.729866    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"643c70dc6e948e9d34f93bf2132bb3e64078e63677e7346de55d90032243574c"} err="failed to get container status \"643c70dc6e948e9d34f93bf2132bb3e64078e63677e7346de55d90032243574c\": rpc error: code = NotFound desc = could not find container \"643c70dc6e948e9d34f93bf2132bb3e64078e63677e7346de55d90032243574c\": container with ID starting with 643c70dc6e948e9d34f93bf2132bb3e64078e63677e7346de55d90032243574c not found: ID does not exist"
Jul 07 17:00:04.729913 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:04.729880    2584 scope.go:117] "RemoveContainer" containerID="963c5d89ae46a7152ce7aadb30659fe7275330c7289a57ad6136b0352044e9dc"
Jul 07 17:00:04.730113 ip-10-0-133-184 kubenswrapper[2584]: E0707 17:00:04.730087    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"963c5d89ae46a7152ce7aadb30659fe7275330c7289a57ad6136b0352044e9dc\": container with ID starting with 963c5d89ae46a7152ce7aadb30659fe7275330c7289a57ad6136b0352044e9dc not found: ID does not exist" containerID="963c5d89ae46a7152ce7aadb30659fe7275330c7289a57ad6136b0352044e9dc"
Jul 07 17:00:04.730210 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:04.730120    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963c5d89ae46a7152ce7aadb30659fe7275330c7289a57ad6136b0352044e9dc"} err="failed to get container status \"963c5d89ae46a7152ce7aadb30659fe7275330c7289a57ad6136b0352044e9dc\": rpc error: code = NotFound desc = could not find container \"963c5d89ae46a7152ce7aadb30659fe7275330c7289a57ad6136b0352044e9dc\": container with ID starting with 963c5d89ae46a7152ce7aadb30659fe7275330c7289a57ad6136b0352044e9dc not found: ID does not exist"
Jul 07 17:00:04.849866 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:04.849839    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zj6nk\" (UniqueName: \"kubernetes.io/projected/d3c5c722-5800-4b51-a234-f31c61786bba-kube-api-access-zj6nk\") pod \"d3c5c722-5800-4b51-a234-f31c61786bba\" (UID: \"d3c5c722-5800-4b51-a234-f31c61786bba\") "
Jul 07 17:00:04.850028 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:04.849880    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3c5c722-5800-4b51-a234-f31c61786bba-kube-rbac-proxy-sar-config\") pod \"d3c5c722-5800-4b51-a234-f31c61786bba\" (UID: \"d3c5c722-5800-4b51-a234-f31c61786bba\") "
Jul 07 17:00:04.850028 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:04.849939    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3c5c722-5800-4b51-a234-f31c61786bba-proxy-tls\") pod \"d3c5c722-5800-4b51-a234-f31c61786bba\" (UID: \"d3c5c722-5800-4b51-a234-f31c61786bba\") "
Jul 07 17:00:04.850028 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:04.849967    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3c5c722-5800-4b51-a234-f31c61786bba-kserve-provision-location\") pod \"d3c5c722-5800-4b51-a234-f31c61786bba\" (UID: \"d3c5c722-5800-4b51-a234-f31c61786bba\") "
Jul 07 17:00:04.850028 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:04.850013    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3c5c722-5800-4b51-a234-f31c61786bba-cabundle-cert\") pod \"d3c5c722-5800-4b51-a234-f31c61786bba\" (UID: \"d3c5c722-5800-4b51-a234-f31c61786bba\") "
Jul 07 17:00:04.850337 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:04.850310    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c5c722-5800-4b51-a234-f31c61786bba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d3c5c722-5800-4b51-a234-f31c61786bba" (UID: "d3c5c722-5800-4b51-a234-f31c61786bba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:00:04.850398 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:04.850344    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3c5c722-5800-4b51-a234-f31c61786bba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d3c5c722-5800-4b51-a234-f31c61786bba" (UID: "d3c5c722-5800-4b51-a234-f31c61786bba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:00:04.850461 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:04.850432    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3c5c722-5800-4b51-a234-f31c61786bba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d3c5c722-5800-4b51-a234-f31c61786bba" (UID: "d3c5c722-5800-4b51-a234-f31c61786bba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:00:04.852222 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:04.852197    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3c5c722-5800-4b51-a234-f31c61786bba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d3c5c722-5800-4b51-a234-f31c61786bba" (UID: "d3c5c722-5800-4b51-a234-f31c61786bba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:00:04.852333 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:04.852263    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c5c722-5800-4b51-a234-f31c61786bba-kube-api-access-zj6nk" (OuterVolumeSpecName: "kube-api-access-zj6nk") pod "d3c5c722-5800-4b51-a234-f31c61786bba" (UID: "d3c5c722-5800-4b51-a234-f31c61786bba"). InnerVolumeSpecName "kube-api-access-zj6nk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:00:04.951249 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:04.951212    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3c5c722-5800-4b51-a234-f31c61786bba-cabundle-cert\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:00:04.951249 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:04.951242    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj6nk\" (UniqueName: \"kubernetes.io/projected/d3c5c722-5800-4b51-a234-f31c61786bba-kube-api-access-zj6nk\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:00:04.951249 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:04.951253    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3c5c722-5800-4b51-a234-f31c61786bba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:00:04.951489 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:04.951285    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3c5c722-5800-4b51-a234-f31c61786bba-proxy-tls\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:00:04.951489 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:04.951294    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3c5c722-5800-4b51-a234-f31c61786bba-kserve-provision-location\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:00:05.698521 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:05.698489    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp"
Jul 07 17:00:05.731869 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:05.731843    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp"]
Jul 07 17:00:05.742418 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:05.742391    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6ff8c4ff99-8sgbp"]
Jul 07 17:00:05.895203 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:05.895171    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" path="/var/lib/kubelet/pods/d3c5c722-5800-4b51-a234-f31c61786bba/volumes"
Jul 07 17:00:07.686485 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:07.686447    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z"
Jul 07 17:00:07.687165 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:07.687126    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 07 17:00:07.687470 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:07.687442    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:00:17.687264 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:17.687221    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 07 17:00:17.687862 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:17.687832    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:00:20.772001 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:20.771969    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 17:00:27.687953 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:27.687887    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 07 17:00:27.688405 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:27.688379    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:00:33.533624 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:33.533588    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 17:00:37.687756 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:37.687707    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 07 17:00:37.688246 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:37.688225    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:00:47.687581 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:47.687490    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 07 17:00:47.688096 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:47.688073    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:00:57.687912 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:57.687857    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 07 17:00:57.688478 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:00:57.688317    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:01:07.687217 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:07.687168    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 07 17:01:07.687708 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:07.687654    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:01:12.489609 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:12.489565    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 17:01:17.688699 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:17.688669    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z"
Jul 07 17:01:17.689174 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:17.688832    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z"
Jul 07 17:01:29.820400 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:29.820369    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 17:01:29.874926 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:29.874899    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z"]
Jul 07 17:01:29.875236 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:29.875187    2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="kserve-container" containerID="cri-o://bd9fb6f4a99ae53c7221a976efe8774ed5eb238dec7ee76e7d186110333dbc6c" gracePeriod=30
Jul 07 17:01:29.875236 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:29.875198    2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="agent" containerID="cri-o://8e6c65b4bfcec5eb6adbd9919162a3159a40b8cee6f52456c149565cfb9bd4e3" gracePeriod=30
Jul 07 17:01:29.875440 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:29.875207    2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="kube-rbac-proxy" containerID="cri-o://46f687ef562569dd1fd272bb4695169cdf272b6f490cd606c4057bed9aeecb6d" gracePeriod=30
Jul 07 17:01:29.923782 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:29.923750    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2"]
Jul 07 17:01:29.924166 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:29.924134    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="agent"
Jul 07 17:01:29.924166 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:29.924167    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="agent"
Jul 07 17:01:29.924293 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:29.924186    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="storage-initializer"
Jul 07 17:01:29.924293 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:29.924191    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="storage-initializer"
Jul 07 17:01:29.924293 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:29.924200    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="kube-rbac-proxy"
Jul 07 17:01:29.924293 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:29.924206    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="kube-rbac-proxy"
Jul 07 17:01:29.924293 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:29.924221    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="kserve-container"
Jul 07 17:01:29.924293 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:29.924226    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="kserve-container"
Jul 07 17:01:29.924293 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:29.924278    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="kserve-container"
Jul 07 17:01:29.924293 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:29.924287    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="kube-rbac-proxy"
Jul 07 17:01:29.924293 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:29.924292    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3c5c722-5800-4b51-a234-f31c61786bba" containerName="agent"
Jul 07 17:01:29.927537 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:29.927484    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 17:01:29.927687 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:29.927606    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2"
Jul 07 17:01:29.933283 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:29.933260    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 07 17:01:29.933545 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:29.933528    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 07 17:01:29.940890 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:29.940864    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2"]
Jul 07 17:01:29.985590 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:29.985562    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2\" (UID: \"74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2"
Jul 07 17:01:29.985758 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:29.985616    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2\" (UID: \"74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2"
Jul 07 17:01:29.985758 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:29.985650    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8m97\" (UniqueName: \"kubernetes.io/projected/74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3-kube-api-access-j8m97\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2\" (UID: \"74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2"
Jul 07 17:01:29.985758 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:29.985674    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2\" (UID: \"74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2"
Jul 07 17:01:29.985758 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:29.985703    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2\" (UID: \"74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2"
Jul 07 17:01:30.086891 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:30.086802    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2\" (UID: \"74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2"
Jul 07 17:01:30.086891 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:30.086849    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8m97\" (UniqueName: \"kubernetes.io/projected/74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3-kube-api-access-j8m97\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2\" (UID: \"74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2"
Jul 07 17:01:30.087160 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:30.086988    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2\" (UID: \"74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2"
Jul 07 17:01:30.087160 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:30.087073    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2\" (UID: \"74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2"
Jul 07 17:01:30.087160 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:30.087105    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2\" (UID: \"74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2"
Jul 07 17:01:30.087500 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:30.087472    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2\" (UID: \"74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2"
Jul 07 17:01:30.087830 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:30.087810    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2\" (UID: \"74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2"
Jul 07 17:01:30.087830 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:30.087822    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2\" (UID: \"74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2"
Jul 07 17:01:30.090112 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:30.090090    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2\" (UID: \"74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2"
Jul 07 17:01:30.090752 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:30.090729    2584 generic.go:356] "Generic (PLEG): container finished" podID="eb319868-44d2-46e9-b425-fc0134930aa9" containerID="46f687ef562569dd1fd272bb4695169cdf272b6f490cd606c4057bed9aeecb6d" exitCode=2
Jul 07 17:01:30.090834 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:30.090802    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" event={"ID":"eb319868-44d2-46e9-b425-fc0134930aa9","Type":"ContainerDied","Data":"46f687ef562569dd1fd272bb4695169cdf272b6f490cd606c4057bed9aeecb6d"}
Jul 07 17:01:30.103806 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:30.103778    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8m97\" (UniqueName: \"kubernetes.io/projected/74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3-kube-api-access-j8m97\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2\" (UID: \"74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2"
Jul 07 17:01:30.187641 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:30.187603    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 17:01:30.240601 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:30.240564    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2"
Jul 07 17:01:30.399705 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:30.399592    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2"]
Jul 07 17:01:30.402600 ip-10-0-133-184 kubenswrapper[2584]: W0707 17:01:30.402575    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74c0ecf7_9826_4cfa_8d5a_fd81bb004ce3.slice/crio-505492566a2d375db95cc83cf1173b5feb8076dbaca2a5fbbc788d7013609c42 WatchSource:0}: Error finding container 505492566a2d375db95cc83cf1173b5feb8076dbaca2a5fbbc788d7013609c42: Status 404 returned error can't find the container with id 505492566a2d375db95cc83cf1173b5feb8076dbaca2a5fbbc788d7013609c42
Jul 07 17:01:31.097931 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:31.097895    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2" event={"ID":"74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3","Type":"ContainerStarted","Data":"dd498400666a717256df0a2c3554e81c7fbc73e1e815ca4cf077354c8afc0703"}
Jul 07 17:01:31.098349 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:31.097951    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2" event={"ID":"74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3","Type":"ContainerStarted","Data":"505492566a2d375db95cc83cf1173b5feb8076dbaca2a5fbbc788d7013609c42"}
Jul 07 17:01:32.681338 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:32.681291    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 07 17:01:35.119232 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:35.119198    2584 generic.go:356] "Generic (PLEG): container finished" podID="eb319868-44d2-46e9-b425-fc0134930aa9" containerID="bd9fb6f4a99ae53c7221a976efe8774ed5eb238dec7ee76e7d186110333dbc6c" exitCode=0
Jul 07 17:01:35.119644 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:35.119268    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" event={"ID":"eb319868-44d2-46e9-b425-fc0134930aa9","Type":"ContainerDied","Data":"bd9fb6f4a99ae53c7221a976efe8774ed5eb238dec7ee76e7d186110333dbc6c"}
Jul 07 17:01:37.681856 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:37.681798    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 07 17:01:37.687251 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:37.687203    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 07 17:01:37.687582 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:37.687561    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:01:42.681495 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:42.681456    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 07 17:01:42.681900 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:42.681580    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z"
Jul 07 17:01:47.681475 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:47.681422    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 07 17:01:47.687952 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:47.687913    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 07 17:01:47.688302 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:47.688277    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:01:52.681604 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:52.681557    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 07 17:01:57.682020 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:57.681973    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 07 17:01:57.687448 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:57.687407    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 07 17:01:57.687578 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:57.687539    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z"
Jul 07 17:01:57.687777 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:57.687759    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 17:01:57.687875 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:57.687862    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z"
Jul 07 17:01:59.139454 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:01:59.139417    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 17:02:00.027642 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.027616    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z"
Jul 07 17:02:00.049687 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.049657    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qj4fn\" (UniqueName: \"kubernetes.io/projected/eb319868-44d2-46e9-b425-fc0134930aa9-kube-api-access-qj4fn\") pod \"eb319868-44d2-46e9-b425-fc0134930aa9\" (UID: \"eb319868-44d2-46e9-b425-fc0134930aa9\") "
Jul 07 17:02:00.049820 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.049693    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb319868-44d2-46e9-b425-fc0134930aa9-proxy-tls\") pod \"eb319868-44d2-46e9-b425-fc0134930aa9\" (UID: \"eb319868-44d2-46e9-b425-fc0134930aa9\") "
Jul 07 17:02:00.049820 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.049736    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb319868-44d2-46e9-b425-fc0134930aa9-kube-rbac-proxy-sar-config\") pod \"eb319868-44d2-46e9-b425-fc0134930aa9\" (UID: \"eb319868-44d2-46e9-b425-fc0134930aa9\") "
Jul 07 17:02:00.049820 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.049783    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb319868-44d2-46e9-b425-fc0134930aa9-kserve-provision-location\") pod \"eb319868-44d2-46e9-b425-fc0134930aa9\" (UID: \"eb319868-44d2-46e9-b425-fc0134930aa9\") "
Jul 07 17:02:00.049923 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.049851    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb319868-44d2-46e9-b425-fc0134930aa9-cabundle-cert\") pod \"eb319868-44d2-46e9-b425-fc0134930aa9\" (UID: \"eb319868-44d2-46e9-b425-fc0134930aa9\") "
Jul 07 17:02:00.050236 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.050205    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb319868-44d2-46e9-b425-fc0134930aa9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eb319868-44d2-46e9-b425-fc0134930aa9" (UID: "eb319868-44d2-46e9-b425-fc0134930aa9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:02:00.050236 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.050226    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb319868-44d2-46e9-b425-fc0134930aa9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eb319868-44d2-46e9-b425-fc0134930aa9" (UID: "eb319868-44d2-46e9-b425-fc0134930aa9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:02:00.050436 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.050325    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb319868-44d2-46e9-b425-fc0134930aa9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eb319868-44d2-46e9-b425-fc0134930aa9" (UID: "eb319868-44d2-46e9-b425-fc0134930aa9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:02:00.051998 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.051976    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb319868-44d2-46e9-b425-fc0134930aa9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eb319868-44d2-46e9-b425-fc0134930aa9" (UID: "eb319868-44d2-46e9-b425-fc0134930aa9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:02:00.051998 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.051986    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb319868-44d2-46e9-b425-fc0134930aa9-kube-api-access-qj4fn" (OuterVolumeSpecName: "kube-api-access-qj4fn") pod "eb319868-44d2-46e9-b425-fc0134930aa9" (UID: "eb319868-44d2-46e9-b425-fc0134930aa9"). InnerVolumeSpecName "kube-api-access-qj4fn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:02:00.151120 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.151087    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb319868-44d2-46e9-b425-fc0134930aa9-cabundle-cert\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:02:00.151120 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.151117    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj4fn\" (UniqueName: \"kubernetes.io/projected/eb319868-44d2-46e9-b425-fc0134930aa9-kube-api-access-qj4fn\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:02:00.151120 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.151127    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb319868-44d2-46e9-b425-fc0134930aa9-proxy-tls\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:02:00.151559 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.151135    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb319868-44d2-46e9-b425-fc0134930aa9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:02:00.151559 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.151144    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb319868-44d2-46e9-b425-fc0134930aa9-kserve-provision-location\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:02:00.240621 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.240586    2584 generic.go:356] "Generic (PLEG): container finished" podID="74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3" containerID="dd498400666a717256df0a2c3554e81c7fbc73e1e815ca4cf077354c8afc0703" exitCode=0
Jul 07 17:02:00.240806 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.240664    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2" event={"ID":"74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3","Type":"ContainerDied","Data":"dd498400666a717256df0a2c3554e81c7fbc73e1e815ca4cf077354c8afc0703"}
Jul 07 17:02:00.242246 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.242223    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 17:02:00.243150 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.243114    2584 generic.go:356] "Generic (PLEG): container finished" podID="eb319868-44d2-46e9-b425-fc0134930aa9" containerID="8e6c65b4bfcec5eb6adbd9919162a3159a40b8cee6f52456c149565cfb9bd4e3" exitCode=0
Jul 07 17:02:00.243248 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.243171    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z"
Jul 07 17:02:00.243248 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.243183    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" event={"ID":"eb319868-44d2-46e9-b425-fc0134930aa9","Type":"ContainerDied","Data":"8e6c65b4bfcec5eb6adbd9919162a3159a40b8cee6f52456c149565cfb9bd4e3"}
Jul 07 17:02:00.243248 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.243239    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z" event={"ID":"eb319868-44d2-46e9-b425-fc0134930aa9","Type":"ContainerDied","Data":"049c81f0a2f2cee5e9a496bf6de22aae8a748a63dc8d008b768119c5f59c6f3b"}
Jul 07 17:02:00.243357 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.243270    2584 scope.go:117] "RemoveContainer" containerID="8e6c65b4bfcec5eb6adbd9919162a3159a40b8cee6f52456c149565cfb9bd4e3"
Jul 07 17:02:00.256401 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.256382    2584 scope.go:117] "RemoveContainer" containerID="46f687ef562569dd1fd272bb4695169cdf272b6f490cd606c4057bed9aeecb6d"
Jul 07 17:02:00.264985 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.264960    2584 scope.go:117] "RemoveContainer" containerID="bd9fb6f4a99ae53c7221a976efe8774ed5eb238dec7ee76e7d186110333dbc6c"
Jul 07 17:02:00.272733 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.272712    2584 scope.go:117] "RemoveContainer" containerID="94b0287db2fec149a2778bc398d4f297b5835c15cd2e69bfaf4e36b6bd1d85c8"
Jul 07 17:02:00.280379 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.280359    2584 scope.go:117] "RemoveContainer" containerID="8e6c65b4bfcec5eb6adbd9919162a3159a40b8cee6f52456c149565cfb9bd4e3"
Jul 07 17:02:00.280654 ip-10-0-133-184 kubenswrapper[2584]: E0707 17:02:00.280637    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e6c65b4bfcec5eb6adbd9919162a3159a40b8cee6f52456c149565cfb9bd4e3\": container with ID starting with 8e6c65b4bfcec5eb6adbd9919162a3159a40b8cee6f52456c149565cfb9bd4e3 not found: ID does not exist" containerID="8e6c65b4bfcec5eb6adbd9919162a3159a40b8cee6f52456c149565cfb9bd4e3"
Jul 07 17:02:00.280705 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.280661    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e6c65b4bfcec5eb6adbd9919162a3159a40b8cee6f52456c149565cfb9bd4e3"} err="failed to get container status \"8e6c65b4bfcec5eb6adbd9919162a3159a40b8cee6f52456c149565cfb9bd4e3\": rpc error: code = NotFound desc = could not find container \"8e6c65b4bfcec5eb6adbd9919162a3159a40b8cee6f52456c149565cfb9bd4e3\": container with ID starting with 8e6c65b4bfcec5eb6adbd9919162a3159a40b8cee6f52456c149565cfb9bd4e3 not found: ID does not exist"
Jul 07 17:02:00.280705 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.280681    2584 scope.go:117] "RemoveContainer" containerID="46f687ef562569dd1fd272bb4695169cdf272b6f490cd606c4057bed9aeecb6d"
Jul 07 17:02:00.280944 ip-10-0-133-184 kubenswrapper[2584]: E0707 17:02:00.280923    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46f687ef562569dd1fd272bb4695169cdf272b6f490cd606c4057bed9aeecb6d\": container with ID starting with 46f687ef562569dd1fd272bb4695169cdf272b6f490cd606c4057bed9aeecb6d not found: ID does not exist" containerID="46f687ef562569dd1fd272bb4695169cdf272b6f490cd606c4057bed9aeecb6d"
Jul 07 17:02:00.280997 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.280949    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46f687ef562569dd1fd272bb4695169cdf272b6f490cd606c4057bed9aeecb6d"} err="failed to get container status \"46f687ef562569dd1fd272bb4695169cdf272b6f490cd606c4057bed9aeecb6d\": rpc error: code = NotFound desc = could not find container \"46f687ef562569dd1fd272bb4695169cdf272b6f490cd606c4057bed9aeecb6d\": container with ID starting with 46f687ef562569dd1fd272bb4695169cdf272b6f490cd606c4057bed9aeecb6d not found: ID does not exist"
Jul 07 17:02:00.280997 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.280965    2584 scope.go:117] "RemoveContainer" containerID="bd9fb6f4a99ae53c7221a976efe8774ed5eb238dec7ee76e7d186110333dbc6c"
Jul 07 17:02:00.281191 ip-10-0-133-184 kubenswrapper[2584]: E0707 17:02:00.281176    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd9fb6f4a99ae53c7221a976efe8774ed5eb238dec7ee76e7d186110333dbc6c\": container with ID starting with bd9fb6f4a99ae53c7221a976efe8774ed5eb238dec7ee76e7d186110333dbc6c not found: ID does not exist" containerID="bd9fb6f4a99ae53c7221a976efe8774ed5eb238dec7ee76e7d186110333dbc6c"
Jul 07 17:02:00.281386 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.281195    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd9fb6f4a99ae53c7221a976efe8774ed5eb238dec7ee76e7d186110333dbc6c"} err="failed to get container status \"bd9fb6f4a99ae53c7221a976efe8774ed5eb238dec7ee76e7d186110333dbc6c\": rpc error: code = NotFound desc = could not find container \"bd9fb6f4a99ae53c7221a976efe8774ed5eb238dec7ee76e7d186110333dbc6c\": container with ID starting with bd9fb6f4a99ae53c7221a976efe8774ed5eb238dec7ee76e7d186110333dbc6c not found: ID does not exist"
Jul 07 17:02:00.281386 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.281208    2584 scope.go:117] "RemoveContainer" containerID="94b0287db2fec149a2778bc398d4f297b5835c15cd2e69bfaf4e36b6bd1d85c8"
Jul 07 17:02:00.281535 ip-10-0-133-184 kubenswrapper[2584]: E0707 17:02:00.281430    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94b0287db2fec149a2778bc398d4f297b5835c15cd2e69bfaf4e36b6bd1d85c8\": container with ID starting with 94b0287db2fec149a2778bc398d4f297b5835c15cd2e69bfaf4e36b6bd1d85c8 not found: ID does not exist" containerID="94b0287db2fec149a2778bc398d4f297b5835c15cd2e69bfaf4e36b6bd1d85c8"
Jul 07 17:02:00.281535 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.281449    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94b0287db2fec149a2778bc398d4f297b5835c15cd2e69bfaf4e36b6bd1d85c8"} err="failed to get container status \"94b0287db2fec149a2778bc398d4f297b5835c15cd2e69bfaf4e36b6bd1d85c8\": rpc error: code = NotFound desc = could not find container \"94b0287db2fec149a2778bc398d4f297b5835c15cd2e69bfaf4e36b6bd1d85c8\": container with ID starting with 94b0287db2fec149a2778bc398d4f297b5835c15cd2e69bfaf4e36b6bd1d85c8 not found: ID does not exist"
Jul 07 17:02:00.309746 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.309715    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z"]
Jul 07 17:02:00.315378 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:00.315355    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-7979f68479-zqv5z"]
Jul 07 17:02:01.895851 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:02:01.895817    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" path="/var/lib/kubelet/pods/eb319868-44d2-46e9-b425-fc0134930aa9/volumes"
Jul 07 17:04:39.020262 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:04:39.015469    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2" event={"ID":"74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3","Type":"ContainerStarted","Data":"77f30a97d72494ff7efb437ce92cdbf2b88d39625701a994dff226c7efbe7563"}
Jul 07 17:04:39.020262 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:04:39.015549    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2" event={"ID":"74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3","Type":"ContainerStarted","Data":"b82c6f5eaac8c5c5b202336ee63865a351976ecd20b57b595a060fbf91ae0daa"}
Jul 07 17:04:39.020262 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:04:39.016625    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2"
Jul 07 17:04:39.020262 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:04:39.016709    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2"
Jul 07 17:04:39.020262 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:04:39.019935    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2" podUID="74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 07 17:04:39.070702 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:04:39.070649    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2" podStartSLOduration=1.9176925360000001 podStartE2EDuration="3m10.070631154s" totalImagesPullingTime="2m38.485757643s" totalInitContainerRuntime="29.667180975s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:01:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:04:39.069339948 +0000 UTC m=+1127.670548670" watchObservedRunningTime="2026-07-07 17:04:39.070631154 +0000 UTC m=+1127.671839858"
Jul 07 17:04:40.020338 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:04:40.020297    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2" podUID="74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 07 17:04:41.024006 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:04:41.023962    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2" podUID="74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 07 17:04:46.028570 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:04:46.028542    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2"
Jul 07 17:04:46.029114 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:04:46.029085    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2" podUID="74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 07 17:04:56.029216 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:04:56.029188    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2"
Jul 07 17:05:00.101789 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:00.101756    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2"]
Jul 07 17:05:00.102242 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:00.101980    2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2" podUID="74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3" containerName="kserve-container" containerID="cri-o://b82c6f5eaac8c5c5b202336ee63865a351976ecd20b57b595a060fbf91ae0daa" gracePeriod=30
Jul 07 17:05:00.102242 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:00.102015    2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2" podUID="74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3" containerName="kube-rbac-proxy" containerID="cri-o://77f30a97d72494ff7efb437ce92cdbf2b88d39625701a994dff226c7efbe7563" gracePeriod=30
Jul 07 17:05:00.202187 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:00.202151    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 17:05:00.991702 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:00.991671    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2"
Jul 07 17:05:01.104363 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.104330    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3-cabundle-cert\") pod \"74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3\" (UID: \"74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3\") "
Jul 07 17:05:01.104826 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.104404    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3-kube-rbac-proxy-sar-config\") pod \"74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3\" (UID: \"74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3\") "
Jul 07 17:05:01.104826 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.104436    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3-kserve-provision-location\") pod \"74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3\" (UID: \"74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3\") "
Jul 07 17:05:01.104826 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.104487    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8m97\" (UniqueName: \"kubernetes.io/projected/74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3-kube-api-access-j8m97\") pod \"74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3\" (UID: \"74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3\") "
Jul 07 17:05:01.104826 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.104608    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3-proxy-tls\") pod \"74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3\" (UID: \"74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3\") "
Jul 07 17:05:01.104826 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.104730    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3" (UID: "74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:05:01.105101 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.104824    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3" (UID: "74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:05:01.105101 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.104926    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3-cabundle-cert\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:05:01.105101 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.104947    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:05:01.106524 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.106503    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3" (UID: "74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:05:01.106686 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.106666    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3" (UID: "74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:05:01.106767 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.106746    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3-kube-api-access-j8m97" (OuterVolumeSpecName: "kube-api-access-j8m97") pod "74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3" (UID: "74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3"). InnerVolumeSpecName "kube-api-access-j8m97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:05:01.114413 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.114386    2584 generic.go:356] "Generic (PLEG): container finished" podID="74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3" containerID="77f30a97d72494ff7efb437ce92cdbf2b88d39625701a994dff226c7efbe7563" exitCode=2
Jul 07 17:05:01.114413 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.114412    2584 generic.go:356] "Generic (PLEG): container finished" podID="74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3" containerID="b82c6f5eaac8c5c5b202336ee63865a351976ecd20b57b595a060fbf91ae0daa" exitCode=0
Jul 07 17:05:01.114547 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.114493    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2" event={"ID":"74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3","Type":"ContainerDied","Data":"77f30a97d72494ff7efb437ce92cdbf2b88d39625701a994dff226c7efbe7563"}
Jul 07 17:05:01.114547 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.114506    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2"
Jul 07 17:05:01.114617 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.114592    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2" event={"ID":"74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3","Type":"ContainerDied","Data":"b82c6f5eaac8c5c5b202336ee63865a351976ecd20b57b595a060fbf91ae0daa"}
Jul 07 17:05:01.114654 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.114626    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2" event={"ID":"74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3","Type":"ContainerDied","Data":"505492566a2d375db95cc83cf1173b5feb8076dbaca2a5fbbc788d7013609c42"}
Jul 07 17:05:01.114687 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.114666    2584 scope.go:117] "RemoveContainer" containerID="77f30a97d72494ff7efb437ce92cdbf2b88d39625701a994dff226c7efbe7563"
Jul 07 17:05:01.123514 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.123496    2584 scope.go:117] "RemoveContainer" containerID="b82c6f5eaac8c5c5b202336ee63865a351976ecd20b57b595a060fbf91ae0daa"
Jul 07 17:05:01.131429 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.131414    2584 scope.go:117] "RemoveContainer" containerID="dd498400666a717256df0a2c3554e81c7fbc73e1e815ca4cf077354c8afc0703"
Jul 07 17:05:01.139078 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.139035    2584 scope.go:117] "RemoveContainer" containerID="77f30a97d72494ff7efb437ce92cdbf2b88d39625701a994dff226c7efbe7563"
Jul 07 17:05:01.139350 ip-10-0-133-184 kubenswrapper[2584]: E0707 17:05:01.139331    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77f30a97d72494ff7efb437ce92cdbf2b88d39625701a994dff226c7efbe7563\": container with ID starting with 77f30a97d72494ff7efb437ce92cdbf2b88d39625701a994dff226c7efbe7563 not found: ID does not exist" containerID="77f30a97d72494ff7efb437ce92cdbf2b88d39625701a994dff226c7efbe7563"
Jul 07 17:05:01.139404 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.139359    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77f30a97d72494ff7efb437ce92cdbf2b88d39625701a994dff226c7efbe7563"} err="failed to get container status \"77f30a97d72494ff7efb437ce92cdbf2b88d39625701a994dff226c7efbe7563\": rpc error: code = NotFound desc = could not find container \"77f30a97d72494ff7efb437ce92cdbf2b88d39625701a994dff226c7efbe7563\": container with ID starting with 77f30a97d72494ff7efb437ce92cdbf2b88d39625701a994dff226c7efbe7563 not found: ID does not exist"
Jul 07 17:05:01.139404 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.139380    2584 scope.go:117] "RemoveContainer" containerID="b82c6f5eaac8c5c5b202336ee63865a351976ecd20b57b595a060fbf91ae0daa"
Jul 07 17:05:01.139589 ip-10-0-133-184 kubenswrapper[2584]: E0707 17:05:01.139576    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b82c6f5eaac8c5c5b202336ee63865a351976ecd20b57b595a060fbf91ae0daa\": container with ID starting with b82c6f5eaac8c5c5b202336ee63865a351976ecd20b57b595a060fbf91ae0daa not found: ID does not exist" containerID="b82c6f5eaac8c5c5b202336ee63865a351976ecd20b57b595a060fbf91ae0daa"
Jul 07 17:05:01.139627 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.139592    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b82c6f5eaac8c5c5b202336ee63865a351976ecd20b57b595a060fbf91ae0daa"} err="failed to get container status \"b82c6f5eaac8c5c5b202336ee63865a351976ecd20b57b595a060fbf91ae0daa\": rpc error: code = NotFound desc = could not find container \"b82c6f5eaac8c5c5b202336ee63865a351976ecd20b57b595a060fbf91ae0daa\": container with ID starting with b82c6f5eaac8c5c5b202336ee63865a351976ecd20b57b595a060fbf91ae0daa not found: ID does not exist"
Jul 07 17:05:01.139627 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.139603    2584 scope.go:117] "RemoveContainer" containerID="dd498400666a717256df0a2c3554e81c7fbc73e1e815ca4cf077354c8afc0703"
Jul 07 17:05:01.139862 ip-10-0-133-184 kubenswrapper[2584]: E0707 17:05:01.139843    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd498400666a717256df0a2c3554e81c7fbc73e1e815ca4cf077354c8afc0703\": container with ID starting with dd498400666a717256df0a2c3554e81c7fbc73e1e815ca4cf077354c8afc0703 not found: ID does not exist" containerID="dd498400666a717256df0a2c3554e81c7fbc73e1e815ca4cf077354c8afc0703"
Jul 07 17:05:01.139910 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.139867    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd498400666a717256df0a2c3554e81c7fbc73e1e815ca4cf077354c8afc0703"} err="failed to get container status \"dd498400666a717256df0a2c3554e81c7fbc73e1e815ca4cf077354c8afc0703\": rpc error: code = NotFound desc = could not find container \"dd498400666a717256df0a2c3554e81c7fbc73e1e815ca4cf077354c8afc0703\": container with ID starting with dd498400666a717256df0a2c3554e81c7fbc73e1e815ca4cf077354c8afc0703 not found: ID does not exist"
Jul 07 17:05:01.139910 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.139883    2584 scope.go:117] "RemoveContainer" containerID="77f30a97d72494ff7efb437ce92cdbf2b88d39625701a994dff226c7efbe7563"
Jul 07 17:05:01.140198 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.140169    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77f30a97d72494ff7efb437ce92cdbf2b88d39625701a994dff226c7efbe7563"} err="failed to get container status \"77f30a97d72494ff7efb437ce92cdbf2b88d39625701a994dff226c7efbe7563\": rpc error: code = NotFound desc = could not find container \"77f30a97d72494ff7efb437ce92cdbf2b88d39625701a994dff226c7efbe7563\": container with ID starting with 77f30a97d72494ff7efb437ce92cdbf2b88d39625701a994dff226c7efbe7563 not found: ID does not exist"
Jul 07 17:05:01.140198 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.140199    2584 scope.go:117] "RemoveContainer" containerID="b82c6f5eaac8c5c5b202336ee63865a351976ecd20b57b595a060fbf91ae0daa"
Jul 07 17:05:01.140489 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.140468    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b82c6f5eaac8c5c5b202336ee63865a351976ecd20b57b595a060fbf91ae0daa"} err="failed to get container status \"b82c6f5eaac8c5c5b202336ee63865a351976ecd20b57b595a060fbf91ae0daa\": rpc error: code = NotFound desc = could not find container \"b82c6f5eaac8c5c5b202336ee63865a351976ecd20b57b595a060fbf91ae0daa\": container with ID starting with b82c6f5eaac8c5c5b202336ee63865a351976ecd20b57b595a060fbf91ae0daa not found: ID does not exist"
Jul 07 17:05:01.140578 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.140491    2584 scope.go:117] "RemoveContainer" containerID="dd498400666a717256df0a2c3554e81c7fbc73e1e815ca4cf077354c8afc0703"
Jul 07 17:05:01.140744 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.140720    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd498400666a717256df0a2c3554e81c7fbc73e1e815ca4cf077354c8afc0703"} err="failed to get container status \"dd498400666a717256df0a2c3554e81c7fbc73e1e815ca4cf077354c8afc0703\": rpc error: code = NotFound desc = could not find container \"dd498400666a717256df0a2c3554e81c7fbc73e1e815ca4cf077354c8afc0703\": container with ID starting with dd498400666a717256df0a2c3554e81c7fbc73e1e815ca4cf077354c8afc0703 not found: ID does not exist"
Jul 07 17:05:01.142599 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.142580    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2"]
Jul 07 17:05:01.144682 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.144662    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-vr5f2"]
Jul 07 17:05:01.205651 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.205619    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3-proxy-tls\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:05:01.205651 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.205645    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3-kserve-provision-location\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:05:01.205651 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.205654    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8m97\" (UniqueName: \"kubernetes.io/projected/74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3-kube-api-access-j8m97\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:05:01.893238 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:01.893203    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3" path="/var/lib/kubelet/pods/74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3/volumes"
Jul 07 17:05:05.359975 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.359921    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 17:05:05.457322 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.457291    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb"]
Jul 07 17:05:05.457670 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.457656    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="storage-initializer"
Jul 07 17:05:05.457719 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.457672    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="storage-initializer"
Jul 07 17:05:05.457719 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.457685    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="kserve-container"
Jul 07 17:05:05.457719 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.457690    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="kserve-container"
Jul 07 17:05:05.457719 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.457700    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3" containerName="storage-initializer"
Jul 07 17:05:05.457719 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.457707    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3" containerName="storage-initializer"
Jul 07 17:05:05.457719 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.457712    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3" containerName="kube-rbac-proxy"
Jul 07 17:05:05.457719 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.457717    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3" containerName="kube-rbac-proxy"
Jul 07 17:05:05.457931 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.457729    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="kube-rbac-proxy"
Jul 07 17:05:05.457931 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.457734    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="kube-rbac-proxy"
Jul 07 17:05:05.457931 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.457741    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="agent"
Jul 07 17:05:05.457931 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.457746    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="agent"
Jul 07 17:05:05.457931 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.457758    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3" containerName="kserve-container"
Jul 07 17:05:05.457931 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.457764    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3" containerName="kserve-container"
Jul 07 17:05:05.457931 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.457811    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="kube-rbac-proxy"
Jul 07 17:05:05.457931 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.457818    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3" containerName="kube-rbac-proxy"
Jul 07 17:05:05.457931 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.457825    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="agent"
Jul 07 17:05:05.457931 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.457832    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb319868-44d2-46e9-b425-fc0134930aa9" containerName="kserve-container"
Jul 07 17:05:05.457931 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.457839    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="74c0ecf7-9826-4cfa-8d5a-fd81bb004ce3" containerName="kserve-container"
Jul 07 17:05:05.490609 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.490581    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb"]
Jul 07 17:05:05.490766 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.490713    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb"
Jul 07 17:05:05.494275 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.494246    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 17:05:05.494275 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.494246    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 17:05:05.494493 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.494296    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 17:05:05.494493 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.494251    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 07 17:05:05.495705 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.495677    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7526s\""
Jul 07 17:05:05.496134 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.496108    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 17:05:05.496556 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.496535    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 07 17:05:05.641633 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.641536    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edc48a53-94c1-4ed9-b860-15ea0745df12-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-ldjvb\" (UID: \"edc48a53-94c1-4ed9-b860-15ea0745df12\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb"
Jul 07 17:05:05.641633 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.641579    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edc48a53-94c1-4ed9-b860-15ea0745df12-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-ldjvb\" (UID: \"edc48a53-94c1-4ed9-b860-15ea0745df12\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb"
Jul 07 17:05:05.641633 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.641604    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkt6t\" (UniqueName: \"kubernetes.io/projected/edc48a53-94c1-4ed9-b860-15ea0745df12-kube-api-access-gkt6t\") pod \"isvc-autogluon-v2-predictor-67b849d456-ldjvb\" (UID: \"edc48a53-94c1-4ed9-b860-15ea0745df12\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb"
Jul 07 17:05:05.641875 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.641702    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edc48a53-94c1-4ed9-b860-15ea0745df12-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-ldjvb\" (UID: \"edc48a53-94c1-4ed9-b860-15ea0745df12\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb"
Jul 07 17:05:05.641875 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.641744    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edc48a53-94c1-4ed9-b860-15ea0745df12-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-ldjvb\" (UID: \"edc48a53-94c1-4ed9-b860-15ea0745df12\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb"
Jul 07 17:05:05.739860 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.739814    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 17:05:05.742758 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.742734    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edc48a53-94c1-4ed9-b860-15ea0745df12-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-ldjvb\" (UID: \"edc48a53-94c1-4ed9-b860-15ea0745df12\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb"
Jul 07 17:05:05.742839 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.742769    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edc48a53-94c1-4ed9-b860-15ea0745df12-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-ldjvb\" (UID: \"edc48a53-94c1-4ed9-b860-15ea0745df12\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb"
Jul 07 17:05:05.742839 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.742789    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkt6t\" (UniqueName: \"kubernetes.io/projected/edc48a53-94c1-4ed9-b860-15ea0745df12-kube-api-access-gkt6t\") pod \"isvc-autogluon-v2-predictor-67b849d456-ldjvb\" (UID: \"edc48a53-94c1-4ed9-b860-15ea0745df12\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb"
Jul 07 17:05:05.742839 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.742830    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edc48a53-94c1-4ed9-b860-15ea0745df12-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-ldjvb\" (UID: \"edc48a53-94c1-4ed9-b860-15ea0745df12\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb"
Jul 07 17:05:05.742960 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.742856    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edc48a53-94c1-4ed9-b860-15ea0745df12-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-ldjvb\" (UID: \"edc48a53-94c1-4ed9-b860-15ea0745df12\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb"
Jul 07 17:05:05.743213 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.743192    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edc48a53-94c1-4ed9-b860-15ea0745df12-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-ldjvb\" (UID: \"edc48a53-94c1-4ed9-b860-15ea0745df12\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb"
Jul 07 17:05:05.743596 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.743577    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edc48a53-94c1-4ed9-b860-15ea0745df12-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-ldjvb\" (UID: \"edc48a53-94c1-4ed9-b860-15ea0745df12\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb"
Jul 07 17:05:05.743641 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.743609    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edc48a53-94c1-4ed9-b860-15ea0745df12-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-ldjvb\" (UID: \"edc48a53-94c1-4ed9-b860-15ea0745df12\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb"
Jul 07 17:05:05.745439 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.745416    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edc48a53-94c1-4ed9-b860-15ea0745df12-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-ldjvb\" (UID: \"edc48a53-94c1-4ed9-b860-15ea0745df12\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb"
Jul 07 17:05:05.753745 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.753723    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkt6t\" (UniqueName: \"kubernetes.io/projected/edc48a53-94c1-4ed9-b860-15ea0745df12-kube-api-access-gkt6t\") pod \"isvc-autogluon-v2-predictor-67b849d456-ldjvb\" (UID: \"edc48a53-94c1-4ed9-b860-15ea0745df12\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb"
Jul 07 17:05:05.807152 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.807117    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb"
Jul 07 17:05:05.958797 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:05.958767    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb"]
Jul 07 17:05:05.960983 ip-10-0-133-184 kubenswrapper[2584]: W0707 17:05:05.960949    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc48a53_94c1_4ed9_b860_15ea0745df12.slice/crio-289e29a2fa2ac711c11e0186804bd9f1b3a0be991657a2687ee4ba5cf81feafe WatchSource:0}: Error finding container 289e29a2fa2ac711c11e0186804bd9f1b3a0be991657a2687ee4ba5cf81feafe: Status 404 returned error can't find the container with id 289e29a2fa2ac711c11e0186804bd9f1b3a0be991657a2687ee4ba5cf81feafe
Jul 07 17:05:06.137254 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:06.137218    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb" event={"ID":"edc48a53-94c1-4ed9-b860-15ea0745df12","Type":"ContainerStarted","Data":"fb13fa53367a207dd82cef80e4204b3b9ca6eec3cbb6f7ca426ab177c3f2b77d"}
Jul 07 17:05:06.137464 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:06.137276    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb" event={"ID":"edc48a53-94c1-4ed9-b860-15ea0745df12","Type":"ContainerStarted","Data":"289e29a2fa2ac711c11e0186804bd9f1b3a0be991657a2687ee4ba5cf81feafe"}
Jul 07 17:05:35.268822 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:35.268777    2584 generic.go:356] "Generic (PLEG): container finished" podID="edc48a53-94c1-4ed9-b860-15ea0745df12" containerID="fb13fa53367a207dd82cef80e4204b3b9ca6eec3cbb6f7ca426ab177c3f2b77d" exitCode=0
Jul 07 17:05:35.269282 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:35.268836    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb" event={"ID":"edc48a53-94c1-4ed9-b860-15ea0745df12","Type":"ContainerDied","Data":"fb13fa53367a207dd82cef80e4204b3b9ca6eec3cbb6f7ca426ab177c3f2b77d"}
Jul 07 17:05:36.275888 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:36.275845    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb" event={"ID":"edc48a53-94c1-4ed9-b860-15ea0745df12","Type":"ContainerStarted","Data":"5ff2773a4d2dd2f7b02bcf9881aeec7b6d72dbc7b6304614f513f08a0898d266"}
Jul 07 17:05:36.276339 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:36.275909    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb" event={"ID":"edc48a53-94c1-4ed9-b860-15ea0745df12","Type":"ContainerStarted","Data":"b8b7936c5e5af84604597bf15c6210271383dcea9c14fb287081f11914ff5a4a"}
Jul 07 17:05:36.276339 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:36.276181    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb"
Jul 07 17:05:36.309735 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:36.309679    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb" podStartSLOduration=2.184809119 podStartE2EDuration="31.309661809s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.12485269s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:05:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:05:36.30847615 +0000 UTC m=+1184.909684858" watchObservedRunningTime="2026-07-07 17:05:36.309661809 +0000 UTC m=+1184.910870514"
Jul 07 17:05:37.281360 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:37.281313    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb"
Jul 07 17:05:43.291070 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:05:43.291011    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb"
Jul 07 17:07:08.892093 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:08.891998    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb"
Jul 07 17:07:15.655104 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:15.655065    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb"]
Jul 07 17:07:15.655509 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:15.655373    2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb" podUID="edc48a53-94c1-4ed9-b860-15ea0745df12" containerName="kserve-container" containerID="cri-o://b8b7936c5e5af84604597bf15c6210271383dcea9c14fb287081f11914ff5a4a" gracePeriod=30
Jul 07 17:07:15.655509 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:15.655397    2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb" podUID="edc48a53-94c1-4ed9-b860-15ea0745df12" containerName="kube-rbac-proxy" containerID="cri-o://5ff2773a4d2dd2f7b02bcf9881aeec7b6d72dbc7b6304614f513f08a0898d266" gracePeriod=30
Jul 07 17:07:15.754313 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:15.754270    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 17:07:16.601280 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.601255    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb"
Jul 07 17:07:16.666958 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.666869    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edc48a53-94c1-4ed9-b860-15ea0745df12-proxy-tls\") pod \"edc48a53-94c1-4ed9-b860-15ea0745df12\" (UID: \"edc48a53-94c1-4ed9-b860-15ea0745df12\") "
Jul 07 17:07:16.666958 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.666946    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edc48a53-94c1-4ed9-b860-15ea0745df12-kube-rbac-proxy-sar-config\") pod \"edc48a53-94c1-4ed9-b860-15ea0745df12\" (UID: \"edc48a53-94c1-4ed9-b860-15ea0745df12\") "
Jul 07 17:07:16.667494 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.666971    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkt6t\" (UniqueName: \"kubernetes.io/projected/edc48a53-94c1-4ed9-b860-15ea0745df12-kube-api-access-gkt6t\") pod \"edc48a53-94c1-4ed9-b860-15ea0745df12\" (UID: \"edc48a53-94c1-4ed9-b860-15ea0745df12\") "
Jul 07 17:07:16.667494 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.666994    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edc48a53-94c1-4ed9-b860-15ea0745df12-kserve-provision-location\") pod \"edc48a53-94c1-4ed9-b860-15ea0745df12\" (UID: \"edc48a53-94c1-4ed9-b860-15ea0745df12\") "
Jul 07 17:07:16.667494 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.667023    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edc48a53-94c1-4ed9-b860-15ea0745df12-cabundle-cert\") pod \"edc48a53-94c1-4ed9-b860-15ea0745df12\" (UID: \"edc48a53-94c1-4ed9-b860-15ea0745df12\") "
Jul 07 17:07:16.667494 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.667344    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edc48a53-94c1-4ed9-b860-15ea0745df12-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "edc48a53-94c1-4ed9-b860-15ea0745df12" (UID: "edc48a53-94c1-4ed9-b860-15ea0745df12"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:07:16.667666 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.667509    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edc48a53-94c1-4ed9-b860-15ea0745df12-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "edc48a53-94c1-4ed9-b860-15ea0745df12" (UID: "edc48a53-94c1-4ed9-b860-15ea0745df12"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:07:16.669340 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.669314    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edc48a53-94c1-4ed9-b860-15ea0745df12-kube-api-access-gkt6t" (OuterVolumeSpecName: "kube-api-access-gkt6t") pod "edc48a53-94c1-4ed9-b860-15ea0745df12" (UID: "edc48a53-94c1-4ed9-b860-15ea0745df12"). InnerVolumeSpecName "kube-api-access-gkt6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:07:16.669340 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.669312    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edc48a53-94c1-4ed9-b860-15ea0745df12-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "edc48a53-94c1-4ed9-b860-15ea0745df12" (UID: "edc48a53-94c1-4ed9-b860-15ea0745df12"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:07:16.671409 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.671385    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc48a53-94c1-4ed9-b860-15ea0745df12-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "edc48a53-94c1-4ed9-b860-15ea0745df12" (UID: "edc48a53-94c1-4ed9-b860-15ea0745df12"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:07:16.720810 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.720770    2584 generic.go:356] "Generic (PLEG): container finished" podID="edc48a53-94c1-4ed9-b860-15ea0745df12" containerID="5ff2773a4d2dd2f7b02bcf9881aeec7b6d72dbc7b6304614f513f08a0898d266" exitCode=2
Jul 07 17:07:16.720810 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.720796    2584 generic.go:356] "Generic (PLEG): container finished" podID="edc48a53-94c1-4ed9-b860-15ea0745df12" containerID="b8b7936c5e5af84604597bf15c6210271383dcea9c14fb287081f11914ff5a4a" exitCode=0
Jul 07 17:07:16.721028 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.720830    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb"
Jul 07 17:07:16.721028 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.720853    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb" event={"ID":"edc48a53-94c1-4ed9-b860-15ea0745df12","Type":"ContainerDied","Data":"5ff2773a4d2dd2f7b02bcf9881aeec7b6d72dbc7b6304614f513f08a0898d266"}
Jul 07 17:07:16.721028 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.720914    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb" event={"ID":"edc48a53-94c1-4ed9-b860-15ea0745df12","Type":"ContainerDied","Data":"b8b7936c5e5af84604597bf15c6210271383dcea9c14fb287081f11914ff5a4a"}
Jul 07 17:07:16.721028 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.720936    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb" event={"ID":"edc48a53-94c1-4ed9-b860-15ea0745df12","Type":"ContainerDied","Data":"289e29a2fa2ac711c11e0186804bd9f1b3a0be991657a2687ee4ba5cf81feafe"}
Jul 07 17:07:16.721028 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.720972    2584 scope.go:117] "RemoveContainer" containerID="5ff2773a4d2dd2f7b02bcf9881aeec7b6d72dbc7b6304614f513f08a0898d266"
Jul 07 17:07:16.730041 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.730021    2584 scope.go:117] "RemoveContainer" containerID="b8b7936c5e5af84604597bf15c6210271383dcea9c14fb287081f11914ff5a4a"
Jul 07 17:07:16.738130 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.738110    2584 scope.go:117] "RemoveContainer" containerID="fb13fa53367a207dd82cef80e4204b3b9ca6eec3cbb6f7ca426ab177c3f2b77d"
Jul 07 17:07:16.746203 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.746176    2584 scope.go:117] "RemoveContainer" containerID="5ff2773a4d2dd2f7b02bcf9881aeec7b6d72dbc7b6304614f513f08a0898d266"
Jul 07 17:07:16.746488 ip-10-0-133-184 kubenswrapper[2584]: E0707 17:07:16.746468    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ff2773a4d2dd2f7b02bcf9881aeec7b6d72dbc7b6304614f513f08a0898d266\": container with ID starting with 5ff2773a4d2dd2f7b02bcf9881aeec7b6d72dbc7b6304614f513f08a0898d266 not found: ID does not exist" containerID="5ff2773a4d2dd2f7b02bcf9881aeec7b6d72dbc7b6304614f513f08a0898d266"
Jul 07 17:07:16.746548 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.746498    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ff2773a4d2dd2f7b02bcf9881aeec7b6d72dbc7b6304614f513f08a0898d266"} err="failed to get container status \"5ff2773a4d2dd2f7b02bcf9881aeec7b6d72dbc7b6304614f513f08a0898d266\": rpc error: code = NotFound desc = could not find container \"5ff2773a4d2dd2f7b02bcf9881aeec7b6d72dbc7b6304614f513f08a0898d266\": container with ID starting with 5ff2773a4d2dd2f7b02bcf9881aeec7b6d72dbc7b6304614f513f08a0898d266 not found: ID does not exist"
Jul 07 17:07:16.746548 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.746518    2584 scope.go:117] "RemoveContainer" containerID="b8b7936c5e5af84604597bf15c6210271383dcea9c14fb287081f11914ff5a4a"
Jul 07 17:07:16.746805 ip-10-0-133-184 kubenswrapper[2584]: E0707 17:07:16.746784    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8b7936c5e5af84604597bf15c6210271383dcea9c14fb287081f11914ff5a4a\": container with ID starting with b8b7936c5e5af84604597bf15c6210271383dcea9c14fb287081f11914ff5a4a not found: ID does not exist" containerID="b8b7936c5e5af84604597bf15c6210271383dcea9c14fb287081f11914ff5a4a"
Jul 07 17:07:16.746858 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.746815    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8b7936c5e5af84604597bf15c6210271383dcea9c14fb287081f11914ff5a4a"} err="failed to get container status \"b8b7936c5e5af84604597bf15c6210271383dcea9c14fb287081f11914ff5a4a\": rpc error: code = NotFound desc = could not find container \"b8b7936c5e5af84604597bf15c6210271383dcea9c14fb287081f11914ff5a4a\": container with ID starting with b8b7936c5e5af84604597bf15c6210271383dcea9c14fb287081f11914ff5a4a not found: ID does not exist"
Jul 07 17:07:16.746858 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.746838    2584 scope.go:117] "RemoveContainer" containerID="fb13fa53367a207dd82cef80e4204b3b9ca6eec3cbb6f7ca426ab177c3f2b77d"
Jul 07 17:07:16.747374 ip-10-0-133-184 kubenswrapper[2584]: E0707 17:07:16.747352    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb13fa53367a207dd82cef80e4204b3b9ca6eec3cbb6f7ca426ab177c3f2b77d\": container with ID starting with fb13fa53367a207dd82cef80e4204b3b9ca6eec3cbb6f7ca426ab177c3f2b77d not found: ID does not exist" containerID="fb13fa53367a207dd82cef80e4204b3b9ca6eec3cbb6f7ca426ab177c3f2b77d"
Jul 07 17:07:16.747488 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.747379    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb13fa53367a207dd82cef80e4204b3b9ca6eec3cbb6f7ca426ab177c3f2b77d"} err="failed to get container status \"fb13fa53367a207dd82cef80e4204b3b9ca6eec3cbb6f7ca426ab177c3f2b77d\": rpc error: code = NotFound desc = could not find container \"fb13fa53367a207dd82cef80e4204b3b9ca6eec3cbb6f7ca426ab177c3f2b77d\": container with ID starting with fb13fa53367a207dd82cef80e4204b3b9ca6eec3cbb6f7ca426ab177c3f2b77d not found: ID does not exist"
Jul 07 17:07:16.747488 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.747398    2584 scope.go:117] "RemoveContainer" containerID="5ff2773a4d2dd2f7b02bcf9881aeec7b6d72dbc7b6304614f513f08a0898d266"
Jul 07 17:07:16.747685 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.747665    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ff2773a4d2dd2f7b02bcf9881aeec7b6d72dbc7b6304614f513f08a0898d266"} err="failed to get container status \"5ff2773a4d2dd2f7b02bcf9881aeec7b6d72dbc7b6304614f513f08a0898d266\": rpc error: code = NotFound desc = could not find container \"5ff2773a4d2dd2f7b02bcf9881aeec7b6d72dbc7b6304614f513f08a0898d266\": container with ID starting with 5ff2773a4d2dd2f7b02bcf9881aeec7b6d72dbc7b6304614f513f08a0898d266 not found: ID does not exist"
Jul 07 17:07:16.747729 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.747687    2584 scope.go:117] "RemoveContainer" containerID="b8b7936c5e5af84604597bf15c6210271383dcea9c14fb287081f11914ff5a4a"
Jul 07 17:07:16.747950 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.747931    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8b7936c5e5af84604597bf15c6210271383dcea9c14fb287081f11914ff5a4a"} err="failed to get container status \"b8b7936c5e5af84604597bf15c6210271383dcea9c14fb287081f11914ff5a4a\": rpc error: code = NotFound desc = could not find container \"b8b7936c5e5af84604597bf15c6210271383dcea9c14fb287081f11914ff5a4a\": container with ID starting with b8b7936c5e5af84604597bf15c6210271383dcea9c14fb287081f11914ff5a4a not found: ID does not exist"
Jul 07 17:07:16.748019 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.747952    2584 scope.go:117] "RemoveContainer" containerID="fb13fa53367a207dd82cef80e4204b3b9ca6eec3cbb6f7ca426ab177c3f2b77d"
Jul 07 17:07:16.748242 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.748226    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb13fa53367a207dd82cef80e4204b3b9ca6eec3cbb6f7ca426ab177c3f2b77d"} err="failed to get container status \"fb13fa53367a207dd82cef80e4204b3b9ca6eec3cbb6f7ca426ab177c3f2b77d\": rpc error: code = NotFound desc = could not find container \"fb13fa53367a207dd82cef80e4204b3b9ca6eec3cbb6f7ca426ab177c3f2b77d\": container with ID starting with fb13fa53367a207dd82cef80e4204b3b9ca6eec3cbb6f7ca426ab177c3f2b77d not found: ID does not exist"
Jul 07 17:07:16.749083 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.749064    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb"]
Jul 07 17:07:16.754909 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.754885    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-ldjvb"]
Jul 07 17:07:16.767572 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.767547    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edc48a53-94c1-4ed9-b860-15ea0745df12-kserve-provision-location\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:07:16.767572 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.767572    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/edc48a53-94c1-4ed9-b860-15ea0745df12-cabundle-cert\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:07:16.767723 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.767584    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/edc48a53-94c1-4ed9-b860-15ea0745df12-proxy-tls\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:07:16.767723 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.767591    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/edc48a53-94c1-4ed9-b860-15ea0745df12-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:07:16.767723 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:16.767601    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkt6t\" (UniqueName: \"kubernetes.io/projected/edc48a53-94c1-4ed9-b860-15ea0745df12-kube-api-access-gkt6t\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:07:17.893242 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:17.893208    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edc48a53-94c1-4ed9-b860-15ea0745df12" path="/var/lib/kubelet/pods/edc48a53-94c1-4ed9-b860-15ea0745df12/volumes"
Jul 07 17:07:20.713464 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:20.713431    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 17:07:20.811545 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:20.811507    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h"]
Jul 07 17:07:20.811875 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:20.811862    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edc48a53-94c1-4ed9-b860-15ea0745df12" containerName="storage-initializer"
Jul 07 17:07:20.811919 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:20.811877    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="edc48a53-94c1-4ed9-b860-15ea0745df12" containerName="storage-initializer"
Jul 07 17:07:20.811919 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:20.811888    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edc48a53-94c1-4ed9-b860-15ea0745df12" containerName="kserve-container"
Jul 07 17:07:20.811919 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:20.811893    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="edc48a53-94c1-4ed9-b860-15ea0745df12" containerName="kserve-container"
Jul 07 17:07:20.811919 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:20.811904    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edc48a53-94c1-4ed9-b860-15ea0745df12" containerName="kube-rbac-proxy"
Jul 07 17:07:20.811919 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:20.811909    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="edc48a53-94c1-4ed9-b860-15ea0745df12" containerName="kube-rbac-proxy"
Jul 07 17:07:20.812143 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:20.811961    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="edc48a53-94c1-4ed9-b860-15ea0745df12" containerName="kube-rbac-proxy"
Jul 07 17:07:20.812143 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:20.811972    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="edc48a53-94c1-4ed9-b860-15ea0745df12" containerName="kserve-container"
Jul 07 17:07:20.813986 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:20.813968    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h"
Jul 07 17:07:20.820584 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:20.820560    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 17:07:20.820788 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:20.820562    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 17:07:20.820788 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:20.820634    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 07 17:07:20.820928 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:20.820888    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 17:07:20.820928 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:20.820891    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 07 17:07:20.821650 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:20.821631    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7526s\""
Jul 07 17:07:20.821753 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:20.821652    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 17:07:20.830857 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:20.830832    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h"]
Jul 07 17:07:20.900556 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:20.900509    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffd72431-887a-44b9-b39e-10f4bcae69be-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h\" (UID: \"ffd72431-887a-44b9-b39e-10f4bcae69be\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h"
Jul 07 17:07:20.900739 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:20.900564    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffd72431-887a-44b9-b39e-10f4bcae69be-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h\" (UID: \"ffd72431-887a-44b9-b39e-10f4bcae69be\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h"
Jul 07 17:07:20.900739 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:20.900665    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dslz\" (UniqueName: \"kubernetes.io/projected/ffd72431-887a-44b9-b39e-10f4bcae69be-kube-api-access-6dslz\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h\" (UID: \"ffd72431-887a-44b9-b39e-10f4bcae69be\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h"
Jul 07 17:07:20.900739 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:20.900711    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffd72431-887a-44b9-b39e-10f4bcae69be-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h\" (UID: \"ffd72431-887a-44b9-b39e-10f4bcae69be\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h"
Jul 07 17:07:20.900848 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:20.900744    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffd72431-887a-44b9-b39e-10f4bcae69be-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h\" (UID: \"ffd72431-887a-44b9-b39e-10f4bcae69be\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h"
Jul 07 17:07:21.001693 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:21.001612    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dslz\" (UniqueName: \"kubernetes.io/projected/ffd72431-887a-44b9-b39e-10f4bcae69be-kube-api-access-6dslz\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h\" (UID: \"ffd72431-887a-44b9-b39e-10f4bcae69be\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h"
Jul 07 17:07:21.001693 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:21.001664    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffd72431-887a-44b9-b39e-10f4bcae69be-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h\" (UID: \"ffd72431-887a-44b9-b39e-10f4bcae69be\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h"
Jul 07 17:07:21.001693 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:21.001685    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffd72431-887a-44b9-b39e-10f4bcae69be-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h\" (UID: \"ffd72431-887a-44b9-b39e-10f4bcae69be\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h"
Jul 07 17:07:21.001917 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:21.001728    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffd72431-887a-44b9-b39e-10f4bcae69be-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h\" (UID: \"ffd72431-887a-44b9-b39e-10f4bcae69be\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h"
Jul 07 17:07:21.001917 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:21.001751    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffd72431-887a-44b9-b39e-10f4bcae69be-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h\" (UID: \"ffd72431-887a-44b9-b39e-10f4bcae69be\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h"
Jul 07 17:07:21.001917 ip-10-0-133-184 kubenswrapper[2584]: E0707 17:07:21.001853    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-serving-cert: secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Jul 07 17:07:21.001917 ip-10-0-133-184 kubenswrapper[2584]: E0707 17:07:21.001912    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffd72431-887a-44b9-b39e-10f4bcae69be-proxy-tls podName:ffd72431-887a-44b9-b39e-10f4bcae69be nodeName:}" failed. No retries permitted until 2026-07-07 17:07:21.501891999 +0000 UTC m=+1290.103100695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ffd72431-887a-44b9-b39e-10f4bcae69be-proxy-tls") pod "isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h" (UID: "ffd72431-887a-44b9-b39e-10f4bcae69be") : secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Jul 07 17:07:21.002219 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:21.002195    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffd72431-887a-44b9-b39e-10f4bcae69be-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h\" (UID: \"ffd72431-887a-44b9-b39e-10f4bcae69be\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h"
Jul 07 17:07:21.002414 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:21.002391    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffd72431-887a-44b9-b39e-10f4bcae69be-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h\" (UID: \"ffd72431-887a-44b9-b39e-10f4bcae69be\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h"
Jul 07 17:07:21.002474 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:21.002433    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffd72431-887a-44b9-b39e-10f4bcae69be-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h\" (UID: \"ffd72431-887a-44b9-b39e-10f4bcae69be\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h"
Jul 07 17:07:21.015616 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:21.015590    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dslz\" (UniqueName: \"kubernetes.io/projected/ffd72431-887a-44b9-b39e-10f4bcae69be-kube-api-access-6dslz\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h\" (UID: \"ffd72431-887a-44b9-b39e-10f4bcae69be\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h"
Jul 07 17:07:21.273410 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:21.273300    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 17:07:21.505413 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:21.505375    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffd72431-887a-44b9-b39e-10f4bcae69be-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h\" (UID: \"ffd72431-887a-44b9-b39e-10f4bcae69be\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h"
Jul 07 17:07:21.508099 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:21.508074    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffd72431-887a-44b9-b39e-10f4bcae69be-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h\" (UID: \"ffd72431-887a-44b9-b39e-10f4bcae69be\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h"
Jul 07 17:07:21.726215 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:21.726166    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h"
Jul 07 17:07:21.881596 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:21.881566    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h"]
Jul 07 17:07:21.883013 ip-10-0-133-184 kubenswrapper[2584]: W0707 17:07:21.882980    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd72431_887a_44b9_b39e_10f4bcae69be.slice/crio-d7b7bae5f74ab435efd587922f0b489b494fdfbe6d624211d1f6acc1e97e36f6 WatchSource:0}: Error finding container d7b7bae5f74ab435efd587922f0b489b494fdfbe6d624211d1f6acc1e97e36f6: Status 404 returned error can't find the container with id d7b7bae5f74ab435efd587922f0b489b494fdfbe6d624211d1f6acc1e97e36f6
Jul 07 17:07:22.751639 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:22.751601    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h" event={"ID":"ffd72431-887a-44b9-b39e-10f4bcae69be","Type":"ContainerStarted","Data":"3f730e67ea2ae606ed4362e30ea130a35d8df4206f39863ce6b43f73d1b5329d"}
Jul 07 17:07:22.752068 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:22.751686    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h" event={"ID":"ffd72431-887a-44b9-b39e-10f4bcae69be","Type":"ContainerStarted","Data":"d7b7bae5f74ab435efd587922f0b489b494fdfbe6d624211d1f6acc1e97e36f6"}
Jul 07 17:07:51.883155 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:51.883118    2584 generic.go:356] "Generic (PLEG): container finished" podID="ffd72431-887a-44b9-b39e-10f4bcae69be" containerID="3f730e67ea2ae606ed4362e30ea130a35d8df4206f39863ce6b43f73d1b5329d" exitCode=0
Jul 07 17:07:51.883615 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:51.883202    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h" event={"ID":"ffd72431-887a-44b9-b39e-10f4bcae69be","Type":"ContainerDied","Data":"3f730e67ea2ae606ed4362e30ea130a35d8df4206f39863ce6b43f73d1b5329d"}
Jul 07 17:07:52.890249 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:52.890214    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h" event={"ID":"ffd72431-887a-44b9-b39e-10f4bcae69be","Type":"ContainerStarted","Data":"8aeebb39a44350e263f32ac5b316b288184669d956e4d9a02267acd028123d31"}
Jul 07 17:07:52.890648 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:52.890280    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h" event={"ID":"ffd72431-887a-44b9-b39e-10f4bcae69be","Type":"ContainerStarted","Data":"b0f229b87c4702c891cfc3ba1451783da5b259c9f9c606c02e676d1f36951a2d"}
Jul 07 17:07:52.890648 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:52.890484    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h"
Jul 07 17:07:52.924084 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:52.924001    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h" podStartSLOduration=3.647520412 podStartE2EDuration="32.923979307s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.276458895s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:07:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:07:52.921506457 +0000 UTC m=+1321.522715165" watchObservedRunningTime="2026-07-07 17:07:52.923979307 +0000 UTC m=+1321.525188014"
Jul 07 17:07:53.894608 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:53.894576    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h"
Jul 07 17:07:59.903613 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:07:59.903586    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h"
Jul 07 17:09:00.170455 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:00.170409    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 17:09:24.892245 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:24.892203    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h"
Jul 07 17:09:30.971294 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:30.971262    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h"]
Jul 07 17:09:30.971741 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:30.971508    2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h" podUID="ffd72431-887a-44b9-b39e-10f4bcae69be" containerName="kserve-container" containerID="cri-o://b0f229b87c4702c891cfc3ba1451783da5b259c9f9c606c02e676d1f36951a2d" gracePeriod=30
Jul 07 17:09:30.971741 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:30.971541    2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h" podUID="ffd72431-887a-44b9-b39e-10f4bcae69be" containerName="kube-rbac-proxy" containerID="cri-o://8aeebb39a44350e263f32ac5b316b288184669d956e4d9a02267acd028123d31" gracePeriod=30
Jul 07 17:09:31.001590 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:31.001555    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 17:09:31.315261 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:31.315167    2584 generic.go:356] "Generic (PLEG): container finished" podID="ffd72431-887a-44b9-b39e-10f4bcae69be" containerID="8aeebb39a44350e263f32ac5b316b288184669d956e4d9a02267acd028123d31" exitCode=2
Jul 07 17:09:31.315261 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:31.315232    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h" event={"ID":"ffd72431-887a-44b9-b39e-10f4bcae69be","Type":"ContainerDied","Data":"8aeebb39a44350e263f32ac5b316b288184669d956e4d9a02267acd028123d31"}
Jul 07 17:09:31.916461 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:31.916433    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h"
Jul 07 17:09:32.038254 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:32.038148    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffd72431-887a-44b9-b39e-10f4bcae69be-kube-rbac-proxy-sar-config\") pod \"ffd72431-887a-44b9-b39e-10f4bcae69be\" (UID: \"ffd72431-887a-44b9-b39e-10f4bcae69be\") "
Jul 07 17:09:32.038254 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:32.038211    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dslz\" (UniqueName: \"kubernetes.io/projected/ffd72431-887a-44b9-b39e-10f4bcae69be-kube-api-access-6dslz\") pod \"ffd72431-887a-44b9-b39e-10f4bcae69be\" (UID: \"ffd72431-887a-44b9-b39e-10f4bcae69be\") "
Jul 07 17:09:32.038254 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:32.038243    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffd72431-887a-44b9-b39e-10f4bcae69be-cabundle-cert\") pod \"ffd72431-887a-44b9-b39e-10f4bcae69be\" (UID: \"ffd72431-887a-44b9-b39e-10f4bcae69be\") "
Jul 07 17:09:32.038848 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:32.038274    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffd72431-887a-44b9-b39e-10f4bcae69be-kserve-provision-location\") pod \"ffd72431-887a-44b9-b39e-10f4bcae69be\" (UID: \"ffd72431-887a-44b9-b39e-10f4bcae69be\") "
Jul 07 17:09:32.038848 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:32.038337    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffd72431-887a-44b9-b39e-10f4bcae69be-proxy-tls\") pod \"ffd72431-887a-44b9-b39e-10f4bcae69be\" (UID: \"ffd72431-887a-44b9-b39e-10f4bcae69be\") "
Jul 07 17:09:32.038848 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:32.038715    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffd72431-887a-44b9-b39e-10f4bcae69be-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ffd72431-887a-44b9-b39e-10f4bcae69be" (UID: "ffd72431-887a-44b9-b39e-10f4bcae69be"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:09:32.038848 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:32.038729    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffd72431-887a-44b9-b39e-10f4bcae69be-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ffd72431-887a-44b9-b39e-10f4bcae69be" (UID: "ffd72431-887a-44b9-b39e-10f4bcae69be"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:09:32.040586 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:32.040553    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd72431-887a-44b9-b39e-10f4bcae69be-kube-api-access-6dslz" (OuterVolumeSpecName: "kube-api-access-6dslz") pod "ffd72431-887a-44b9-b39e-10f4bcae69be" (UID: "ffd72431-887a-44b9-b39e-10f4bcae69be"). InnerVolumeSpecName "kube-api-access-6dslz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:09:32.040687 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:32.040658    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffd72431-887a-44b9-b39e-10f4bcae69be-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ffd72431-887a-44b9-b39e-10f4bcae69be" (UID: "ffd72431-887a-44b9-b39e-10f4bcae69be"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:09:32.043391 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:32.043370    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd72431-887a-44b9-b39e-10f4bcae69be-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ffd72431-887a-44b9-b39e-10f4bcae69be" (UID: "ffd72431-887a-44b9-b39e-10f4bcae69be"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:09:32.139884 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:32.139827    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffd72431-887a-44b9-b39e-10f4bcae69be-kserve-provision-location\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:09:32.139884 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:32.139874    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffd72431-887a-44b9-b39e-10f4bcae69be-proxy-tls\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:09:32.139884 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:32.139883    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffd72431-887a-44b9-b39e-10f4bcae69be-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:09:32.139884 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:32.139892    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dslz\" (UniqueName: \"kubernetes.io/projected/ffd72431-887a-44b9-b39e-10f4bcae69be-kube-api-access-6dslz\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:09:32.139884 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:32.139904    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffd72431-887a-44b9-b39e-10f4bcae69be-cabundle-cert\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:09:32.321436 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:32.321351    2584 generic.go:356] "Generic (PLEG): container finished" podID="ffd72431-887a-44b9-b39e-10f4bcae69be" containerID="b0f229b87c4702c891cfc3ba1451783da5b259c9f9c606c02e676d1f36951a2d" exitCode=0
Jul 07 17:09:32.321436 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:32.321394    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h" event={"ID":"ffd72431-887a-44b9-b39e-10f4bcae69be","Type":"ContainerDied","Data":"b0f229b87c4702c891cfc3ba1451783da5b259c9f9c606c02e676d1f36951a2d"}
Jul 07 17:09:32.321436 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:32.321412    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h"
Jul 07 17:09:32.321651 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:32.321455    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h" event={"ID":"ffd72431-887a-44b9-b39e-10f4bcae69be","Type":"ContainerDied","Data":"d7b7bae5f74ab435efd587922f0b489b494fdfbe6d624211d1f6acc1e97e36f6"}
Jul 07 17:09:32.321651 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:32.321492    2584 scope.go:117] "RemoveContainer" containerID="8aeebb39a44350e263f32ac5b316b288184669d956e4d9a02267acd028123d31"
Jul 07 17:09:32.337384 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:32.337364    2584 scope.go:117] "RemoveContainer" containerID="b0f229b87c4702c891cfc3ba1451783da5b259c9f9c606c02e676d1f36951a2d"
Jul 07 17:09:32.345087 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:32.345031    2584 scope.go:117] "RemoveContainer" containerID="3f730e67ea2ae606ed4362e30ea130a35d8df4206f39863ce6b43f73d1b5329d"
Jul 07 17:09:32.350296 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:32.350263    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h"]
Jul 07 17:09:32.353232 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:32.353205    2584 scope.go:117] "RemoveContainer" containerID="8aeebb39a44350e263f32ac5b316b288184669d956e4d9a02267acd028123d31"
Jul 07 17:09:32.353510 ip-10-0-133-184 kubenswrapper[2584]: E0707 17:09:32.353491    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aeebb39a44350e263f32ac5b316b288184669d956e4d9a02267acd028123d31\": container with ID starting with 8aeebb39a44350e263f32ac5b316b288184669d956e4d9a02267acd028123d31 not found: ID does not exist" containerID="8aeebb39a44350e263f32ac5b316b288184669d956e4d9a02267acd028123d31"
Jul 07 17:09:32.353563 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:32.353520    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aeebb39a44350e263f32ac5b316b288184669d956e4d9a02267acd028123d31"} err="failed to get container status \"8aeebb39a44350e263f32ac5b316b288184669d956e4d9a02267acd028123d31\": rpc error: code = NotFound desc = could not find container \"8aeebb39a44350e263f32ac5b316b288184669d956e4d9a02267acd028123d31\": container with ID starting with 8aeebb39a44350e263f32ac5b316b288184669d956e4d9a02267acd028123d31 not found: ID does not exist"
Jul 07 17:09:32.353563 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:32.353539    2584 scope.go:117] "RemoveContainer" containerID="b0f229b87c4702c891cfc3ba1451783da5b259c9f9c606c02e676d1f36951a2d"
Jul 07 17:09:32.353805 ip-10-0-133-184 kubenswrapper[2584]: E0707 17:09:32.353785    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0f229b87c4702c891cfc3ba1451783da5b259c9f9c606c02e676d1f36951a2d\": container with ID starting with b0f229b87c4702c891cfc3ba1451783da5b259c9f9c606c02e676d1f36951a2d not found: ID does not exist" containerID="b0f229b87c4702c891cfc3ba1451783da5b259c9f9c606c02e676d1f36951a2d"
Jul 07 17:09:32.353873 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:32.353815    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0f229b87c4702c891cfc3ba1451783da5b259c9f9c606c02e676d1f36951a2d"} err="failed to get container status \"b0f229b87c4702c891cfc3ba1451783da5b259c9f9c606c02e676d1f36951a2d\": rpc error: code = NotFound desc = could not find container \"b0f229b87c4702c891cfc3ba1451783da5b259c9f9c606c02e676d1f36951a2d\": container with ID starting with b0f229b87c4702c891cfc3ba1451783da5b259c9f9c606c02e676d1f36951a2d not found: ID does not exist"
Jul 07 17:09:32.353873 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:32.353837    2584 scope.go:117] "RemoveContainer" containerID="3f730e67ea2ae606ed4362e30ea130a35d8df4206f39863ce6b43f73d1b5329d"
Jul 07 17:09:32.354266 ip-10-0-133-184 kubenswrapper[2584]: E0707 17:09:32.354232    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f730e67ea2ae606ed4362e30ea130a35d8df4206f39863ce6b43f73d1b5329d\": container with ID starting with 3f730e67ea2ae606ed4362e30ea130a35d8df4206f39863ce6b43f73d1b5329d not found: ID does not exist" containerID="3f730e67ea2ae606ed4362e30ea130a35d8df4206f39863ce6b43f73d1b5329d"
Jul 07 17:09:32.354366 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:32.354260    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f730e67ea2ae606ed4362e30ea130a35d8df4206f39863ce6b43f73d1b5329d"} err="failed to get container status \"3f730e67ea2ae606ed4362e30ea130a35d8df4206f39863ce6b43f73d1b5329d\": rpc error: code = NotFound desc = could not find container \"3f730e67ea2ae606ed4362e30ea130a35d8df4206f39863ce6b43f73d1b5329d\": container with ID starting with 3f730e67ea2ae606ed4362e30ea130a35d8df4206f39863ce6b43f73d1b5329d not found: ID does not exist"
Jul 07 17:09:32.356711 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:32.356692    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-t5x6h"]
Jul 07 17:09:33.893504 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:33.893470    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffd72431-887a-44b9-b39e-10f4bcae69be" path="/var/lib/kubelet/pods/ffd72431-887a-44b9-b39e-10f4bcae69be/volumes"
Jul 07 17:09:36.029603 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.029555    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 17:09:36.096106 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.096069    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw"]
Jul 07 17:09:36.096574 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.096560    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffd72431-887a-44b9-b39e-10f4bcae69be" containerName="kube-rbac-proxy"
Jul 07 17:09:36.096617 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.096578    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffd72431-887a-44b9-b39e-10f4bcae69be" containerName="kube-rbac-proxy"
Jul 07 17:09:36.096617 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.096598    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffd72431-887a-44b9-b39e-10f4bcae69be" containerName="storage-initializer"
Jul 07 17:09:36.096617 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.096606    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffd72431-887a-44b9-b39e-10f4bcae69be" containerName="storage-initializer"
Jul 07 17:09:36.096707 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.096633    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffd72431-887a-44b9-b39e-10f4bcae69be" containerName="kserve-container"
Jul 07 17:09:36.096707 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.096641    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffd72431-887a-44b9-b39e-10f4bcae69be" containerName="kserve-container"
Jul 07 17:09:36.096770 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.096727    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffd72431-887a-44b9-b39e-10f4bcae69be" containerName="kube-rbac-proxy"
Jul 07 17:09:36.096770 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.096739    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffd72431-887a-44b9-b39e-10f4bcae69be" containerName="kserve-container"
Jul 07 17:09:36.102278 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.102256    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw"
Jul 07 17:09:36.105619 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.105592    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 17:09:36.105775 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.105598    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 07 17:09:36.105775 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.105640    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 17:09:36.107218 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.107190    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 17:09:36.107351 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.107245    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7526s\""
Jul 07 17:09:36.107351 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.107260    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 07 17:09:36.107351 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.107246    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 17:09:36.111553 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.111533    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw"]
Jul 07 17:09:36.224398 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.224363    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 17:09:36.276072 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.276010    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3623d86-926a-4b00-a6bf-01c8d1d21dcf-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw\" (UID: \"b3623d86-926a-4b00-a6bf-01c8d1d21dcf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw"
Jul 07 17:09:36.276243 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.276096    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3623d86-926a-4b00-a6bf-01c8d1d21dcf-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw\" (UID: \"b3623d86-926a-4b00-a6bf-01c8d1d21dcf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw"
Jul 07 17:09:36.276243 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.276139    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfh7p\" (UniqueName: \"kubernetes.io/projected/b3623d86-926a-4b00-a6bf-01c8d1d21dcf-kube-api-access-jfh7p\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw\" (UID: \"b3623d86-926a-4b00-a6bf-01c8d1d21dcf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw"
Jul 07 17:09:36.276243 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.276158    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3623d86-926a-4b00-a6bf-01c8d1d21dcf-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw\" (UID: \"b3623d86-926a-4b00-a6bf-01c8d1d21dcf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw"
Jul 07 17:09:36.276243 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.276200    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3623d86-926a-4b00-a6bf-01c8d1d21dcf-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw\" (UID: \"b3623d86-926a-4b00-a6bf-01c8d1d21dcf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw"
Jul 07 17:09:36.376988 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.376945    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3623d86-926a-4b00-a6bf-01c8d1d21dcf-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw\" (UID: \"b3623d86-926a-4b00-a6bf-01c8d1d21dcf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw"
Jul 07 17:09:36.377241 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.377014    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfh7p\" (UniqueName: \"kubernetes.io/projected/b3623d86-926a-4b00-a6bf-01c8d1d21dcf-kube-api-access-jfh7p\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw\" (UID: \"b3623d86-926a-4b00-a6bf-01c8d1d21dcf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw"
Jul 07 17:09:36.377241 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.377033    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3623d86-926a-4b00-a6bf-01c8d1d21dcf-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw\" (UID: \"b3623d86-926a-4b00-a6bf-01c8d1d21dcf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw"
Jul 07 17:09:36.377241 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.377076    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3623d86-926a-4b00-a6bf-01c8d1d21dcf-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw\" (UID: \"b3623d86-926a-4b00-a6bf-01c8d1d21dcf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw"
Jul 07 17:09:36.377241 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.377109    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3623d86-926a-4b00-a6bf-01c8d1d21dcf-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw\" (UID: \"b3623d86-926a-4b00-a6bf-01c8d1d21dcf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw"
Jul 07 17:09:36.377524 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.377497    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3623d86-926a-4b00-a6bf-01c8d1d21dcf-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw\" (UID: \"b3623d86-926a-4b00-a6bf-01c8d1d21dcf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw"
Jul 07 17:09:36.377733 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.377700    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3623d86-926a-4b00-a6bf-01c8d1d21dcf-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw\" (UID: \"b3623d86-926a-4b00-a6bf-01c8d1d21dcf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw"
Jul 07 17:09:36.377848 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.377757    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3623d86-926a-4b00-a6bf-01c8d1d21dcf-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw\" (UID: \"b3623d86-926a-4b00-a6bf-01c8d1d21dcf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw"
Jul 07 17:09:36.379678 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.379662    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3623d86-926a-4b00-a6bf-01c8d1d21dcf-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw\" (UID: \"b3623d86-926a-4b00-a6bf-01c8d1d21dcf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw"
Jul 07 17:09:36.388303 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.388279    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfh7p\" (UniqueName: \"kubernetes.io/projected/b3623d86-926a-4b00-a6bf-01c8d1d21dcf-kube-api-access-jfh7p\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw\" (UID: \"b3623d86-926a-4b00-a6bf-01c8d1d21dcf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw"
Jul 07 17:09:36.414162 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.414134    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw"
Jul 07 17:09:36.550946 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:36.550920    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw"]
Jul 07 17:09:36.552892 ip-10-0-133-184 kubenswrapper[2584]: W0707 17:09:36.552865    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3623d86_926a_4b00_a6bf_01c8d1d21dcf.slice/crio-c559248d4526539579c8aa92705803f47584951a8ae4599c0863c6e8d5971f46 WatchSource:0}: Error finding container c559248d4526539579c8aa92705803f47584951a8ae4599c0863c6e8d5971f46: Status 404 returned error can't find the container with id c559248d4526539579c8aa92705803f47584951a8ae4599c0863c6e8d5971f46
Jul 07 17:09:37.348749 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:37.348712    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw" event={"ID":"b3623d86-926a-4b00-a6bf-01c8d1d21dcf","Type":"ContainerStarted","Data":"390b1692798624ac90ecbc10ec516990a332f871d4db630514f2091901f9f686"}
Jul 07 17:09:37.349238 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:37.348803    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw" event={"ID":"b3623d86-926a-4b00-a6bf-01c8d1d21dcf","Type":"ContainerStarted","Data":"c559248d4526539579c8aa92705803f47584951a8ae4599c0863c6e8d5971f46"}
Jul 07 17:09:42.373542 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:42.373504    2584 generic.go:356] "Generic (PLEG): container finished" podID="b3623d86-926a-4b00-a6bf-01c8d1d21dcf" containerID="390b1692798624ac90ecbc10ec516990a332f871d4db630514f2091901f9f686" exitCode=0
Jul 07 17:09:42.373929 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:42.373580    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw" event={"ID":"b3623d86-926a-4b00-a6bf-01c8d1d21dcf","Type":"ContainerDied","Data":"390b1692798624ac90ecbc10ec516990a332f871d4db630514f2091901f9f686"}
Jul 07 17:09:43.380520 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:43.380489    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw" event={"ID":"b3623d86-926a-4b00-a6bf-01c8d1d21dcf","Type":"ContainerStarted","Data":"b8c8570a4b898e5337572340a7925c3f8eb1b892e9869dc9593553d907acff69"}
Jul 07 17:09:43.380917 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:43.380548    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw" event={"ID":"b3623d86-926a-4b00-a6bf-01c8d1d21dcf","Type":"ContainerStarted","Data":"c97ef9e4b3e6480ff869b12ceed84240fad246d5f46e3300cadb4b7a5948c58f"}
Jul 07 17:09:43.380917 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:43.380735    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw"
Jul 07 17:09:43.405733 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:43.405681    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw" podStartSLOduration=2.432013241 podStartE2EDuration="7.405665085s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.973651844s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:09:43.403781098 +0000 UTC m=+1432.004989801" watchObservedRunningTime="2026-07-07 17:09:43.405665085 +0000 UTC m=+1432.006873788"
Jul 07 17:09:44.388409 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:44.388379    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw"
Jul 07 17:09:50.399889 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:09:50.399857    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw"
Jul 07 17:10:43.731418 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:10:43.731382    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 17:11:21.895430 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:21.895338    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw"
Jul 07 17:11:26.213008 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:26.212975    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw"]
Jul 07 17:11:26.213467 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:26.213262    2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw" podUID="b3623d86-926a-4b00-a6bf-01c8d1d21dcf" containerName="kserve-container" containerID="cri-o://c97ef9e4b3e6480ff869b12ceed84240fad246d5f46e3300cadb4b7a5948c58f" gracePeriod=30
Jul 07 17:11:26.213467 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:26.213310    2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw" podUID="b3623d86-926a-4b00-a6bf-01c8d1d21dcf" containerName="kube-rbac-proxy" containerID="cri-o://b8c8570a4b898e5337572340a7925c3f8eb1b892e9869dc9593553d907acff69" gracePeriod=30
Jul 07 17:11:26.237678 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:26.237648    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 17:11:26.826712 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:26.826679    2584 generic.go:356] "Generic (PLEG): container finished" podID="b3623d86-926a-4b00-a6bf-01c8d1d21dcf" containerID="b8c8570a4b898e5337572340a7925c3f8eb1b892e9869dc9593553d907acff69" exitCode=2
Jul 07 17:11:26.826892 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:26.826760    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw" event={"ID":"b3623d86-926a-4b00-a6bf-01c8d1d21dcf","Type":"ContainerDied","Data":"b8c8570a4b898e5337572340a7925c3f8eb1b892e9869dc9593553d907acff69"}
Jul 07 17:11:30.394159 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:30.394117    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw" podUID="b3623d86-926a-4b00-a6bf-01c8d1d21dcf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Jul 07 17:11:31.064900 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.064876    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw"
Jul 07 17:11:31.178474 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.178374    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3623d86-926a-4b00-a6bf-01c8d1d21dcf-proxy-tls\") pod \"b3623d86-926a-4b00-a6bf-01c8d1d21dcf\" (UID: \"b3623d86-926a-4b00-a6bf-01c8d1d21dcf\") "
Jul 07 17:11:31.178474 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.178419    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfh7p\" (UniqueName: \"kubernetes.io/projected/b3623d86-926a-4b00-a6bf-01c8d1d21dcf-kube-api-access-jfh7p\") pod \"b3623d86-926a-4b00-a6bf-01c8d1d21dcf\" (UID: \"b3623d86-926a-4b00-a6bf-01c8d1d21dcf\") "
Jul 07 17:11:31.178474 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.178443    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3623d86-926a-4b00-a6bf-01c8d1d21dcf-cabundle-cert\") pod \"b3623d86-926a-4b00-a6bf-01c8d1d21dcf\" (UID: \"b3623d86-926a-4b00-a6bf-01c8d1d21dcf\") "
Jul 07 17:11:31.178474 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.178469    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3623d86-926a-4b00-a6bf-01c8d1d21dcf-kube-rbac-proxy-sar-config\") pod \"b3623d86-926a-4b00-a6bf-01c8d1d21dcf\" (UID: \"b3623d86-926a-4b00-a6bf-01c8d1d21dcf\") "
Jul 07 17:11:31.178897 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.178573    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3623d86-926a-4b00-a6bf-01c8d1d21dcf-kserve-provision-location\") pod \"b3623d86-926a-4b00-a6bf-01c8d1d21dcf\" (UID: \"b3623d86-926a-4b00-a6bf-01c8d1d21dcf\") "
Jul 07 17:11:31.178897 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.178840    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3623d86-926a-4b00-a6bf-01c8d1d21dcf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b3623d86-926a-4b00-a6bf-01c8d1d21dcf" (UID: "b3623d86-926a-4b00-a6bf-01c8d1d21dcf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:11:31.178897 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.178870    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3623d86-926a-4b00-a6bf-01c8d1d21dcf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b3623d86-926a-4b00-a6bf-01c8d1d21dcf" (UID: "b3623d86-926a-4b00-a6bf-01c8d1d21dcf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:11:31.181375 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.181345    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3623d86-926a-4b00-a6bf-01c8d1d21dcf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b3623d86-926a-4b00-a6bf-01c8d1d21dcf" (UID: "b3623d86-926a-4b00-a6bf-01c8d1d21dcf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:11:31.181481 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.181387    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3623d86-926a-4b00-a6bf-01c8d1d21dcf-kube-api-access-jfh7p" (OuterVolumeSpecName: "kube-api-access-jfh7p") pod "b3623d86-926a-4b00-a6bf-01c8d1d21dcf" (UID: "b3623d86-926a-4b00-a6bf-01c8d1d21dcf"). InnerVolumeSpecName "kube-api-access-jfh7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:11:31.183138 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.183113    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3623d86-926a-4b00-a6bf-01c8d1d21dcf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b3623d86-926a-4b00-a6bf-01c8d1d21dcf" (UID: "b3623d86-926a-4b00-a6bf-01c8d1d21dcf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:11:31.279806 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.279766    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3623d86-926a-4b00-a6bf-01c8d1d21dcf-proxy-tls\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:11:31.279806 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.279800    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfh7p\" (UniqueName: \"kubernetes.io/projected/b3623d86-926a-4b00-a6bf-01c8d1d21dcf-kube-api-access-jfh7p\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:11:31.279806 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.279812    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3623d86-926a-4b00-a6bf-01c8d1d21dcf-cabundle-cert\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:11:31.280075 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.279821    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3623d86-926a-4b00-a6bf-01c8d1d21dcf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:11:31.280075 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.279830    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3623d86-926a-4b00-a6bf-01c8d1d21dcf-kserve-provision-location\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:11:31.418495 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.418460    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s"]
Jul 07 17:11:31.418888 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.418794    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3623d86-926a-4b00-a6bf-01c8d1d21dcf" containerName="kserve-container"
Jul 07 17:11:31.418888 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.418805    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3623d86-926a-4b00-a6bf-01c8d1d21dcf" containerName="kserve-container"
Jul 07 17:11:31.418888 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.418813    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3623d86-926a-4b00-a6bf-01c8d1d21dcf" containerName="storage-initializer"
Jul 07 17:11:31.418888 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.418819    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3623d86-926a-4b00-a6bf-01c8d1d21dcf" containerName="storage-initializer"
Jul 07 17:11:31.418888 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.418826    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3623d86-926a-4b00-a6bf-01c8d1d21dcf" containerName="kube-rbac-proxy"
Jul 07 17:11:31.418888 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.418832    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3623d86-926a-4b00-a6bf-01c8d1d21dcf" containerName="kube-rbac-proxy"
Jul 07 17:11:31.418888 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.418885    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3623d86-926a-4b00-a6bf-01c8d1d21dcf" containerName="kube-rbac-proxy"
Jul 07 17:11:31.419130 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.418898    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3623d86-926a-4b00-a6bf-01c8d1d21dcf" containerName="kserve-container"
Jul 07 17:11:31.423771 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.423754    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s"
Jul 07 17:11:31.426898 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.426871    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 07 17:11:31.427079 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.427017    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 07 17:11:31.438390 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.438357    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s"]
Jul 07 17:11:31.582922 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.582889    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44b0a47a-2d88-4e7b-ad9e-153639de4bdd-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s\" (UID: \"44b0a47a-2d88-4e7b-ad9e-153639de4bdd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s"
Jul 07 17:11:31.583143 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.582944    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44b0a47a-2d88-4e7b-ad9e-153639de4bdd-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s\" (UID: \"44b0a47a-2d88-4e7b-ad9e-153639de4bdd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s"
Jul 07 17:11:31.583143 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.583022    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j7wq\" (UniqueName: \"kubernetes.io/projected/44b0a47a-2d88-4e7b-ad9e-153639de4bdd-kube-api-access-8j7wq\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s\" (UID: \"44b0a47a-2d88-4e7b-ad9e-153639de4bdd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s"
Jul 07 17:11:31.583143 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.583087    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44b0a47a-2d88-4e7b-ad9e-153639de4bdd-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s\" (UID: \"44b0a47a-2d88-4e7b-ad9e-153639de4bdd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s"
Jul 07 17:11:31.583268 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.583172    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44b0a47a-2d88-4e7b-ad9e-153639de4bdd-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s\" (UID: \"44b0a47a-2d88-4e7b-ad9e-153639de4bdd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s"
Jul 07 17:11:31.684304 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.684220    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44b0a47a-2d88-4e7b-ad9e-153639de4bdd-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s\" (UID: \"44b0a47a-2d88-4e7b-ad9e-153639de4bdd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s"
Jul 07 17:11:31.684304 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.684280    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44b0a47a-2d88-4e7b-ad9e-153639de4bdd-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s\" (UID: \"44b0a47a-2d88-4e7b-ad9e-153639de4bdd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s"
Jul 07 17:11:31.684496 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.684310    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44b0a47a-2d88-4e7b-ad9e-153639de4bdd-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s\" (UID: \"44b0a47a-2d88-4e7b-ad9e-153639de4bdd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s"
Jul 07 17:11:31.684496 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.684353    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44b0a47a-2d88-4e7b-ad9e-153639de4bdd-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s\" (UID: \"44b0a47a-2d88-4e7b-ad9e-153639de4bdd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s"
Jul 07 17:11:31.684496 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.684379    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j7wq\" (UniqueName: \"kubernetes.io/projected/44b0a47a-2d88-4e7b-ad9e-153639de4bdd-kube-api-access-8j7wq\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s\" (UID: \"44b0a47a-2d88-4e7b-ad9e-153639de4bdd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s"
Jul 07 17:11:31.684496 ip-10-0-133-184 kubenswrapper[2584]: E0707 17:11:31.684421    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-serving-cert: secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Jul 07 17:11:31.684636 ip-10-0-133-184 kubenswrapper[2584]: E0707 17:11:31.684502    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44b0a47a-2d88-4e7b-ad9e-153639de4bdd-proxy-tls podName:44b0a47a-2d88-4e7b-ad9e-153639de4bdd nodeName:}" failed. No retries permitted until 2026-07-07 17:11:32.184480933 +0000 UTC m=+1540.785689618 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/44b0a47a-2d88-4e7b-ad9e-153639de4bdd-proxy-tls") pod "isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s" (UID: "44b0a47a-2d88-4e7b-ad9e-153639de4bdd") : secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Jul 07 17:11:31.684717 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.684683    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44b0a47a-2d88-4e7b-ad9e-153639de4bdd-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s\" (UID: \"44b0a47a-2d88-4e7b-ad9e-153639de4bdd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s"
Jul 07 17:11:31.685107 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.685086    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44b0a47a-2d88-4e7b-ad9e-153639de4bdd-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s\" (UID: \"44b0a47a-2d88-4e7b-ad9e-153639de4bdd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s"
Jul 07 17:11:31.685192 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.685091    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44b0a47a-2d88-4e7b-ad9e-153639de4bdd-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s\" (UID: \"44b0a47a-2d88-4e7b-ad9e-153639de4bdd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s"
Jul 07 17:11:31.694147 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.694125    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j7wq\" (UniqueName: \"kubernetes.io/projected/44b0a47a-2d88-4e7b-ad9e-153639de4bdd-kube-api-access-8j7wq\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s\" (UID: \"44b0a47a-2d88-4e7b-ad9e-153639de4bdd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s"
Jul 07 17:11:31.850262 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.850222    2584 generic.go:356] "Generic (PLEG): container finished" podID="b3623d86-926a-4b00-a6bf-01c8d1d21dcf" containerID="c97ef9e4b3e6480ff869b12ceed84240fad246d5f46e3300cadb4b7a5948c58f" exitCode=0
Jul 07 17:11:31.850467 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.850277    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw" event={"ID":"b3623d86-926a-4b00-a6bf-01c8d1d21dcf","Type":"ContainerDied","Data":"c97ef9e4b3e6480ff869b12ceed84240fad246d5f46e3300cadb4b7a5948c58f"}
Jul 07 17:11:31.850467 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.850304    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw"
Jul 07 17:11:31.850467 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.850361    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw" event={"ID":"b3623d86-926a-4b00-a6bf-01c8d1d21dcf","Type":"ContainerDied","Data":"c559248d4526539579c8aa92705803f47584951a8ae4599c0863c6e8d5971f46"}
Jul 07 17:11:31.850467 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.850389    2584 scope.go:117] "RemoveContainer" containerID="b8c8570a4b898e5337572340a7925c3f8eb1b892e9869dc9593553d907acff69"
Jul 07 17:11:31.860451 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.860434    2584 scope.go:117] "RemoveContainer" containerID="c97ef9e4b3e6480ff869b12ceed84240fad246d5f46e3300cadb4b7a5948c58f"
Jul 07 17:11:31.871088 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.871034    2584 scope.go:117] "RemoveContainer" containerID="390b1692798624ac90ecbc10ec516990a332f871d4db630514f2091901f9f686"
Jul 07 17:11:31.875399 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.875371    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw"]
Jul 07 17:11:31.881470 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.881443    2584 scope.go:117] "RemoveContainer" containerID="b8c8570a4b898e5337572340a7925c3f8eb1b892e9869dc9593553d907acff69"
Jul 07 17:11:31.881763 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.881735    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-chjcw"]
Jul 07 17:11:31.881849 ip-10-0-133-184 kubenswrapper[2584]: E0707 17:11:31.881788    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8c8570a4b898e5337572340a7925c3f8eb1b892e9869dc9593553d907acff69\": container with ID starting with b8c8570a4b898e5337572340a7925c3f8eb1b892e9869dc9593553d907acff69 not found: ID does not exist" containerID="b8c8570a4b898e5337572340a7925c3f8eb1b892e9869dc9593553d907acff69"
Jul 07 17:11:31.881849 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.881813    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8c8570a4b898e5337572340a7925c3f8eb1b892e9869dc9593553d907acff69"} err="failed to get container status \"b8c8570a4b898e5337572340a7925c3f8eb1b892e9869dc9593553d907acff69\": rpc error: code = NotFound desc = could not find container \"b8c8570a4b898e5337572340a7925c3f8eb1b892e9869dc9593553d907acff69\": container with ID starting with b8c8570a4b898e5337572340a7925c3f8eb1b892e9869dc9593553d907acff69 not found: ID does not exist"
Jul 07 17:11:31.881849 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.881833    2584 scope.go:117] "RemoveContainer" containerID="c97ef9e4b3e6480ff869b12ceed84240fad246d5f46e3300cadb4b7a5948c58f"
Jul 07 17:11:31.882154 ip-10-0-133-184 kubenswrapper[2584]: E0707 17:11:31.882127    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c97ef9e4b3e6480ff869b12ceed84240fad246d5f46e3300cadb4b7a5948c58f\": container with ID starting with c97ef9e4b3e6480ff869b12ceed84240fad246d5f46e3300cadb4b7a5948c58f not found: ID does not exist" containerID="c97ef9e4b3e6480ff869b12ceed84240fad246d5f46e3300cadb4b7a5948c58f"
Jul 07 17:11:31.882233 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.882163    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c97ef9e4b3e6480ff869b12ceed84240fad246d5f46e3300cadb4b7a5948c58f"} err="failed to get container status \"c97ef9e4b3e6480ff869b12ceed84240fad246d5f46e3300cadb4b7a5948c58f\": rpc error: code = NotFound desc = could not find container \"c97ef9e4b3e6480ff869b12ceed84240fad246d5f46e3300cadb4b7a5948c58f\": container with ID starting with c97ef9e4b3e6480ff869b12ceed84240fad246d5f46e3300cadb4b7a5948c58f not found: ID does not exist"
Jul 07 17:11:31.882233 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.882185    2584 scope.go:117] "RemoveContainer" containerID="390b1692798624ac90ecbc10ec516990a332f871d4db630514f2091901f9f686"
Jul 07 17:11:31.882436 ip-10-0-133-184 kubenswrapper[2584]: E0707 17:11:31.882413    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"390b1692798624ac90ecbc10ec516990a332f871d4db630514f2091901f9f686\": container with ID starting with 390b1692798624ac90ecbc10ec516990a332f871d4db630514f2091901f9f686 not found: ID does not exist" containerID="390b1692798624ac90ecbc10ec516990a332f871d4db630514f2091901f9f686"
Jul 07 17:11:31.882520 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.882444    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"390b1692798624ac90ecbc10ec516990a332f871d4db630514f2091901f9f686"} err="failed to get container status \"390b1692798624ac90ecbc10ec516990a332f871d4db630514f2091901f9f686\": rpc error: code = NotFound desc = could not find container \"390b1692798624ac90ecbc10ec516990a332f871d4db630514f2091901f9f686\": container with ID starting with 390b1692798624ac90ecbc10ec516990a332f871d4db630514f2091901f9f686 not found: ID does not exist"
Jul 07 17:11:31.893347 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:31.893322    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3623d86-926a-4b00-a6bf-01c8d1d21dcf" path="/var/lib/kubelet/pods/b3623d86-926a-4b00-a6bf-01c8d1d21dcf/volumes"
Jul 07 17:11:32.188032 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:32.187997    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44b0a47a-2d88-4e7b-ad9e-153639de4bdd-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s\" (UID: \"44b0a47a-2d88-4e7b-ad9e-153639de4bdd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s"
Jul 07 17:11:32.190587 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:32.190556    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44b0a47a-2d88-4e7b-ad9e-153639de4bdd-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s\" (UID: \"44b0a47a-2d88-4e7b-ad9e-153639de4bdd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s"
Jul 07 17:11:32.271571 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:32.271533    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 17:11:32.334090 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:32.334031    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s"
Jul 07 17:11:32.344698 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:32.344664    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 17:11:32.477538 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:32.477514    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s"]
Jul 07 17:11:32.479644 ip-10-0-133-184 kubenswrapper[2584]: W0707 17:11:32.479616    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b0a47a_2d88_4e7b_ad9e_153639de4bdd.slice/crio-a26dae7c1e5e2b4e214587aa1eb3f826945dff63098ee069090f9f1d329bde34 WatchSource:0}: Error finding container a26dae7c1e5e2b4e214587aa1eb3f826945dff63098ee069090f9f1d329bde34: Status 404 returned error can't find the container with id a26dae7c1e5e2b4e214587aa1eb3f826945dff63098ee069090f9f1d329bde34
Jul 07 17:11:32.858032 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:32.857988    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s" event={"ID":"44b0a47a-2d88-4e7b-ad9e-153639de4bdd","Type":"ContainerStarted","Data":"87451f0c2bd88c156802a297e0a2146da6ac946a7f36a512e93bd357152f9cd5"}
Jul 07 17:11:32.858226 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:32.858069    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s" event={"ID":"44b0a47a-2d88-4e7b-ad9e-153639de4bdd","Type":"ContainerStarted","Data":"a26dae7c1e5e2b4e214587aa1eb3f826945dff63098ee069090f9f1d329bde34"}
Jul 07 17:11:41.897346 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:41.897317    2584 generic.go:356] "Generic (PLEG): container finished" podID="44b0a47a-2d88-4e7b-ad9e-153639de4bdd" containerID="87451f0c2bd88c156802a297e0a2146da6ac946a7f36a512e93bd357152f9cd5" exitCode=0
Jul 07 17:11:41.897666 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:41.897393    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s" event={"ID":"44b0a47a-2d88-4e7b-ad9e-153639de4bdd","Type":"ContainerDied","Data":"87451f0c2bd88c156802a297e0a2146da6ac946a7f36a512e93bd357152f9cd5"}
Jul 07 17:11:42.903465 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:42.903421    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s" event={"ID":"44b0a47a-2d88-4e7b-ad9e-153639de4bdd","Type":"ContainerStarted","Data":"2c645af33b694ce1f9f14addae8cad741041d6342d35e459119c30a3e3cb08f6"}
Jul 07 17:11:42.903868 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:42.903509    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s" event={"ID":"44b0a47a-2d88-4e7b-ad9e-153639de4bdd","Type":"ContainerStarted","Data":"8eee9bb6cfc91873f1bf16820ca3024490911940245f88796f2378ea0020e3d7"}
Jul 07 17:11:42.903868 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:42.903733    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s"
Jul 07 17:11:42.925812 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:42.925757    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s" podStartSLOduration=2.598406942 podStartE2EDuration="11.925741356s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.327334414s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:11:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:11:42.923923668 +0000 UTC m=+1551.525132374" watchObservedRunningTime="2026-07-07 17:11:42.925741356 +0000 UTC m=+1551.526950060"
Jul 07 17:11:43.907593 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:43.907561    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s"
Jul 07 17:11:43.908872 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:43.908847    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s" podUID="44b0a47a-2d88-4e7b-ad9e-153639de4bdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 07 17:11:44.911973 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:44.911929    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s" podUID="44b0a47a-2d88-4e7b-ad9e-153639de4bdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 07 17:11:49.916636 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:49.916603    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s"
Jul 07 17:11:49.917276 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:49.917251    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s" podUID="44b0a47a-2d88-4e7b-ad9e-153639de4bdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 07 17:11:59.918019 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:11:59.917988    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s"
Jul 07 17:12:01.534350 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:01.534310    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s"]
Jul 07 17:12:01.534881 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:01.534596    2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s" podUID="44b0a47a-2d88-4e7b-ad9e-153639de4bdd" containerName="kserve-container" containerID="cri-o://8eee9bb6cfc91873f1bf16820ca3024490911940245f88796f2378ea0020e3d7" gracePeriod=30
Jul 07 17:12:01.534881 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:01.534639    2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s" podUID="44b0a47a-2d88-4e7b-ad9e-153639de4bdd" containerName="kube-rbac-proxy" containerID="cri-o://2c645af33b694ce1f9f14addae8cad741041d6342d35e459119c30a3e3cb08f6" gracePeriod=30
Jul 07 17:12:01.588690 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:01.588656    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 17:12:01.990806 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:01.990773    2584 generic.go:356] "Generic (PLEG): container finished" podID="44b0a47a-2d88-4e7b-ad9e-153639de4bdd" containerID="2c645af33b694ce1f9f14addae8cad741041d6342d35e459119c30a3e3cb08f6" exitCode=2
Jul 07 17:12:01.990984 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:01.990847    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s" event={"ID":"44b0a47a-2d88-4e7b-ad9e-153639de4bdd","Type":"ContainerDied","Data":"2c645af33b694ce1f9f14addae8cad741041d6342d35e459119c30a3e3cb08f6"}
Jul 07 17:12:02.482996 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:02.482967    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s"
Jul 07 17:12:02.540474 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:02.540435    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44b0a47a-2d88-4e7b-ad9e-153639de4bdd-kube-rbac-proxy-sar-config\") pod \"44b0a47a-2d88-4e7b-ad9e-153639de4bdd\" (UID: \"44b0a47a-2d88-4e7b-ad9e-153639de4bdd\") "
Jul 07 17:12:02.540474 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:02.540473    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8j7wq\" (UniqueName: \"kubernetes.io/projected/44b0a47a-2d88-4e7b-ad9e-153639de4bdd-kube-api-access-8j7wq\") pod \"44b0a47a-2d88-4e7b-ad9e-153639de4bdd\" (UID: \"44b0a47a-2d88-4e7b-ad9e-153639de4bdd\") "
Jul 07 17:12:02.541022 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:02.540502    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44b0a47a-2d88-4e7b-ad9e-153639de4bdd-cabundle-cert\") pod \"44b0a47a-2d88-4e7b-ad9e-153639de4bdd\" (UID: \"44b0a47a-2d88-4e7b-ad9e-153639de4bdd\") "
Jul 07 17:12:02.541022 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:02.540527    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44b0a47a-2d88-4e7b-ad9e-153639de4bdd-kserve-provision-location\") pod \"44b0a47a-2d88-4e7b-ad9e-153639de4bdd\" (UID: \"44b0a47a-2d88-4e7b-ad9e-153639de4bdd\") "
Jul 07 17:12:02.541022 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:02.540574    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44b0a47a-2d88-4e7b-ad9e-153639de4bdd-proxy-tls\") pod \"44b0a47a-2d88-4e7b-ad9e-153639de4bdd\" (UID: \"44b0a47a-2d88-4e7b-ad9e-153639de4bdd\") "
Jul 07 17:12:02.541022 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:02.540845    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44b0a47a-2d88-4e7b-ad9e-153639de4bdd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "44b0a47a-2d88-4e7b-ad9e-153639de4bdd" (UID: "44b0a47a-2d88-4e7b-ad9e-153639de4bdd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:12:02.541022 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:02.540913    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44b0a47a-2d88-4e7b-ad9e-153639de4bdd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "44b0a47a-2d88-4e7b-ad9e-153639de4bdd" (UID: "44b0a47a-2d88-4e7b-ad9e-153639de4bdd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:12:02.541707 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:02.541679    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b0a47a-2d88-4e7b-ad9e-153639de4bdd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "44b0a47a-2d88-4e7b-ad9e-153639de4bdd" (UID: "44b0a47a-2d88-4e7b-ad9e-153639de4bdd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:12:02.542785 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:02.542765    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44b0a47a-2d88-4e7b-ad9e-153639de4bdd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "44b0a47a-2d88-4e7b-ad9e-153639de4bdd" (UID: "44b0a47a-2d88-4e7b-ad9e-153639de4bdd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:12:02.542891 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:02.542876    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b0a47a-2d88-4e7b-ad9e-153639de4bdd-kube-api-access-8j7wq" (OuterVolumeSpecName: "kube-api-access-8j7wq") pod "44b0a47a-2d88-4e7b-ad9e-153639de4bdd" (UID: "44b0a47a-2d88-4e7b-ad9e-153639de4bdd"). InnerVolumeSpecName "kube-api-access-8j7wq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:12:02.641958 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:02.641910    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44b0a47a-2d88-4e7b-ad9e-153639de4bdd-proxy-tls\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:12:02.641958 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:02.641953    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44b0a47a-2d88-4e7b-ad9e-153639de4bdd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:12:02.641958 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:02.641963    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j7wq\" (UniqueName: \"kubernetes.io/projected/44b0a47a-2d88-4e7b-ad9e-153639de4bdd-kube-api-access-8j7wq\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:12:02.641958 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:02.641974    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44b0a47a-2d88-4e7b-ad9e-153639de4bdd-cabundle-cert\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:12:02.642260 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:02.641982    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44b0a47a-2d88-4e7b-ad9e-153639de4bdd-kserve-provision-location\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:12:02.996756 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:02.996668    2584 generic.go:356] "Generic (PLEG): container finished" podID="44b0a47a-2d88-4e7b-ad9e-153639de4bdd" containerID="8eee9bb6cfc91873f1bf16820ca3024490911940245f88796f2378ea0020e3d7" exitCode=0
Jul 07 17:12:02.996756 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:02.996736    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s"
Jul 07 17:12:02.996934 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:02.996751    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s" event={"ID":"44b0a47a-2d88-4e7b-ad9e-153639de4bdd","Type":"ContainerDied","Data":"8eee9bb6cfc91873f1bf16820ca3024490911940245f88796f2378ea0020e3d7"}
Jul 07 17:12:02.996934 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:02.996817    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s" event={"ID":"44b0a47a-2d88-4e7b-ad9e-153639de4bdd","Type":"ContainerDied","Data":"a26dae7c1e5e2b4e214587aa1eb3f826945dff63098ee069090f9f1d329bde34"}
Jul 07 17:12:02.996934 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:02.996845    2584 scope.go:117] "RemoveContainer" containerID="2c645af33b694ce1f9f14addae8cad741041d6342d35e459119c30a3e3cb08f6"
Jul 07 17:12:03.005751 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:03.005733    2584 scope.go:117] "RemoveContainer" containerID="8eee9bb6cfc91873f1bf16820ca3024490911940245f88796f2378ea0020e3d7"
Jul 07 17:12:03.013947 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:03.013925    2584 scope.go:117] "RemoveContainer" containerID="87451f0c2bd88c156802a297e0a2146da6ac946a7f36a512e93bd357152f9cd5"
Jul 07 17:12:03.022592 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:03.022557    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s"]
Jul 07 17:12:03.028744 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:03.022725    2584 scope.go:117] "RemoveContainer" containerID="2c645af33b694ce1f9f14addae8cad741041d6342d35e459119c30a3e3cb08f6"
Jul 07 17:12:03.028744 ip-10-0-133-184 kubenswrapper[2584]: E0707 17:12:03.023144    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c645af33b694ce1f9f14addae8cad741041d6342d35e459119c30a3e3cb08f6\": container with ID starting with 2c645af33b694ce1f9f14addae8cad741041d6342d35e459119c30a3e3cb08f6 not found: ID does not exist" containerID="2c645af33b694ce1f9f14addae8cad741041d6342d35e459119c30a3e3cb08f6"
Jul 07 17:12:03.028744 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:03.023185    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c645af33b694ce1f9f14addae8cad741041d6342d35e459119c30a3e3cb08f6"} err="failed to get container status \"2c645af33b694ce1f9f14addae8cad741041d6342d35e459119c30a3e3cb08f6\": rpc error: code = NotFound desc = could not find container \"2c645af33b694ce1f9f14addae8cad741041d6342d35e459119c30a3e3cb08f6\": container with ID starting with 2c645af33b694ce1f9f14addae8cad741041d6342d35e459119c30a3e3cb08f6 not found: ID does not exist"
Jul 07 17:12:03.028744 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:03.023230    2584 scope.go:117] "RemoveContainer" containerID="8eee9bb6cfc91873f1bf16820ca3024490911940245f88796f2378ea0020e3d7"
Jul 07 17:12:03.028744 ip-10-0-133-184 kubenswrapper[2584]: E0707 17:12:03.023644    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eee9bb6cfc91873f1bf16820ca3024490911940245f88796f2378ea0020e3d7\": container with ID starting with 8eee9bb6cfc91873f1bf16820ca3024490911940245f88796f2378ea0020e3d7 not found: ID does not exist" containerID="8eee9bb6cfc91873f1bf16820ca3024490911940245f88796f2378ea0020e3d7"
Jul 07 17:12:03.028744 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:03.023679    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eee9bb6cfc91873f1bf16820ca3024490911940245f88796f2378ea0020e3d7"} err="failed to get container status \"8eee9bb6cfc91873f1bf16820ca3024490911940245f88796f2378ea0020e3d7\": rpc error: code = NotFound desc = could not find container \"8eee9bb6cfc91873f1bf16820ca3024490911940245f88796f2378ea0020e3d7\": container with ID starting with 8eee9bb6cfc91873f1bf16820ca3024490911940245f88796f2378ea0020e3d7 not found: ID does not exist"
Jul 07 17:12:03.028744 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:03.023704    2584 scope.go:117] "RemoveContainer" containerID="87451f0c2bd88c156802a297e0a2146da6ac946a7f36a512e93bd357152f9cd5"
Jul 07 17:12:03.028744 ip-10-0-133-184 kubenswrapper[2584]: E0707 17:12:03.023987    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87451f0c2bd88c156802a297e0a2146da6ac946a7f36a512e93bd357152f9cd5\": container with ID starting with 87451f0c2bd88c156802a297e0a2146da6ac946a7f36a512e93bd357152f9cd5 not found: ID does not exist" containerID="87451f0c2bd88c156802a297e0a2146da6ac946a7f36a512e93bd357152f9cd5"
Jul 07 17:12:03.028744 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:03.024016    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87451f0c2bd88c156802a297e0a2146da6ac946a7f36a512e93bd357152f9cd5"} err="failed to get container status \"87451f0c2bd88c156802a297e0a2146da6ac946a7f36a512e93bd357152f9cd5\": rpc error: code = NotFound desc = could not find container \"87451f0c2bd88c156802a297e0a2146da6ac946a7f36a512e93bd357152f9cd5\": container with ID starting with 87451f0c2bd88c156802a297e0a2146da6ac946a7f36a512e93bd357152f9cd5 not found: ID does not exist"
Jul 07 17:12:03.029973 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:03.029944    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-rww9s"]
Jul 07 17:12:03.893921 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:03.893885    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44b0a47a-2d88-4e7b-ad9e-153639de4bdd" path="/var/lib/kubelet/pods/44b0a47a-2d88-4e7b-ad9e-153639de4bdd/volumes"
Jul 07 17:12:07.554217 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:07.554183    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-d8pbc/must-gather-z9bxj"]
Jul 07 17:12:07.554584 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:07.554504    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44b0a47a-2d88-4e7b-ad9e-153639de4bdd" containerName="storage-initializer"
Jul 07 17:12:07.554584 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:07.554514    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="44b0a47a-2d88-4e7b-ad9e-153639de4bdd" containerName="storage-initializer"
Jul 07 17:12:07.554584 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:07.554536    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44b0a47a-2d88-4e7b-ad9e-153639de4bdd" containerName="kube-rbac-proxy"
Jul 07 17:12:07.554584 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:07.554541    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="44b0a47a-2d88-4e7b-ad9e-153639de4bdd" containerName="kube-rbac-proxy"
Jul 07 17:12:07.554584 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:07.554551    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44b0a47a-2d88-4e7b-ad9e-153639de4bdd" containerName="kserve-container"
Jul 07 17:12:07.554584 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:07.554557    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="44b0a47a-2d88-4e7b-ad9e-153639de4bdd" containerName="kserve-container"
Jul 07 17:12:07.554781 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:07.554608    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="44b0a47a-2d88-4e7b-ad9e-153639de4bdd" containerName="kube-rbac-proxy"
Jul 07 17:12:07.554781 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:07.554616    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="44b0a47a-2d88-4e7b-ad9e-153639de4bdd" containerName="kserve-container"
Jul 07 17:12:07.559426 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:07.559407    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d8pbc/must-gather-z9bxj"
Jul 07 17:12:07.564411 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:07.564387    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d8pbc\"/\"openshift-service-ca.crt\""
Jul 07 17:12:07.564546 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:07.564483    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d8pbc\"/\"kube-root-ca.crt\""
Jul 07 17:12:07.565828 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:07.565803    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-d8pbc\"/\"default-dockercfg-gmvkv\""
Jul 07 17:12:07.571778 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:07.571754    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d8pbc/must-gather-z9bxj"]
Jul 07 17:12:07.588963 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:07.588935    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 17:12:07.686479 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:07.686435    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr25v\" (UniqueName: \"kubernetes.io/projected/cd1ed70e-55cb-41be-aa12-768fc5bbfe23-kube-api-access-dr25v\") pod \"must-gather-z9bxj\" (UID: \"cd1ed70e-55cb-41be-aa12-768fc5bbfe23\") " pod="openshift-must-gather-d8pbc/must-gather-z9bxj"
Jul 07 17:12:07.686479 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:07.686476    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cd1ed70e-55cb-41be-aa12-768fc5bbfe23-must-gather-output\") pod \"must-gather-z9bxj\" (UID: \"cd1ed70e-55cb-41be-aa12-768fc5bbfe23\") " pod="openshift-must-gather-d8pbc/must-gather-z9bxj"
Jul 07 17:12:07.702881 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:07.702852    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 17:12:07.787759 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:07.787727    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr25v\" (UniqueName: \"kubernetes.io/projected/cd1ed70e-55cb-41be-aa12-768fc5bbfe23-kube-api-access-dr25v\") pod \"must-gather-z9bxj\" (UID: \"cd1ed70e-55cb-41be-aa12-768fc5bbfe23\") " pod="openshift-must-gather-d8pbc/must-gather-z9bxj"
Jul 07 17:12:07.787933 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:07.787779    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cd1ed70e-55cb-41be-aa12-768fc5bbfe23-must-gather-output\") pod \"must-gather-z9bxj\" (UID: \"cd1ed70e-55cb-41be-aa12-768fc5bbfe23\") " pod="openshift-must-gather-d8pbc/must-gather-z9bxj"
Jul 07 17:12:07.788155 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:07.788137    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cd1ed70e-55cb-41be-aa12-768fc5bbfe23-must-gather-output\") pod \"must-gather-z9bxj\" (UID: \"cd1ed70e-55cb-41be-aa12-768fc5bbfe23\") " pod="openshift-must-gather-d8pbc/must-gather-z9bxj"
Jul 07 17:12:07.798312 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:07.798282    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr25v\" (UniqueName: \"kubernetes.io/projected/cd1ed70e-55cb-41be-aa12-768fc5bbfe23-kube-api-access-dr25v\") pod \"must-gather-z9bxj\" (UID: \"cd1ed70e-55cb-41be-aa12-768fc5bbfe23\") " pod="openshift-must-gather-d8pbc/must-gather-z9bxj"
Jul 07 17:12:07.882153 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:07.882113    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d8pbc/must-gather-z9bxj"
Jul 07 17:12:08.020191 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:08.020167    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d8pbc/must-gather-z9bxj"]
Jul 07 17:12:08.022141 ip-10-0-133-184 kubenswrapper[2584]: W0707 17:12:08.022110    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd1ed70e_55cb_41be_aa12_768fc5bbfe23.slice/crio-7ccf3380580d7438da284bd45dcec42ad8e7e47691923f43a00dbe8a4a797e29 WatchSource:0}: Error finding container 7ccf3380580d7438da284bd45dcec42ad8e7e47691923f43a00dbe8a4a797e29: Status 404 returned error can't find the container with id 7ccf3380580d7438da284bd45dcec42ad8e7e47691923f43a00dbe8a4a797e29
Jul 07 17:12:08.023930 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:08.023911    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 17:12:09.029536 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:09.029395    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d8pbc/must-gather-z9bxj" event={"ID":"cd1ed70e-55cb-41be-aa12-768fc5bbfe23","Type":"ContainerStarted","Data":"7ccf3380580d7438da284bd45dcec42ad8e7e47691923f43a00dbe8a4a797e29"}
Jul 07 17:12:15.063412 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:15.063370    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d8pbc/must-gather-z9bxj" event={"ID":"cd1ed70e-55cb-41be-aa12-768fc5bbfe23","Type":"ContainerStarted","Data":"6d2801ef709466655289e2b60cf618d45d171487b20e2cf3ea999ae4a9fcf2af"}
Jul 07 17:12:15.063857 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:15.063419    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d8pbc/must-gather-z9bxj" event={"ID":"cd1ed70e-55cb-41be-aa12-768fc5bbfe23","Type":"ContainerStarted","Data":"9ed569c0a3724620cda60b8df4e3a020ab90384e8e6a7af472db402b5bc7c29f"}
Jul 07 17:12:15.081971 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:15.081915    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-d8pbc/must-gather-z9bxj" podStartSLOduration=2.079989245 podStartE2EDuration="8.081898088s" totalImagesPullingTime="6.001908843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:12:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:12:15.080695819 +0000 UTC m=+1583.681904525" watchObservedRunningTime="2026-07-07 17:12:15.081898088 +0000 UTC m=+1583.683106856"
Jul 07 17:12:32.154446 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:32.154407    2584 generic.go:356] "Generic (PLEG): container finished" podID="cd1ed70e-55cb-41be-aa12-768fc5bbfe23" containerID="9ed569c0a3724620cda60b8df4e3a020ab90384e8e6a7af472db402b5bc7c29f" exitCode=0
Jul 07 17:12:32.154867 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:32.154480    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d8pbc/must-gather-z9bxj" event={"ID":"cd1ed70e-55cb-41be-aa12-768fc5bbfe23","Type":"ContainerDied","Data":"9ed569c0a3724620cda60b8df4e3a020ab90384e8e6a7af472db402b5bc7c29f"}
Jul 07 17:12:32.154925 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:32.154867    2584 scope.go:117] "RemoveContainer" containerID="9ed569c0a3724620cda60b8df4e3a020ab90384e8e6a7af472db402b5bc7c29f"
Jul 07 17:12:32.616104 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:32.616070    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-d8pbc_must-gather-z9bxj_cd1ed70e-55cb-41be-aa12-768fc5bbfe23/gather/0.log"
Jul 07 17:12:33.135091 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:33.135034    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 17:12:33.320632 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:33.320597    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 17:12:33.511367 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:33.511273    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 17:12:36.094671 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:36.094640    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2bkmf_b2627fd1-997d-4ce5-84aa-e4b5b85cc307/global-pull-secret-syncer/0.log"
Jul 07 17:12:36.301928 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:36.301897    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log"
Jul 07 17:12:36.451430 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:36.451335    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-184.ec2.internal_47045ed184587f2ba81f9c13a0f2aefa/haproxy/0.log"
Jul 07 17:12:38.059834 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:38.059797    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-d8pbc/must-gather-z9bxj"]
Jul 07 17:12:38.060382 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:38.059988    2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-d8pbc/must-gather-z9bxj" podUID="cd1ed70e-55cb-41be-aa12-768fc5bbfe23" containerName="copy" containerID="cri-o://6d2801ef709466655289e2b60cf618d45d171487b20e2cf3ea999ae4a9fcf2af" gracePeriod=2
Jul 07 17:12:38.066282 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:38.066254    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-d8pbc/must-gather-z9bxj"]
Jul 07 17:12:38.180898 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:38.180861    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-d8pbc_must-gather-z9bxj_cd1ed70e-55cb-41be-aa12-768fc5bbfe23/copy/0.log"
Jul 07 17:12:38.181345 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:38.181315    2584 generic.go:356] "Generic (PLEG): container finished" podID="cd1ed70e-55cb-41be-aa12-768fc5bbfe23" containerID="6d2801ef709466655289e2b60cf618d45d171487b20e2cf3ea999ae4a9fcf2af" exitCode=143
Jul 07 17:12:38.292988 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:38.292965    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-d8pbc_must-gather-z9bxj_cd1ed70e-55cb-41be-aa12-768fc5bbfe23/copy/0.log"
Jul 07 17:12:38.294388 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:38.294368    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d8pbc/must-gather-z9bxj"
Jul 07 17:12:38.296991 ip-10-0-133-184 kubenswrapper[2584]: E0707 17:12:38.296967    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-z9bxj\" is forbidden: User \"system:node:ip-10-0-133-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-d8pbc\": no relationship found between node 'ip-10-0-133-184.ec2.internal' and this object" podUID="cd1ed70e-55cb-41be-aa12-768fc5bbfe23" pod="openshift-must-gather-d8pbc/must-gather-z9bxj"
Jul 07 17:12:38.359122 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:38.359096    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cd1ed70e-55cb-41be-aa12-768fc5bbfe23-must-gather-output\") pod \"cd1ed70e-55cb-41be-aa12-768fc5bbfe23\" (UID: \"cd1ed70e-55cb-41be-aa12-768fc5bbfe23\") "
Jul 07 17:12:38.359274 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:38.359140    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dr25v\" (UniqueName: \"kubernetes.io/projected/cd1ed70e-55cb-41be-aa12-768fc5bbfe23-kube-api-access-dr25v\") pod \"cd1ed70e-55cb-41be-aa12-768fc5bbfe23\" (UID: \"cd1ed70e-55cb-41be-aa12-768fc5bbfe23\") "
Jul 07 17:12:38.360452 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:38.360423    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1ed70e-55cb-41be-aa12-768fc5bbfe23-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "cd1ed70e-55cb-41be-aa12-768fc5bbfe23" (UID: "cd1ed70e-55cb-41be-aa12-768fc5bbfe23"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:12:38.361472 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:38.361447    2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd1ed70e-55cb-41be-aa12-768fc5bbfe23-kube-api-access-dr25v" (OuterVolumeSpecName: "kube-api-access-dr25v") pod "cd1ed70e-55cb-41be-aa12-768fc5bbfe23" (UID: "cd1ed70e-55cb-41be-aa12-768fc5bbfe23"). InnerVolumeSpecName "kube-api-access-dr25v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:12:38.460450 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:38.460407    2584 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cd1ed70e-55cb-41be-aa12-768fc5bbfe23-must-gather-output\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:12:38.460450 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:38.460440    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr25v\" (UniqueName: \"kubernetes.io/projected/cd1ed70e-55cb-41be-aa12-768fc5bbfe23-kube-api-access-dr25v\") on node \"ip-10-0-133-184.ec2.internal\" DevicePath \"\""
Jul 07 17:12:38.961132 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:38.961095    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 17:12:39.050400 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:39.050364    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjbh8_a1a8fd82-5f36-474b-a780-b30b1d5122ca/konnectivity-agent/0.log" limit=1024
Jul 07 17:12:39.186902 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:39.186873    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-d8pbc_must-gather-z9bxj_cd1ed70e-55cb-41be-aa12-768fc5bbfe23/copy/0.log"
Jul 07 17:12:39.187375 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:39.187309    2584 scope.go:117] "RemoveContainer" containerID="6d2801ef709466655289e2b60cf618d45d171487b20e2cf3ea999ae4a9fcf2af"
Jul 07 17:12:39.187375 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:39.187322    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d8pbc/must-gather-z9bxj"
Jul 07 17:12:39.191084 ip-10-0-133-184 kubenswrapper[2584]: E0707 17:12:39.191041    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-z9bxj\" is forbidden: User \"system:node:ip-10-0-133-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-d8pbc\": no relationship found between node 'ip-10-0-133-184.ec2.internal' and this object" podUID="cd1ed70e-55cb-41be-aa12-768fc5bbfe23" pod="openshift-must-gather-d8pbc/must-gather-z9bxj"
Jul 07 17:12:39.196441 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:39.196425    2584 scope.go:117] "RemoveContainer" containerID="9ed569c0a3724620cda60b8df4e3a020ab90384e8e6a7af472db402b5bc7c29f"
Jul 07 17:12:39.200946 ip-10-0-133-184 kubenswrapper[2584]: E0707 17:12:39.200913    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-z9bxj\" is forbidden: User \"system:node:ip-10-0-133-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-d8pbc\": no relationship found between node 'ip-10-0-133-184.ec2.internal' and this object" podUID="cd1ed70e-55cb-41be-aa12-768fc5bbfe23" pod="openshift-must-gather-d8pbc/must-gather-z9bxj"
Jul 07 17:12:39.893122 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:39.893087    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd1ed70e-55cb-41be-aa12-768fc5bbfe23" path="/var/lib/kubelet/pods/cd1ed70e-55cb-41be-aa12-768fc5bbfe23/volumes"
Jul 07 17:12:40.294352 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:40.294324    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_58834c6a-cd00-428f-8a2b-37b529c9aadf/alertmanager/0.log"
Jul 07 17:12:40.322811 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:40.322785    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_58834c6a-cd00-428f-8a2b-37b529c9aadf/config-reloader/0.log"
Jul 07 17:12:40.355145 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:40.355123    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_58834c6a-cd00-428f-8a2b-37b529c9aadf/kube-rbac-proxy-web/0.log"
Jul 07 17:12:40.386787 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:40.386741    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_58834c6a-cd00-428f-8a2b-37b529c9aadf/kube-rbac-proxy/0.log"
Jul 07 17:12:40.421844 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:40.421816    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_58834c6a-cd00-428f-8a2b-37b529c9aadf/kube-rbac-proxy-metric/0.log"
Jul 07 17:12:40.452763 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:40.452682    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_58834c6a-cd00-428f-8a2b-37b529c9aadf/prom-label-proxy/0.log"
Jul 07 17:12:40.485679 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:40.485641    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_58834c6a-cd00-428f-8a2b-37b529c9aadf/init-config-reloader/0.log"
Jul 07 17:12:40.694953 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:40.694927    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-xzvh8_80c6c3b9-e9c7-4305-96cf-cb9124673a77/monitoring-plugin/0.log"
Jul 07 17:12:40.813311 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:40.813237    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q6m8v_ee4b1222-3d0f-485e-a082-745d5a8f1cb2/node-exporter/0.log"
Jul 07 17:12:40.840428 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:40.840397    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q6m8v_ee4b1222-3d0f-485e-a082-745d5a8f1cb2/kube-rbac-proxy/0.log"
Jul 07 17:12:40.870527 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:40.870504    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q6m8v_ee4b1222-3d0f-485e-a082-745d5a8f1cb2/init-textfile/0.log"
Jul 07 17:12:40.983478 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:40.983451    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-rw2p9_2449686a-76f3-46db-806b-953bc02530f3/kube-rbac-proxy-main/0.log"
Jul 07 17:12:41.016791 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:41.016765    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-rw2p9_2449686a-76f3-46db-806b-953bc02530f3/kube-rbac-proxy-self/0.log"
Jul 07 17:12:41.043184 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:41.043157    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-rw2p9_2449686a-76f3-46db-806b-953bc02530f3/openshift-state-metrics/0.log"
Jul 07 17:12:41.090424 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:41.090344    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_96441ff7-552d-4991-be04-8efb1673fbf3/prometheus/0.log"
Jul 07 17:12:41.116988 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:41.116968    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_96441ff7-552d-4991-be04-8efb1673fbf3/config-reloader/0.log"
Jul 07 17:12:41.153224 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:41.153194    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_96441ff7-552d-4991-be04-8efb1673fbf3/thanos-sidecar/0.log"
Jul 07 17:12:41.183361 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:41.183331    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_96441ff7-552d-4991-be04-8efb1673fbf3/kube-rbac-proxy-web/0.log"
Jul 07 17:12:41.221952 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:41.221924    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_96441ff7-552d-4991-be04-8efb1673fbf3/kube-rbac-proxy/0.log"
Jul 07 17:12:41.273329 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:41.273304    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_96441ff7-552d-4991-be04-8efb1673fbf3/kube-rbac-proxy-thanos/0.log"
Jul 07 17:12:41.357465 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:41.357436    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_96441ff7-552d-4991-be04-8efb1673fbf3/init-config-reloader/0.log"
Jul 07 17:12:41.395979 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:41.395950    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-j7zx8_b6b0ea8f-b17a-4ef6-8096-03e87c790d4c/prometheus-operator/0.log"
Jul 07 17:12:41.456620 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:41.456597    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-j7zx8_b6b0ea8f-b17a-4ef6-8096-03e87c790d4c/kube-rbac-proxy/0.log"
Jul 07 17:12:41.505102 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:41.505075    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-6zkdb_25fef8f4-e7f3-4274-a226-e07d12bc3efd/prometheus-operator-admission-webhook/0.log"
Jul 07 17:12:41.676987 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:41.676902    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55f89bcf5d-dp6c6_8f33dbb1-01ec-4e73-9cae-9a20e299c0b5/thanos-query/0.log"
Jul 07 17:12:41.734690 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:41.734664    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55f89bcf5d-dp6c6_8f33dbb1-01ec-4e73-9cae-9a20e299c0b5/kube-rbac-proxy-web/0.log"
Jul 07 17:12:41.808145 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:41.808116    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55f89bcf5d-dp6c6_8f33dbb1-01ec-4e73-9cae-9a20e299c0b5/kube-rbac-proxy/0.log"
Jul 07 17:12:41.865988 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:41.865956    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55f89bcf5d-dp6c6_8f33dbb1-01ec-4e73-9cae-9a20e299c0b5/prom-label-proxy/0.log"
Jul 07 17:12:41.933154 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:41.933041    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55f89bcf5d-dp6c6_8f33dbb1-01ec-4e73-9cae-9a20e299c0b5/kube-rbac-proxy-rules/0.log"
Jul 07 17:12:41.978562 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:41.978537    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55f89bcf5d-dp6c6_8f33dbb1-01ec-4e73-9cae-9a20e299c0b5/kube-rbac-proxy-metrics/0.log"
Jul 07 17:12:43.977914 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:43.977882    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s9nz9/perf-node-gather-daemonset-z96pg"]
Jul 07 17:12:43.978427 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:43.978216    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd1ed70e-55cb-41be-aa12-768fc5bbfe23" containerName="gather"
Jul 07 17:12:43.978427 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:43.978227    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd1ed70e-55cb-41be-aa12-768fc5bbfe23" containerName="gather"
Jul 07 17:12:43.978427 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:43.978241    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd1ed70e-55cb-41be-aa12-768fc5bbfe23" containerName="copy"
Jul 07 17:12:43.978427 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:43.978246    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd1ed70e-55cb-41be-aa12-768fc5bbfe23" containerName="copy"
Jul 07 17:12:43.978427 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:43.978304    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd1ed70e-55cb-41be-aa12-768fc5bbfe23" containerName="copy"
Jul 07 17:12:43.978427 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:43.978312    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd1ed70e-55cb-41be-aa12-768fc5bbfe23" containerName="gather"
Jul 07 17:12:43.980915 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:43.980896    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s9nz9/perf-node-gather-daemonset-z96pg"
Jul 07 17:12:43.988454 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:43.988434    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s9nz9\"/\"kube-root-ca.crt\""
Jul 07 17:12:43.988543 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:43.988500    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-s9nz9\"/\"default-dockercfg-thlq4\""
Jul 07 17:12:43.989869 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:43.989852    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s9nz9\"/\"openshift-service-ca.crt\""
Jul 07 17:12:43.994121 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:43.994102    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s9nz9/perf-node-gather-daemonset-z96pg"]
Jul 07 17:12:44.008618 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:44.008591    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/89924189-4ef1-42f1-9c50-b35bf153b700-proc\") pod \"perf-node-gather-daemonset-z96pg\" (UID: \"89924189-4ef1-42f1-9c50-b35bf153b700\") " pod="openshift-must-gather-s9nz9/perf-node-gather-daemonset-z96pg"
Jul 07 17:12:44.008742 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:44.008638    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89924189-4ef1-42f1-9c50-b35bf153b700-sys\") pod \"perf-node-gather-daemonset-z96pg\" (UID: \"89924189-4ef1-42f1-9c50-b35bf153b700\") " pod="openshift-must-gather-s9nz9/perf-node-gather-daemonset-z96pg"
Jul 07 17:12:44.008742 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:44.008677    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/89924189-4ef1-42f1-9c50-b35bf153b700-lib-modules\") pod \"perf-node-gather-daemonset-z96pg\" (UID: \"89924189-4ef1-42f1-9c50-b35bf153b700\") " pod="openshift-must-gather-s9nz9/perf-node-gather-daemonset-z96pg"
Jul 07 17:12:44.008837 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:44.008821    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/89924189-4ef1-42f1-9c50-b35bf153b700-podres\") pod \"perf-node-gather-daemonset-z96pg\" (UID: \"89924189-4ef1-42f1-9c50-b35bf153b700\") " pod="openshift-must-gather-s9nz9/perf-node-gather-daemonset-z96pg"
Jul 07 17:12:44.008882 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:44.008863    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgcxl\" (UniqueName: \"kubernetes.io/projected/89924189-4ef1-42f1-9c50-b35bf153b700-kube-api-access-vgcxl\") pod \"perf-node-gather-daemonset-z96pg\" (UID: \"89924189-4ef1-42f1-9c50-b35bf153b700\") " pod="openshift-must-gather-s9nz9/perf-node-gather-daemonset-z96pg"
Jul 07 17:12:44.110036 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:44.109996    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89924189-4ef1-42f1-9c50-b35bf153b700-sys\") pod \"perf-node-gather-daemonset-z96pg\" (UID: \"89924189-4ef1-42f1-9c50-b35bf153b700\") " pod="openshift-must-gather-s9nz9/perf-node-gather-daemonset-z96pg"
Jul 07 17:12:44.110268 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:44.110074    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/89924189-4ef1-42f1-9c50-b35bf153b700-lib-modules\") pod \"perf-node-gather-daemonset-z96pg\" (UID: \"89924189-4ef1-42f1-9c50-b35bf153b700\") " pod="openshift-must-gather-s9nz9/perf-node-gather-daemonset-z96pg"
Jul 07 17:12:44.110268 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:44.110128    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/89924189-4ef1-42f1-9c50-b35bf153b700-podres\") pod \"perf-node-gather-daemonset-z96pg\" (UID: \"89924189-4ef1-42f1-9c50-b35bf153b700\") " pod="openshift-must-gather-s9nz9/perf-node-gather-daemonset-z96pg"
Jul 07 17:12:44.110268 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:44.110131    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89924189-4ef1-42f1-9c50-b35bf153b700-sys\") pod \"perf-node-gather-daemonset-z96pg\" (UID: \"89924189-4ef1-42f1-9c50-b35bf153b700\") " pod="openshift-must-gather-s9nz9/perf-node-gather-daemonset-z96pg"
Jul 07 17:12:44.110268 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:44.110159    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgcxl\" (UniqueName: \"kubernetes.io/projected/89924189-4ef1-42f1-9c50-b35bf153b700-kube-api-access-vgcxl\") pod \"perf-node-gather-daemonset-z96pg\" (UID: \"89924189-4ef1-42f1-9c50-b35bf153b700\") " pod="openshift-must-gather-s9nz9/perf-node-gather-daemonset-z96pg"
Jul 07 17:12:44.110268 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:44.110202    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/89924189-4ef1-42f1-9c50-b35bf153b700-proc\") pod \"perf-node-gather-daemonset-z96pg\" (UID: \"89924189-4ef1-42f1-9c50-b35bf153b700\") " pod="openshift-must-gather-s9nz9/perf-node-gather-daemonset-z96pg"
Jul 07 17:12:44.110268 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:44.110224    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/89924189-4ef1-42f1-9c50-b35bf153b700-podres\") pod \"perf-node-gather-daemonset-z96pg\" (UID: \"89924189-4ef1-42f1-9c50-b35bf153b700\") " pod="openshift-must-gather-s9nz9/perf-node-gather-daemonset-z96pg"
Jul 07 17:12:44.110268 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:44.110244    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/89924189-4ef1-42f1-9c50-b35bf153b700-lib-modules\") pod \"perf-node-gather-daemonset-z96pg\" (UID: \"89924189-4ef1-42f1-9c50-b35bf153b700\") " pod="openshift-must-gather-s9nz9/perf-node-gather-daemonset-z96pg"
Jul 07 17:12:44.110556 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:44.110303    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/89924189-4ef1-42f1-9c50-b35bf153b700-proc\") pod \"perf-node-gather-daemonset-z96pg\" (UID: \"89924189-4ef1-42f1-9c50-b35bf153b700\") " pod="openshift-must-gather-s9nz9/perf-node-gather-daemonset-z96pg"
Jul 07 17:12:44.125214 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:44.125189    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgcxl\" (UniqueName: \"kubernetes.io/projected/89924189-4ef1-42f1-9c50-b35bf153b700-kube-api-access-vgcxl\") pod \"perf-node-gather-daemonset-z96pg\" (UID: \"89924189-4ef1-42f1-9c50-b35bf153b700\") " pod="openshift-must-gather-s9nz9/perf-node-gather-daemonset-z96pg"
Jul 07 17:12:44.290864 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:44.290775    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s9nz9/perf-node-gather-daemonset-z96pg"
Jul 07 17:12:44.362431 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:44.362408    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-xvlzm_c50b90e0-2467-43e9-ba20-83e720cab84d/download-server/0.log"
Jul 07 17:12:44.438165 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:44.438141    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s9nz9/perf-node-gather-daemonset-z96pg"]
Jul 07 17:12:44.440103 ip-10-0-133-184 kubenswrapper[2584]: W0707 17:12:44.440068    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod89924189_4ef1_42f1_9c50_b35bf153b700.slice/crio-3525f7a32b77b731d49bf6b9cdf26b3e29b3d7ff3bc80fd9944880d107b1ece5 WatchSource:0}: Error finding container 3525f7a32b77b731d49bf6b9cdf26b3e29b3d7ff3bc80fd9944880d107b1ece5: Status 404 returned error can't find the container with id 3525f7a32b77b731d49bf6b9cdf26b3e29b3d7ff3bc80fd9944880d107b1ece5
Jul 07 17:12:45.214883 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:45.214847    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s9nz9/perf-node-gather-daemonset-z96pg" event={"ID":"89924189-4ef1-42f1-9c50-b35bf153b700","Type":"ContainerStarted","Data":"a3951c13eaa50ccbe586b664dcb37a501ab6c0c0087ed11b9da444980c428d69"}
Jul 07 17:12:45.215389 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:45.214905    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s9nz9/perf-node-gather-daemonset-z96pg" event={"ID":"89924189-4ef1-42f1-9c50-b35bf153b700","Type":"ContainerStarted","Data":"3525f7a32b77b731d49bf6b9cdf26b3e29b3d7ff3bc80fd9944880d107b1ece5"}
Jul 07 17:12:45.215389 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:45.214957    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-s9nz9/perf-node-gather-daemonset-z96pg"
Jul 07 17:12:45.237403 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:45.237356    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-s9nz9/perf-node-gather-daemonset-z96pg" podStartSLOduration=2.237340643 podStartE2EDuration="2.237340643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:12:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:12:45.236429914 +0000 UTC m=+1613.837638618" watchObservedRunningTime="2026-07-07 17:12:45.237340643 +0000 UTC m=+1613.838549346"
Jul 07 17:12:45.991364 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:45.991333    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wv8wf_f3b39298-a197-40cb-8363-912fdc7dbcc8/dns/0.log"
Jul 07 17:12:46.019129 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:46.019101    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wv8wf_f3b39298-a197-40cb-8363-912fdc7dbcc8/kube-rbac-proxy/0.log"
Jul 07 17:12:46.098429 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:46.098395    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ptgp6_3d18f299-6f72-46ff-abf2-0ccf120fc875/dns-node-resolver/0.log"
Jul 07 17:12:46.757210 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:46.757184    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cpbvp_266c4079-c003-4676-a480-fb0e19a14cce/node-ca/0.log"
Jul 07 17:12:48.232963 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:48.232937    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ml4f2_b8e32c6b-206f-4882-b1c0-066d26486d41/serve-healthcheck-canary/0.log"
Jul 07 17:12:48.794399 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:48.794368    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jk247_600fd20e-5eba-4886-9bda-68a5c995d9a8/kube-rbac-proxy/0.log"
Jul 07 17:12:48.838373 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:48.838341    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jk247_600fd20e-5eba-4886-9bda-68a5c995d9a8/exporter/0.log"
Jul 07 17:12:48.870981 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:48.870953    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jk247_600fd20e-5eba-4886-9bda-68a5c995d9a8/extractor/0.log"
Jul 07 17:12:51.229198 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:51.229170    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-s9nz9/perf-node-gather-daemonset-z96pg"
Jul 07 17:12:51.673148 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:51.673117    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-gj2jw_f5c865f5-0889-4158-a322-d7cb04474e3b/s3-init/0.log"
Jul 07 17:12:51.707701 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:51.707679    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-gj2jw_f5c865f5-0889-4158-a322-d7cb04474e3b/download-hf-model/0.log"
Jul 07 17:12:51.811532 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:51.811502    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-s2j2b_eda6df85-b2f8-4a77-8641-7df86bce9d6a/s3-tls-init-custom/0.log"
Jul 07 17:12:51.843837 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:51.843814    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-s2j2b_eda6df85-b2f8-4a77-8641-7df86bce9d6a/download-hf-model/0.log"
Jul 07 17:12:51.894543 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:51.894509    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-rpzc4_c35abd2a-853f-4e82-8faf-635a125534a4/s3-tls-init-serving/0.log"
Jul 07 17:12:51.938067 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:51.937981    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-rpzc4_c35abd2a-853f-4e82-8faf-635a125534a4/download-hf-model/0.log"
Jul 07 17:12:57.320996 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:57.320965    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-2q4wx_edc60915-17c5-41d2-b611-4d875a7f18a0/migrator/0.log"
Jul 07 17:12:57.385499 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:57.385466    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-2q4wx_edc60915-17c5-41d2-b611-4d875a7f18a0/graceful-termination/0.log"
Jul 07 17:12:57.835663 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:57.835631    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-dcrfk_f45b3a44-6bd5-4cee-912e-5edc4d056972/kube-storage-version-migrator-operator/1.log"
Jul 07 17:12:57.836808 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:57.836791    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-dcrfk_f45b3a44-6bd5-4cee-912e-5edc4d056972/kube-storage-version-migrator-operator/0.log"
Jul 07 17:12:59.170897 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:59.170867    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-phn99_3b4f517c-5cf2-440c-ad4a-bf3aa0934616/kube-multus-additional-cni-plugins/0.log"
Jul 07 17:12:59.279194 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:59.279161    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-phn99_3b4f517c-5cf2-440c-ad4a-bf3aa0934616/egress-router-binary-copy/0.log"
Jul 07 17:12:59.319874 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:59.319843    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-phn99_3b4f517c-5cf2-440c-ad4a-bf3aa0934616/cni-plugins/0.log"
Jul 07 17:12:59.350487 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:59.350459    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-phn99_3b4f517c-5cf2-440c-ad4a-bf3aa0934616/bond-cni-plugin/0.log"
Jul 07 17:12:59.389613 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:59.389585    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-phn99_3b4f517c-5cf2-440c-ad4a-bf3aa0934616/routeoverride-cni/0.log"
Jul 07 17:12:59.442658 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:59.442575    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-phn99_3b4f517c-5cf2-440c-ad4a-bf3aa0934616/whereabouts-cni-bincopy/0.log"
Jul 07 17:12:59.479409 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:59.479378    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-phn99_3b4f517c-5cf2-440c-ad4a-bf3aa0934616/whereabouts-cni/0.log"
Jul 07 17:12:59.836854 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:59.836775    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-kdq9n_73096d96-d52d-47ee-95a8-2242bcfb4888/kube-multus/0.log"
Jul 07 17:12:59.878234 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:59.878200    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9kj8c_7f7e543c-8cdd-4f28-adb5-570b6004c796/network-metrics-daemon/0.log"
Jul 07 17:12:59.916656 ip-10-0-133-184 kubenswrapper[2584]: I0707 17:12:59.916626    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9kj8c_7f7e543c-8cdd-4f28-adb5-570b6004c796/kube-rbac-proxy/0.log"