Jul 21 01:36:20.272927 ip-10-0-129-219 systemd[1]: Starting Kubernetes Kubelet...
Jul 21 01:36:20.747345 ip-10-0-129-219 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 21 01:36:20.747345 ip-10-0-129-219 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 21 01:36:20.747345 ip-10-0-129-219 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 21 01:36:20.747345 ip-10-0-129-219 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 21 01:36:20.747345 ip-10-0-129-219 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 21 01:36:20.749916 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.749782    2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 21 01:36:20.753940 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.753916    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 21 01:36:20.753940 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.753934    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 21 01:36:20.753940 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.753939    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 21 01:36:20.753940 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.753943    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 21 01:36:20.753940 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.753948    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 21 01:36:20.754238 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.753953    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 21 01:36:20.754238 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.753957    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 21 01:36:20.754238 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.753961    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 21 01:36:20.754238 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.753965    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 21 01:36:20.754238 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.753969    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 21 01:36:20.754238 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.753974    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 21 01:36:20.754238 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.753978    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 21 01:36:20.754238 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.753982    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 21 01:36:20.754238 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.753988    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 21 01:36:20.754238 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.753992    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 21 01:36:20.754238 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.753996    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 21 01:36:20.754238 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754000    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 21 01:36:20.754238 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754004    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 21 01:36:20.754238 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754008    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 21 01:36:20.754238 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754013    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 21 01:36:20.754238 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754017    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 21 01:36:20.754238 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754022    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 21 01:36:20.754238 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754026    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 21 01:36:20.754238 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754031    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 21 01:36:20.754238 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754035    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 21 01:36:20.755122 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754039    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 21 01:36:20.755122 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754044    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 21 01:36:20.755122 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754048    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 21 01:36:20.755122 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754052    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 21 01:36:20.755122 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754056    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 21 01:36:20.755122 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754060    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 21 01:36:20.755122 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754065    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 21 01:36:20.755122 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754070    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 21 01:36:20.755122 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754074    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 21 01:36:20.755122 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754079    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 21 01:36:20.755122 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754083    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 21 01:36:20.755122 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754088    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 21 01:36:20.755122 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754092    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 21 01:36:20.755122 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754100    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 21 01:36:20.755122 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754105    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 21 01:36:20.755122 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754110    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 21 01:36:20.755122 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754114    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 21 01:36:20.755122 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754119    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 21 01:36:20.755122 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754125    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 21 01:36:20.755122 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754130    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 21 01:36:20.756058 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754134    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 21 01:36:20.756058 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754139    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 21 01:36:20.756058 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754143    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 21 01:36:20.756058 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754148    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 21 01:36:20.756058 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754152    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 21 01:36:20.756058 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754157    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 21 01:36:20.756058 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754161    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 21 01:36:20.756058 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754166    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 21 01:36:20.756058 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754170    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 21 01:36:20.756058 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754175    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 21 01:36:20.756058 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754179    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 21 01:36:20.756058 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754187    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 21 01:36:20.756058 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754193    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 21 01:36:20.756058 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754200    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 21 01:36:20.756058 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754207    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 21 01:36:20.756058 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754213    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 21 01:36:20.756058 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754218    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 21 01:36:20.756058 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754223    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 21 01:36:20.756058 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754228    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 21 01:36:20.756919 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754233    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 21 01:36:20.756919 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754237    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 21 01:36:20.756919 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754241    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 21 01:36:20.756919 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754246    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 21 01:36:20.756919 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754250    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 21 01:36:20.756919 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754255    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 21 01:36:20.756919 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754261    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 21 01:36:20.756919 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754270    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 21 01:36:20.756919 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754275    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 21 01:36:20.756919 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754280    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 21 01:36:20.756919 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754284    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 21 01:36:20.756919 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754289    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 21 01:36:20.756919 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754294    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 21 01:36:20.756919 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754298    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 21 01:36:20.756919 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754302    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 21 01:36:20.756919 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754307    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 21 01:36:20.756919 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754312    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 21 01:36:20.756919 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754316    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 21 01:36:20.756919 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754321    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 21 01:36:20.757866 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754325    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 21 01:36:20.757866 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754330    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 21 01:36:20.757866 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754335    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 21 01:36:20.757866 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754339    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 21 01:36:20.757866 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754344    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 21 01:36:20.757866 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754348    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 21 01:36:20.757866 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754353    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 21 01:36:20.757866 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754358    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 21 01:36:20.757866 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754362    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 21 01:36:20.757866 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754367    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 21 01:36:20.757866 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754371    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 21 01:36:20.757866 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754375    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 21 01:36:20.757866 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754380    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 21 01:36:20.757866 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754385    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 21 01:36:20.757866 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754389    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 21 01:36:20.757866 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.754396    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 21 01:36:20.757866 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755668    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 21 01:36:20.757866 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755679    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 21 01:36:20.757866 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755703    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 21 01:36:20.757866 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755709    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 21 01:36:20.758788 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755714    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 21 01:36:20.758788 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755720    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 21 01:36:20.758788 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755725    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 21 01:36:20.758788 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755729    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 21 01:36:20.758788 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755733    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 21 01:36:20.758788 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755738    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 21 01:36:20.758788 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755742    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 21 01:36:20.758788 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755747    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 21 01:36:20.758788 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755751    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 21 01:36:20.758788 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755756    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 21 01:36:20.758788 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755760    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 21 01:36:20.758788 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755765    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 21 01:36:20.758788 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755769    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 21 01:36:20.758788 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755774    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 21 01:36:20.758788 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755778    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 21 01:36:20.758788 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755783    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 21 01:36:20.758788 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755787    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 21 01:36:20.758788 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755792    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 21 01:36:20.758788 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755796    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 21 01:36:20.758788 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755801    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 21 01:36:20.759537 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755805    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 21 01:36:20.759537 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755810    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 21 01:36:20.759537 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755814    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 21 01:36:20.759537 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755819    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 21 01:36:20.759537 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755823    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 21 01:36:20.759537 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755828    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 21 01:36:20.759537 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755832    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 21 01:36:20.759537 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755837    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 21 01:36:20.759537 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755842    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 21 01:36:20.759537 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755846    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 21 01:36:20.759537 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755851    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 21 01:36:20.759537 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755857    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 21 01:36:20.759537 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755862    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 21 01:36:20.759537 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755867    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 21 01:36:20.759537 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755872    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 21 01:36:20.759537 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755877    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 21 01:36:20.759537 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755881    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 21 01:36:20.759537 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755885    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 21 01:36:20.759537 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755891    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 21 01:36:20.760261 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755898    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 21 01:36:20.760261 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755903    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 21 01:36:20.760261 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755908    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 21 01:36:20.760261 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755912    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 21 01:36:20.760261 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755917    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 21 01:36:20.760261 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755921    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 21 01:36:20.760261 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755926    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 21 01:36:20.760261 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755931    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 21 01:36:20.760261 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755935    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 21 01:36:20.760261 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755940    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 21 01:36:20.760261 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755944    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 21 01:36:20.760261 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755952    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 21 01:36:20.760261 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755957    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 21 01:36:20.760261 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755963    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 21 01:36:20.760261 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755967    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 21 01:36:20.760261 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755972    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 21 01:36:20.760261 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755978    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 21 01:36:20.760261 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755982    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 21 01:36:20.760261 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755987    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 21 01:36:20.760935 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.755992    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 21 01:36:20.760935 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756001    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 21 01:36:20.760935 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756006    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 21 01:36:20.760935 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756012    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 21 01:36:20.760935 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756016    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 21 01:36:20.760935 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756021    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 21 01:36:20.760935 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756026    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 21 01:36:20.760935 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756030    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 21 01:36:20.760935 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756035    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 21 01:36:20.760935 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756044    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 21 01:36:20.760935 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756053    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 21 01:36:20.760935 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756058    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 21 01:36:20.760935 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756063    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 21 01:36:20.760935 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756068    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 21 01:36:20.760935 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756073    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 21 01:36:20.760935 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756078    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 21 01:36:20.760935 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756082    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 21 01:36:20.760935 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756087    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 21 01:36:20.760935 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756092    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 21 01:36:20.760935 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756096    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 21 01:36:20.761479 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756101    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 21 01:36:20.761479 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756105    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 21 01:36:20.761479 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756110    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 21 01:36:20.761479 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756115    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 21 01:36:20.761479 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756119    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 21 01:36:20.761479 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756124    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 21 01:36:20.761479 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756129    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 21 01:36:20.761479 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756134    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 21 01:36:20.761479 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756138    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 21 01:36:20.761479 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756145    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 21 01:36:20.761479 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756151    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 21 01:36:20.761479 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756155    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 21 01:36:20.761479 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756160    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 21 01:36:20.761479 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756167    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 21 01:36:20.761479 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756172    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 21 01:36:20.761479 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756177    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 21 01:36:20.761479 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756181    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 21 01:36:20.761479 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756864    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 21 01:36:20.761479 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756873    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 21 01:36:20.761479 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756878    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 21 01:36:20.762121 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756883    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 21 01:36:20.762121 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756889    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 21 01:36:20.762121 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756893    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 21 01:36:20.762121 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756898    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 21 01:36:20.762121 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756903    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 21 01:36:20.762121 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756907    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 21 01:36:20.762121 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756912    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 21 01:36:20.762121 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756916    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 21 01:36:20.762121 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756920    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 21 01:36:20.762121 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756924    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 21 01:36:20.762121 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756929    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 21 01:36:20.762121 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756934    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 21 01:36:20.762121 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756938    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 21 01:36:20.762121 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756943    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 21 01:36:20.762121 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756947    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 21 01:36:20.762121 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756952    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 21 01:36:20.762121 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756957    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 21 01:36:20.762121 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756962    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 21 01:36:20.762121 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756966    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 21 01:36:20.762121 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756971    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 21 01:36:20.762643 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756976    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 21 01:36:20.762643 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756981    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 21 01:36:20.762643 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756986    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 21 01:36:20.762643 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756990    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 21 01:36:20.762643 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.756994    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 21 01:36:20.762643 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757000    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 21 01:36:20.762643 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757005    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 21 01:36:20.762643 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757010    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 21 01:36:20.762643 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757014    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 21 01:36:20.762643 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757018    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 21 01:36:20.762643 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757023    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 21 01:36:20.762643 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757027    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 21 01:36:20.762643 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757032    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 21 01:36:20.762643 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757036    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 21 01:36:20.762643 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757041    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 21 01:36:20.762643 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757046    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 21 01:36:20.762643 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757050    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 21 01:36:20.762643 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757055    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 21 01:36:20.762643 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757059    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 21 01:36:20.763178 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757063    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 21 01:36:20.763178 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757068    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 21 01:36:20.763178 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757073    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 21 01:36:20.763178 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757077    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 21 01:36:20.763178 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757081    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 21 01:36:20.763178 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757086    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 21 01:36:20.763178 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757090    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 21 01:36:20.763178 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757095    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 21 01:36:20.763178 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757099    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 21 01:36:20.763178 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757104    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 21 01:36:20.763178 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757108    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 21 01:36:20.763178 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757112    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 21 01:36:20.763178 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757117    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 21 01:36:20.763178 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757121    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 21 01:36:20.763178 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757126    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 21 01:36:20.763178 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757130    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 21 01:36:20.763178 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757134    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 21 01:36:20.763178 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757139    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 21 01:36:20.763178 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757144    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 21 01:36:20.763178 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757149    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 21 01:36:20.763719 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757157    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 21 01:36:20.763719 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757163    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 21 01:36:20.763719 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757168    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 21 01:36:20.763719 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757173    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 21 01:36:20.763719 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757177    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 21 01:36:20.763719 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757182    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 21 01:36:20.763719 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757186    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 21 01:36:20.763719 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757191    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 21 01:36:20.763719 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757202    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 21 01:36:20.763719 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757206    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 21 01:36:20.763719 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757212    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 21 01:36:20.763719 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757218    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 21 01:36:20.763719 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757223    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 21 01:36:20.763719 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757228    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 21 01:36:20.763719 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757232    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 21 01:36:20.763719 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757237    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 21 01:36:20.763719 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757242    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 21 01:36:20.763719 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757249    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 21 01:36:20.763719 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757254    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 21 01:36:20.763719 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757258    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 21 01:36:20.764295 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757263    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 21 01:36:20.764295 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757268    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 21 01:36:20.764295 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757272    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 21 01:36:20.764295 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757277    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 21 01:36:20.764295 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757282    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 21 01:36:20.764295 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757287    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 21 01:36:20.764295 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757293    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 21 01:36:20.764295 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757298    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 21 01:36:20.764295 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757302    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 21 01:36:20.764295 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757307    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 21 01:36:20.764295 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757312    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 21 01:36:20.764295 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757317    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 21 01:36:20.764295 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757322    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 21 01:36:20.764295 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757327    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 21 01:36:20.764295 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757334    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 21 01:36:20.764295 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757340    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 21 01:36:20.764295 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.757346    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 21 01:36:20.764295 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757449    2575 flags.go:64] FLAG: --address="0.0.0.0"
Jul 21 01:36:20.764295 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757460    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 21 01:36:20.764295 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757472    2575 flags.go:64] FLAG: --anonymous-auth="true"
Jul 21 01:36:20.764827 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757480    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 21 01:36:20.764827 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757488    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 21 01:36:20.764827 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757494    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 21 01:36:20.764827 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757502    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 21 01:36:20.764827 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757509    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 21 01:36:20.764827 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757514    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 21 01:36:20.764827 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757519    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 21 01:36:20.764827 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757526    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 21 01:36:20.764827 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757532    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 21 01:36:20.764827 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757543    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 21 01:36:20.764827 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757548    2575 flags.go:64] FLAG: --cgroup-root=""
Jul 21 01:36:20.764827 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757553    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 21 01:36:20.764827 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757558    2575 flags.go:64] FLAG: --client-ca-file=""
Jul 21 01:36:20.764827 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757564    2575 flags.go:64] FLAG: --cloud-provider="external"
Jul 21 01:36:20.764827 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757569    2575 flags.go:64] FLAG: --cluster-dns="[]"
Jul 21 01:36:20.764827 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757576    2575 flags.go:64] FLAG: --cluster-domain=""
Jul 21 01:36:20.764827 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757582    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 21 01:36:20.764827 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757588    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 21 01:36:20.764827 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757594    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 21 01:36:20.764827 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757599    2575 flags.go:64] FLAG: --container-log-max-files="5"
Jul 21 01:36:20.764827 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757606    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 21 01:36:20.764827 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757611    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 21 01:36:20.764827 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757616    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 21 01:36:20.765480 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757621    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 21 01:36:20.765480 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757626    2575 flags.go:64] FLAG: --contention-profiling="false"
Jul 21 01:36:20.765480 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757632    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 21 01:36:20.765480 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757637    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 21 01:36:20.765480 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757642    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 21 01:36:20.765480 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757647    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 21 01:36:20.765480 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757654    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 21 01:36:20.765480 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757659    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 21 01:36:20.765480 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757665    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 21 01:36:20.765480 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757670    2575 flags.go:64] FLAG: --enable-load-reader="false"
Jul 21 01:36:20.765480 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757675    2575 flags.go:64] FLAG: --enable-server="true"
Jul 21 01:36:20.765480 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757681    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 21 01:36:20.765480 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757710    2575 flags.go:64] FLAG: --event-burst="100"
Jul 21 01:36:20.765480 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757716    2575 flags.go:64] FLAG: --event-qps="50"
Jul 21 01:36:20.765480 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757721    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 21 01:36:20.765480 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757726    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 21 01:36:20.765480 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757732    2575 flags.go:64] FLAG: --eviction-hard=""
Jul 21 01:36:20.765480 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757738    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 21 01:36:20.765480 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757746    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 21 01:36:20.765480 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757751    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 21 01:36:20.765480 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757756    2575 flags.go:64] FLAG: --eviction-soft=""
Jul 21 01:36:20.765480 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757761    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 21 01:36:20.765480 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757769    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 21 01:36:20.765480 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757775    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 21 01:36:20.765480 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757780    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 21 01:36:20.766178 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757785    2575 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 21 01:36:20.766178 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757791    2575 flags.go:64] FLAG: --fail-swap-on="true"
Jul 21 01:36:20.766178 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757796    2575 flags.go:64] FLAG: --feature-gates=""
Jul 21 01:36:20.766178 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757802    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 21 01:36:20.766178 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757807    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 21 01:36:20.766178 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757813    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 21 01:36:20.766178 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757818    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 21 01:36:20.766178 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757824    2575 flags.go:64] FLAG: --healthz-port="10248"
Jul 21 01:36:20.766178 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757829    2575 flags.go:64] FLAG: --help="false"
Jul 21 01:36:20.766178 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757834    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-129-219.ec2.internal"
Jul 21 01:36:20.766178 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757840    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 21 01:36:20.766178 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757845    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 21 01:36:20.766178 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757850    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 21 01:36:20.766178 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757857    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 21 01:36:20.766178 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757862    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 21 01:36:20.766178 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757867    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 21 01:36:20.766178 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757872    2575 flags.go:64] FLAG: --image-service-endpoint=""
Jul 21 01:36:20.766178 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757877    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 21 01:36:20.766178 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757882    2575 flags.go:64] FLAG: --kube-api-burst="100"
Jul 21 01:36:20.766178 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757887    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 21 01:36:20.766178 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757893    2575 flags.go:64] FLAG: --kube-api-qps="50"
Jul 21 01:36:20.766178 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757898    2575 flags.go:64] FLAG: --kube-reserved=""
Jul 21 01:36:20.766178 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757904    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 21 01:36:20.766178 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757909    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 21 01:36:20.766829 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757914    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 21 01:36:20.766829 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757921    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 21 01:36:20.766829 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757926    2575 flags.go:64] FLAG: --lock-file=""
Jul 21 01:36:20.766829 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757931    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 21 01:36:20.766829 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757936    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 21 01:36:20.766829 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757944    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 21 01:36:20.766829 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757954    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 21 01:36:20.766829 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757959    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 21 01:36:20.766829 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757964    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 21 01:36:20.766829 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757969    2575 flags.go:64] FLAG: --logging-format="text"
Jul 21 01:36:20.766829 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757975    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 21 01:36:20.766829 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757981    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 21 01:36:20.766829 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757987    2575 flags.go:64] FLAG: --manifest-url=""
Jul 21 01:36:20.766829 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757992    2575 flags.go:64] FLAG: --manifest-url-header=""
Jul 21 01:36:20.766829 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.757999    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 21 01:36:20.766829 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758005    2575 flags.go:64] FLAG: --max-open-files="1000000"
Jul 21 01:36:20.766829 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758012    2575 flags.go:64] FLAG: --max-pods="110"
Jul 21 01:36:20.766829 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758017    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 21 01:36:20.766829 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758022    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 21 01:36:20.766829 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758027    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 21 01:36:20.766829 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758032    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 21 01:36:20.766829 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758038    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 21 01:36:20.766829 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758043    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 21 01:36:20.766829 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758049    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 21 01:36:20.766829 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758062    2575 flags.go:64] FLAG: --node-status-max-images="50"
Jul 21 01:36:20.767486 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758067    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 21 01:36:20.767486 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758072    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 21 01:36:20.767486 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758078    2575 flags.go:64] FLAG: --pod-cidr=""
Jul 21 01:36:20.767486 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758083    2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 21 01:36:20.767486 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758093    2575 flags.go:64] FLAG: --pod-manifest-path=""
Jul 21 01:36:20.767486 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758098    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 21 01:36:20.767486 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758104    2575 flags.go:64] FLAG: --pods-per-core="0"
Jul 21 01:36:20.767486 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758109    2575 flags.go:64] FLAG: --port="10250"
Jul 21 01:36:20.767486 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758117    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 21 01:36:20.767486 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758122    2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09fcf5973410c2584"
Jul 21 01:36:20.767486 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758127    2575 flags.go:64] FLAG: --qos-reserved=""
Jul 21 01:36:20.767486 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758133    2575 flags.go:64] FLAG: --read-only-port="10255"
Jul 21 01:36:20.767486 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758138    2575 flags.go:64] FLAG: --register-node="true"
Jul 21 01:36:20.767486 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758145    2575 flags.go:64] FLAG: --register-with-taints=""
Jul 21 01:36:20.767486 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758151    2575 flags.go:64] FLAG: --registry-burst="10"
Jul 21 01:36:20.767486 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758156    2575 flags.go:64] FLAG: --registry-qps="5"
Jul 21 01:36:20.767486 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758161    2575 flags.go:64] FLAG: --reserved-cpus=""
Jul 21 01:36:20.767486 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758166    2575 flags.go:64] FLAG: --reserved-memory=""
Jul 21 01:36:20.767486 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758173    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 21 01:36:20.767486 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758179    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 21 01:36:20.767486 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758184    2575 flags.go:64] FLAG: --rotate-certificates="false"
Jul 21 01:36:20.767486 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758189    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 21 01:36:20.767486 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758194    2575 flags.go:64] FLAG: --runonce="false"
Jul 21 01:36:20.767486 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758199    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 21 01:36:20.767486 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758204    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 21 01:36:20.768200 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758209    2575 flags.go:64] FLAG: --seccomp-default="false"
Jul 21 01:36:20.768200 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758214    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 21 01:36:20.768200 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758220    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 21 01:36:20.768200 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758225    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 21 01:36:20.768200 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758230    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 21 01:36:20.768200 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758235    2575 flags.go:64] FLAG: --storage-driver-password="root"
Jul 21 01:36:20.768200 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758240    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 21 01:36:20.768200 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758245    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 21 01:36:20.768200 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758250    2575 flags.go:64] FLAG: --storage-driver-user="root"
Jul 21 01:36:20.768200 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758255    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 21 01:36:20.768200 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758261    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 21 01:36:20.768200 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758267    2575 flags.go:64] FLAG: --system-cgroups=""
Jul 21 01:36:20.768200 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758272    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 21 01:36:20.768200 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758282    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 21 01:36:20.768200 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758286    2575 flags.go:64] FLAG: --tls-cert-file=""
Jul 21 01:36:20.768200 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758293    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 21 01:36:20.768200 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758301    2575 flags.go:64] FLAG: --tls-min-version=""
Jul 21 01:36:20.768200 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758306    2575 flags.go:64] FLAG: --tls-private-key-file=""
Jul 21 01:36:20.768200 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758311    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 21 01:36:20.768200 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758316    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 21 01:36:20.768200 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758323    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 21 01:36:20.768200 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758328    2575 flags.go:64] FLAG: --v="2"
Jul 21 01:36:20.768200 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758335    2575 flags.go:64] FLAG: --version="false"
Jul 21 01:36:20.768200 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758342    2575 flags.go:64] FLAG: --vmodule=""
Jul 21 01:36:20.768200 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758349    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 21 01:36:20.768945 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.758354    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 21 01:36:20.768945 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758582    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 21 01:36:20.768945 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758589    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 21 01:36:20.768945 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758595    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 21 01:36:20.768945 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758600    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 21 01:36:20.768945 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758605    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 21 01:36:20.768945 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758610    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 21 01:36:20.768945 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758615    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 21 01:36:20.768945 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758621    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 21 01:36:20.768945 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758626    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 21 01:36:20.768945 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758631    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 21 01:36:20.768945 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758636    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 21 01:36:20.768945 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758641    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 21 01:36:20.768945 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758646    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 21 01:36:20.768945 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758651    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 21 01:36:20.768945 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758656    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 21 01:36:20.768945 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758660    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 21 01:36:20.768945 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758665    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 21 01:36:20.768945 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758669    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 21 01:36:20.768945 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758675    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 21 01:36:20.769548 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758680    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 21 01:36:20.769548 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758704    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 21 01:36:20.769548 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758713    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 21 01:36:20.769548 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758719    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 21 01:36:20.769548 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758724    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 21 01:36:20.769548 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758729    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 21 01:36:20.769548 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758734    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 21 01:36:20.769548 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758741    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 21 01:36:20.769548 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758746    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 21 01:36:20.769548 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758752    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 21 01:36:20.769548 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758757    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 21 01:36:20.769548 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758762    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 21 01:36:20.769548 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758766    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 21 01:36:20.769548 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758771    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 21 01:36:20.769548 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758776    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 21 01:36:20.769548 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758781    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 21 01:36:20.769548 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758788    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 21 01:36:20.769548 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758794    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 21 01:36:20.769548 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758798    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 21 01:36:20.770375 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758803    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 21 01:36:20.770375 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758808    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 21 01:36:20.770375 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758813    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 21 01:36:20.770375 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758818    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 21 01:36:20.770375 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758823    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 21 01:36:20.770375 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758828    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 21 01:36:20.770375 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758833    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 21 01:36:20.770375 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758838    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 21 01:36:20.770375 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758843    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 21 01:36:20.770375 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758848    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 21 01:36:20.770375 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758853    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 21 01:36:20.770375 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758857    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 21 01:36:20.770375 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758861    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 21 01:36:20.770375 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758867    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 21 01:36:20.770375 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758872    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 21 01:36:20.770375 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758879    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 21 01:36:20.770375 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758884    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 21 01:36:20.770375 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758888    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 21 01:36:20.770375 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758893    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 21 01:36:20.770375 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758898    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 21 01:36:20.770997 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758904    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 21 01:36:20.770997 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758908    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 21 01:36:20.770997 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758913    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 21 01:36:20.770997 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758917    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 21 01:36:20.770997 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758922    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 21 01:36:20.770997 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758927    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 21 01:36:20.770997 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758931    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 21 01:36:20.770997 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758936    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 21 01:36:20.770997 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758940    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 21 01:36:20.770997 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758944    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 21 01:36:20.770997 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758949    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 21 01:36:20.770997 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758953    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 21 01:36:20.770997 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758958    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 21 01:36:20.770997 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758963    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 21 01:36:20.770997 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758969    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 21 01:36:20.770997 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758976    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 21 01:36:20.770997 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758980    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 21 01:36:20.770997 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758985    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 21 01:36:20.770997 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758989    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 21 01:36:20.771529 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758994    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 21 01:36:20.771529 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.758998    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 21 01:36:20.771529 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.759002    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 21 01:36:20.771529 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.759008    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 21 01:36:20.771529 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.759012    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 21 01:36:20.771529 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.759016    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 21 01:36:20.771529 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.759021    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 21 01:36:20.771529 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.759026    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 21 01:36:20.771529 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.759033    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 21 01:36:20.771529 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.759037    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 21 01:36:20.771529 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.759041    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 21 01:36:20.771529 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.759046    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 21 01:36:20.771529 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.759051    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 21 01:36:20.771529 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.759058    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 21 01:36:20.771529 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.759063    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 21 01:36:20.771529 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.759067    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 21 01:36:20.771529 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.759071    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 21 01:36:20.771529 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.759076    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 21 01:36:20.771529 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.759080    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 21 01:36:20.771529 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.759085    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 21 01:36:20.772208 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.759090    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 21 01:36:20.772208 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.759095    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 21 01:36:20.772208 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.759103    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 21 01:36:20.772208 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.766619    2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 21 01:36:20.772208 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.766636    2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 21 01:36:20.772208 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.766654    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 21 01:36:20.772208 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.766663    2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 21 01:36:20.772208 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766737    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 21 01:36:20.772208 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766743    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 21 01:36:20.772208 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766747    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 21 01:36:20.772208 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766751    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 21 01:36:20.772208 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766754    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 21 01:36:20.772208 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766757    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 21 01:36:20.772208 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766760    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 21 01:36:20.772683 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766764    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 21 01:36:20.772683 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766767    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 21 01:36:20.772683 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766770    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 21 01:36:20.772683 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766773    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 21 01:36:20.772683 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766776    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 21 01:36:20.772683 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766780    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 21 01:36:20.772683 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766783    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 21 01:36:20.772683 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766786    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 21 01:36:20.772683 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766790    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 21 01:36:20.772683 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766794    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 21 01:36:20.772683 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766798    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 21 01:36:20.772683 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766801    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 21 01:36:20.772683 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766804    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 21 01:36:20.772683 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766807    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 21 01:36:20.772683 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766810    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 21 01:36:20.772683 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766813    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 21 01:36:20.772683 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766815    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 21 01:36:20.772683 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766818    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 21 01:36:20.772683 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766822    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 21 01:36:20.772683 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766824    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 21 01:36:20.773225 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766827    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 21 01:36:20.773225 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766831    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 21 01:36:20.773225 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766836    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 21 01:36:20.773225 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766840    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 21 01:36:20.773225 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766844    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 21 01:36:20.773225 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766847    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 21 01:36:20.773225 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766851    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 21 01:36:20.773225 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766854    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 21 01:36:20.773225 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766856    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 21 01:36:20.773225 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766859    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 21 01:36:20.773225 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766863    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 21 01:36:20.773225 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766865    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 21 01:36:20.773225 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766868    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 21 01:36:20.773225 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766871    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 21 01:36:20.773225 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766874    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 21 01:36:20.773225 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766877    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 21 01:36:20.773225 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766879    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 21 01:36:20.773225 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766882    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 21 01:36:20.773225 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766886    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 21 01:36:20.773825 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766889    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 21 01:36:20.773825 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766891    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 21 01:36:20.773825 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766894    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 21 01:36:20.773825 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766897    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 21 01:36:20.773825 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766899    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 21 01:36:20.773825 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766902    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 21 01:36:20.773825 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766905    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 21 01:36:20.773825 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766907    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 21 01:36:20.773825 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766910    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 21 01:36:20.773825 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766913    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 21 01:36:20.773825 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766916    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 21 01:36:20.773825 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766918    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 21 01:36:20.773825 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766921    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 21 01:36:20.773825 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766924    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 21 01:36:20.773825 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766926    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 21 01:36:20.773825 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766930    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 21 01:36:20.773825 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766933    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 21 01:36:20.773825 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766936    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 21 01:36:20.773825 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766938    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 21 01:36:20.773825 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766941    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 21 01:36:20.774336 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766944    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 21 01:36:20.774336 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766947    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 21 01:36:20.774336 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766949    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 21 01:36:20.774336 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766952    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 21 01:36:20.774336 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766955    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 21 01:36:20.774336 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766957    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 21 01:36:20.774336 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766961    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 21 01:36:20.774336 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766964    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 21 01:36:20.774336 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766966    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 21 01:36:20.774336 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766969    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 21 01:36:20.774336 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766972    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 21 01:36:20.774336 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766975    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 21 01:36:20.774336 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766978    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 21 01:36:20.774336 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766981    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 21 01:36:20.774336 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766984    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 21 01:36:20.774336 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766987    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 21 01:36:20.774336 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766989    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 21 01:36:20.774336 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766992    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 21 01:36:20.774336 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766995    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 21 01:36:20.774336 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.766998    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 21 01:36:20.774897 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767000    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 21 01:36:20.774897 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767003    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 21 01:36:20.774897 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767006    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 21 01:36:20.774897 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767010    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 21 01:36:20.774897 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767014    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 21 01:36:20.774897 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767017    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 21 01:36:20.774897 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767020    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 21 01:36:20.774897 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767024    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 21 01:36:20.774897 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767027    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 21 01:36:20.774897 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767030    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 21 01:36:20.774897 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767033    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 21 01:36:20.774897 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767036    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 21 01:36:20.774897 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767039    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 21 01:36:20.774897 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.767044    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 21 01:36:20.775277 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767164    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 21 01:36:20.775277 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767169    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 21 01:36:20.775277 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767173    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 21 01:36:20.775277 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767176    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 21 01:36:20.775277 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767178    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 21 01:36:20.775277 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767181    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 21 01:36:20.775277 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767184    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 21 01:36:20.775277 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767188    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 21 01:36:20.775277 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767192    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 21 01:36:20.775277 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767195    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 21 01:36:20.775277 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767198    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 21 01:36:20.775277 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767202    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 21 01:36:20.775277 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767206    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 21 01:36:20.775277 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767209    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 21 01:36:20.775277 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767212    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 21 01:36:20.775277 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767214    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 21 01:36:20.775277 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767217    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 21 01:36:20.775277 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767220    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 21 01:36:20.775277 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767223    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 21 01:36:20.775277 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767226    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 21 01:36:20.775830 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767228    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 21 01:36:20.775830 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767231    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 21 01:36:20.775830 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767235    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 21 01:36:20.775830 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767238    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 21 01:36:20.775830 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767241    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 21 01:36:20.775830 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767244    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 21 01:36:20.775830 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767246    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 21 01:36:20.775830 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767249    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 21 01:36:20.775830 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767252    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 21 01:36:20.775830 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767254    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 21 01:36:20.775830 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767257    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 21 01:36:20.775830 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767259    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 21 01:36:20.775830 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767262    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 21 01:36:20.775830 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767265    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 21 01:36:20.775830 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767268    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 21 01:36:20.775830 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767270    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 21 01:36:20.775830 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767273    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 21 01:36:20.775830 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767276    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 21 01:36:20.775830 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767278    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 21 01:36:20.775830 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767281    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 21 01:36:20.776343 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767284    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 21 01:36:20.776343 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767286    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 21 01:36:20.776343 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767289    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 21 01:36:20.776343 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767291    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 21 01:36:20.776343 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767295    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 21 01:36:20.776343 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767297    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 21 01:36:20.776343 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767300    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 21 01:36:20.776343 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767302    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 21 01:36:20.776343 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767305    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 21 01:36:20.776343 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767308    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 21 01:36:20.776343 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767311    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 21 01:36:20.776343 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767314    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 21 01:36:20.776343 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767316    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 21 01:36:20.776343 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767319    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 21 01:36:20.776343 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767322    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 21 01:36:20.776343 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767325    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 21 01:36:20.776343 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767327    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 21 01:36:20.776343 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767330    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 21 01:36:20.776343 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767333    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 21 01:36:20.776343 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767336    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 21 01:36:20.776929 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767339    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 21 01:36:20.776929 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767341    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 21 01:36:20.776929 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767344    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 21 01:36:20.776929 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767347    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 21 01:36:20.776929 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767350    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 21 01:36:20.776929 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767353    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 21 01:36:20.776929 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767355    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 21 01:36:20.776929 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767358    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 21 01:36:20.776929 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767361    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 21 01:36:20.776929 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767364    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 21 01:36:20.776929 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767366    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 21 01:36:20.776929 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767369    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 21 01:36:20.776929 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767371    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 21 01:36:20.776929 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767374    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 21 01:36:20.776929 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767377    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 21 01:36:20.776929 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767379    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 21 01:36:20.776929 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767382    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 21 01:36:20.776929 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767385    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 21 01:36:20.776929 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767388    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 21 01:36:20.777411 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767392    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 21 01:36:20.777411 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767395    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 21 01:36:20.777411 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767399    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 21 01:36:20.777411 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767401    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 21 01:36:20.777411 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767404    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 21 01:36:20.777411 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767407    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 21 01:36:20.777411 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767410    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 21 01:36:20.777411 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767413    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 21 01:36:20.777411 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767415    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 21 01:36:20.777411 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767418    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 21 01:36:20.777411 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767422    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 21 01:36:20.777411 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767425    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 21 01:36:20.777411 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767428    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 21 01:36:20.777411 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767431    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 21 01:36:20.777411 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767434    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 21 01:36:20.777411 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767436    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 21 01:36:20.777411 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767439    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 21 01:36:20.777411 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767442    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 21 01:36:20.777411 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767444    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 21 01:36:20.777963 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:20.767447    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 21 01:36:20.777963 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.767452    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 21 01:36:20.777963 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.767577    2575 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 21 01:36:20.777963 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.770045    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 21 01:36:20.777963 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.772109    2575 server.go:1019] "Starting client certificate rotation"
Jul 21 01:36:20.777963 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.772216    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 21 01:36:20.777963 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.772258    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 21 01:36:20.777963 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.774968    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 21 01:36:20.777963 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.776217    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 21 01:36:20.786087 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.786069    2575 log.go:25] "Validated CRI v1 runtime API"
Jul 21 01:36:20.788216 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.788196    2575 log.go:25] "Validated CRI v1 image API"
Jul 21 01:36:20.789059 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.789038    2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 21 01:36:20.794351 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.794331    2575 fs.go:135] Filesystem UUIDs: map[136628f9-886b-44f2-8d27-4a0c1f8753f5:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 af240dba-d2b3-4ebe-b6e4-cb81e130c836:/dev/nvme0n1p3]
Jul 21 01:36:20.794422 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.794350    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 21 01:36:20.800966 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.800833    2575 manager.go:216] Machine: {Timestamp:2026-07-21 01:36:20.79919403 +0000 UTC m=+0.396834377 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099929 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2093331c7da8c56cc7488351385eae SystemUUID:ec209333-1c7d-a8c5-6cc7-488351385eae BootID:2761434d-6547-4673-bbc6-bbd933a4aae3 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:fa:f2:92:8f:25 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:fa:f2:92:8f:25 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:f0:25:a5:b0:56 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 21 01:36:20.800966 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.800952    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 21 01:36:20.801087 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.801032    2575 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 21 01:36:20.802597 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.802578    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 21 01:36:20.804334 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.804305    2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 21 01:36:20.804473 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.804337    2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-219.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 21 01:36:20.805720 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.805709    2575 topology_manager.go:138] "Creating topology manager with none policy"
Jul 21 01:36:20.805768 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.805723    2575 container_manager_linux.go:309] "Creating device plugin manager"
Jul 21 01:36:20.805768 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.805731    2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 21 01:36:20.806354 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.806343    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 21 01:36:20.807776 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.807765    2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 21 01:36:20.811782 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.811769    2575 state_mem.go:36] "Initialized new in-memory state store"
Jul 21 01:36:20.811898 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.811887    2575 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 21 01:36:20.811939 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.811924    2575 kubelet.go:486] "Attempting to sync node with API server"
Jul 21 01:36:20.811969 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.811940    2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 21 01:36:20.811969 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.811955    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 21 01:36:20.811969 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.811965    2575 kubelet.go:398] "Adding apiserver pod source"
Jul 21 01:36:20.812065 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.811979    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 21 01:36:20.812317 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.812305    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 21 01:36:20.813604 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.813578    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 21 01:36:20.818388 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.818369    2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 21 01:36:20.819951 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.819934    2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 21 01:36:20.820036 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.819975    2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 21 01:36:20.820857 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.820844    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 21 01:36:20.820956 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.820866    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 21 01:36:20.820956 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.820876    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 21 01:36:20.820956 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.820884    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 21 01:36:20.820956 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.820899    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 21 01:36:20.820956 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.820907    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 21 01:36:20.820956 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.820921    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 21 01:36:20.820956 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.820929    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 21 01:36:20.820956 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.820940    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 21 01:36:20.820956 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.820949    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 21 01:36:20.821345 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.820962    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 21 01:36:20.821345 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.820971    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 21 01:36:20.821345 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.821000    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 21 01:36:20.821345 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.821009    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 21 01:36:20.824297 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:20.824268    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-219.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 21 01:36:20.824297 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:20.824274    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 21 01:36:20.824447 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.824400    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-219.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 21 01:36:20.824507 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.824482    2575 server.go:1268] "Started kubelet"
Jul 21 01:36:20.824611 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.824585    2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 21 01:36:20.824611 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.824571    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 21 01:36:20.824735 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.824645    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 21 01:36:20.825434 ip-10-0-129-219 systemd[1]: Started Kubernetes Kubelet.
Jul 21 01:36:20.825758 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.825736    2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 21 01:36:20.825871 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.825768    2575 server.go:310] "Adding debug handlers to kubelet server"
Jul 21 01:36:20.825871 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.825803    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gcd9t"
Jul 21 01:36:20.831129 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:20.831109    2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 21 01:36:20.833452 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.833438    2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 21 01:36:20.833515 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.833455    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 21 01:36:20.833610 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:20.833572    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-219.ec2.internal\" not found"
Jul 21 01:36:20.833893 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.833854    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gcd9t"
Jul 21 01:36:20.834314 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.834298    2575 factory.go:55] Registering systemd factory
Jul 21 01:36:20.834314 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.834315    2575 factory.go:223] Registration of the systemd container factory successfully
Jul 21 01:36:20.834445 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.834349    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 21 01:36:20.834445 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.834350    2575 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 21 01:36:20.834445 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.834373    2575 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 21 01:36:20.834585 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.834497    2575 reconstruct.go:97] "Volume reconstruction finished"
Jul 21 01:36:20.834585 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.834531    2575 factory.go:153] Registering CRI-O factory
Jul 21 01:36:20.834585 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.834554    2575 factory.go:223] Registration of the crio container factory successfully
Jul 21 01:36:20.834585 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.834531    2575 reconciler.go:29] "Reconciler: start to sync state"
Jul 21 01:36:20.834779 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.834608    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 21 01:36:20.834779 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.834657    2575 factory.go:103] Registering Raw factory
Jul 21 01:36:20.834779 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.834673    2575 manager.go:1196] Started watching for new ooms in manager
Jul 21 01:36:20.835208 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.835189    2575 manager.go:349] Starting recovery of all containers
Jul 21 01:36:20.840451 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:20.840421    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 21 01:36:20.841574 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:20.840400    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-219.ec2.internal.18c42a39f8fdfd5f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-219.ec2.internal,UID:ip-10-0-129-219.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-219.ec2.internal,},FirstTimestamp:2026-07-21 01:36:20.824456543 +0000 UTC m=+0.422096888,LastTimestamp:2026-07-21 01:36:20.824456543 +0000 UTC m=+0.422096888,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-219.ec2.internal,}"
Jul 21 01:36:20.841780 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:20.841749    2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-219.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 21 01:36:20.841882 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.841775    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 21 01:36:20.847299 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.847271    2575 manager.go:354] Recovery completed
Jul 21 01:36:20.850799 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:20.850776    2575 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 21 01:36:20.854041 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.854028    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 21 01:36:20.855965 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.855951    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasSufficientMemory"
Jul 21 01:36:20.856015 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.855979    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasNoDiskPressure"
Jul 21 01:36:20.856015 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.855989    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasSufficientPID"
Jul 21 01:36:20.856523 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.856508    2575 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 21 01:36:20.856523 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.856521    2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 21 01:36:20.856615 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.856539    2575 state_mem.go:36] "Initialized new in-memory state store"
Jul 21 01:36:20.856615 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.856581    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 21 01:36:20.857357 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.857342    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasSufficientMemory"
Jul 21 01:36:20.857399 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.857370    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasNoDiskPressure"
Jul 21 01:36:20.857399 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.857380    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasSufficientPID"
Jul 21 01:36:20.858731 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.858719    2575 policy_none.go:49] "None policy: Start"
Jul 21 01:36:20.858776 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.858736    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 21 01:36:20.858776 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.858747    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 21 01:36:20.860799 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.860787    2575 policy_none.go:47] "Start"
Jul 21 01:36:20.899428 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.899307    2575 manager.go:337] "Starting Device Plugin manager"
Jul 21 01:36:20.899515 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:20.899450    2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 21 01:36:20.899515 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.899461    2575 server.go:87] "Starting device plugin registration server"
Jul 21 01:36:20.899767 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.899753    2575 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 21 01:36:20.899809 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.899772    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 21 01:36:20.899884 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.899853    2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 21 01:36:20.899993 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.899952    2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 21 01:36:20.899993 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:20.899965    2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 21 01:36:20.900683 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:20.900610    2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 21 01:36:20.900751 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:20.900709    2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-219.ec2.internal\" not found"
Jul 21 01:36:21.001234 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.001168    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 21 01:36:21.003428 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.003410    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasSufficientMemory"
Jul 21 01:36:21.003494 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.003443    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasNoDiskPressure"
Jul 21 01:36:21.003494 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.003482    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasSufficientPID"
Jul 21 01:36:21.003560 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.003524    2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-219.ec2.internal"
Jul 21 01:36:21.012099 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.012070    2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-219.ec2.internal"
Jul 21 01:36:21.012099 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:21.012095    2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-219.ec2.internal\": node \"ip-10-0-129-219.ec2.internal\" not found"
Jul 21 01:36:21.019325 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.019299    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 21 01:36:21.019411 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.019343    2575 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 21 01:36:21.019411 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.019363    2575 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 21 01:36:21.019411 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:21.019399    2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 21 01:36:21.021565 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.021549    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 21 01:36:21.028180 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:21.028150    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-219.ec2.internal\" not found"
Jul 21 01:36:21.119746 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.119707    2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-219.ec2.internal"]
Jul 21 01:36:21.119904 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.119792    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 21 01:36:21.123981 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.123964    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasSufficientMemory"
Jul 21 01:36:21.124052 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.123995    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasNoDiskPressure"
Jul 21 01:36:21.124052 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.124006    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasSufficientPID"
Jul 21 01:36:21.126312 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.126298    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 21 01:36:21.126433 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.126420    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal"
Jul 21 01:36:21.126474 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.126448    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 21 01:36:21.127103 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.127087    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasSufficientMemory"
Jul 21 01:36:21.127103 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.127095    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasSufficientMemory"
Jul 21 01:36:21.127212 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.127115    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasNoDiskPressure"
Jul 21 01:36:21.127212 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.127119    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasNoDiskPressure"
Jul 21 01:36:21.127212 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.127131    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasSufficientPID"
Jul 21 01:36:21.127212 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.127134    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasSufficientPID"
Jul 21 01:36:21.129107 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:21.129090    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-219.ec2.internal\" not found"
Jul 21 01:36:21.129224 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.129211    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-219.ec2.internal"
Jul 21 01:36:21.129272 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.129236    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 21 01:36:21.130025 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.130007    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasSufficientMemory"
Jul 21 01:36:21.130128 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.130035    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasNoDiskPressure"
Jul 21 01:36:21.130128 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.130045    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeHasSufficientPID"
Jul 21 01:36:21.137036 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.137018    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4f87f057fc96d679b5a4e615c23da522-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal\" (UID: \"4f87f057fc96d679b5a4e615c23da522\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal"
Jul 21 01:36:21.137113 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.137043    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f87f057fc96d679b5a4e615c23da522-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal\" (UID: \"4f87f057fc96d679b5a4e615c23da522\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal"
Jul 21 01:36:21.137113 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.137066    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8bb34cb64ea3f44a8e9569f1b50f1dce-config\") pod \"kube-apiserver-proxy-ip-10-0-129-219.ec2.internal\" (UID: \"8bb34cb64ea3f44a8e9569f1b50f1dce\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-219.ec2.internal"
Jul 21 01:36:21.151547 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:21.151521    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-219.ec2.internal\" not found" node="ip-10-0-129-219.ec2.internal"
Jul 21 01:36:21.156210 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:21.156192    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-219.ec2.internal\" not found" node="ip-10-0-129-219.ec2.internal"
Jul 21 01:36:21.229273 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:21.229230    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-219.ec2.internal\" not found"
Jul 21 01:36:21.237585 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.237565    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4f87f057fc96d679b5a4e615c23da522-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal\" (UID: \"4f87f057fc96d679b5a4e615c23da522\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal"
Jul 21 01:36:21.237650 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.237595    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f87f057fc96d679b5a4e615c23da522-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal\" (UID: \"4f87f057fc96d679b5a4e615c23da522\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal"
Jul 21 01:36:21.237705 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.237613    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8bb34cb64ea3f44a8e9569f1b50f1dce-config\") pod \"kube-apiserver-proxy-ip-10-0-129-219.ec2.internal\" (UID: \"8bb34cb64ea3f44a8e9569f1b50f1dce\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-219.ec2.internal"
Jul 21 01:36:21.237705 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.237673    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4f87f057fc96d679b5a4e615c23da522-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal\" (UID: \"4f87f057fc96d679b5a4e615c23da522\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal"
Jul 21 01:36:21.237705 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.237696    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f87f057fc96d679b5a4e615c23da522-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal\" (UID: \"4f87f057fc96d679b5a4e615c23da522\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal"
Jul 21 01:36:21.237801 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.237705    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8bb34cb64ea3f44a8e9569f1b50f1dce-config\") pod \"kube-apiserver-proxy-ip-10-0-129-219.ec2.internal\" (UID: \"8bb34cb64ea3f44a8e9569f1b50f1dce\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-219.ec2.internal"
Jul 21 01:36:21.330507 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:21.330471    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-219.ec2.internal\" not found"
Jul 21 01:36:21.438268 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:21.438220    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-219.ec2.internal\" not found"
Jul 21 01:36:21.453680 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.453647    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal"
Jul 21 01:36:21.460708 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.460679    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-219.ec2.internal"
Jul 21 01:36:21.538784 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:21.538738    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-219.ec2.internal\" not found"
Jul 21 01:36:21.640899 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:21.640818    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-219.ec2.internal\" not found"
Jul 21 01:36:21.741911 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:21.741877    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-219.ec2.internal\" not found"
Jul 21 01:36:21.771112 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.771079    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 21 01:36:21.771571 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.771257    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 21 01:36:21.839175 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.839137    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-20 01:31:21 +0000 UTC" deadline="2028-02-02 20:44:04.922131903 +0000 UTC"
Jul 21 01:36:21.839175 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.839170    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13483h7m43.082965513s"
Jul 21 01:36:21.842734 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:21.842715    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-219.ec2.internal\" not found"
Jul 21 01:36:21.852188 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:21.852166    2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 21 01:36:21.943754 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:21.943682    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-219.ec2.internal\" not found"
Jul 21 01:36:22.044680 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:22.044642    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-219.ec2.internal\" not found"
Jul 21 01:36:22.085276 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:22.085244    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bb34cb64ea3f44a8e9569f1b50f1dce.slice/crio-66cc896543964c0d5a91da9928c012b8a7a18b53f4b31bbb968df6e2a36ef6be WatchSource:0}: Error finding container 66cc896543964c0d5a91da9928c012b8a7a18b53f4b31bbb968df6e2a36ef6be: Status 404 returned error can't find the container with id 66cc896543964c0d5a91da9928c012b8a7a18b53f4b31bbb968df6e2a36ef6be
Jul 21 01:36:22.090570 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.090547    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 21 01:36:22.096921 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:22.096899    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f87f057fc96d679b5a4e615c23da522.slice/crio-2e9b7c27c92c6c631c7aaff702d61a3a6384a8862e04d0952a08f64ae79c0759 WatchSource:0}: Error finding container 2e9b7c27c92c6c631c7aaff702d61a3a6384a8862e04d0952a08f64ae79c0759: Status 404 returned error can't find the container with id 2e9b7c27c92c6c631c7aaff702d61a3a6384a8862e04d0952a08f64ae79c0759
Jul 21 01:36:22.145385 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:22.145340    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-219.ec2.internal\" not found"
Jul 21 01:36:22.216370 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.216282    2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 21 01:36:22.233810 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.233780    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-219.ec2.internal"
Jul 21 01:36:22.245430 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.245411    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 21 01:36:22.248065 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.248045    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal"
Jul 21 01:36:22.261888 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.261870    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 21 01:36:22.317531 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.317503    2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 21 01:36:22.813714 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.813675    2575 apiserver.go:52] "Watching apiserver"
Jul 21 01:36:22.819913 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.819889    2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 21 01:36:22.820702 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.820287    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal","openshift-network-diagnostics/network-check-target-xbk6c","openshift-network-operator/iptables-alerter-dr894","openshift-ovn-kubernetes/ovnkube-node-vzgqj","kube-system/global-pull-secret-syncer-pmhdx","kube-system/konnectivity-agent-25bgn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95vkq","openshift-cluster-node-tuning-operator/tuned-wl7t9","openshift-dns/node-resolver-fw8b6","openshift-multus/multus-additional-cni-plugins-gl7hj","openshift-multus/multus-kr6h6","openshift-multus/network-metrics-daemon-245g5","kube-system/kube-apiserver-proxy-ip-10-0-129-219.ec2.internal","openshift-image-registry/node-ca-kzvp4"]
Jul 21 01:36:22.823422 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.823377    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95vkq"
Jul 21 01:36:22.826190 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.825824    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 21 01:36:22.826190 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.825960    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 21 01:36:22.826190 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.826058    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 21 01:36:22.826406 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.826287    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-p5lx4\""
Jul 21 01:36:22.827086 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.826557    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.829478 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.828425    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 21 01:36:22.829478 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.828725    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 21 01:36:22.829478 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.828983    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 21 01:36:22.829478 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.829327    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dr894"
Jul 21 01:36:22.830146 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.830008    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xbkjd\""
Jul 21 01:36:22.830146 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.830086    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 21 01:36:22.831606 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.831585    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lttgh\""
Jul 21 01:36:22.831719 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.831691    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 21 01:36:22.831927 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.831897    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 21 01:36:22.832001 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.831957    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 21 01:36:22.832001 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.831966    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.835289 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.834700    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:36:22.835289 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:22.834831    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmhdx" podUID="5ee75f94-10ca-4a6a-a2f4-077eb597a94f"
Jul 21 01:36:22.835289 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.834070    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 21 01:36:22.837395 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.835896    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 21 01:36:22.837395 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.836669    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 21 01:36:22.837395 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.836752    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 21 01:36:22.837395 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.836673    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 21 01:36:22.837395 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.836993    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 21 01:36:22.837395 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.837191    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-52l9p\""
Jul 21 01:36:22.837395 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.837363    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 21 01:36:22.841368 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.841231    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-25bgn"
Jul 21 01:36:22.843641 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.843614    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fw8b6"
Jul 21 01:36:22.844415 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.844396    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 21 01:36:22.844497 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.844415    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zdfzr\""
Jul 21 01:36:22.844497 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.844423    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 21 01:36:22.845954 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.845934    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 21 01:36:22.846057 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.846035    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z8wv2\""
Jul 21 01:36:22.846207 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.846189    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 21 01:36:22.846809 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.846551    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kzvp4"
Jul 21 01:36:22.848813 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.848791    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xbk6c"
Jul 21 01:36:22.848920 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.848907    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 21 01:36:22.849032 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:22.848902    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xbk6c" podUID="26238877-47e2-4ed6-8df9-c21f38f7578c"
Jul 21 01:36:22.849032 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.849028    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 21 01:36:22.849202 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.849091    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-log-socket\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.849202 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.849117    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mlcvp\""
Jul 21 01:36:22.849202 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.849124    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-host-cni-bin\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.849202 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.849176    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p49mq\" (UniqueName: \"kubernetes.io/projected/40c4613a-d37f-4b0d-bdfe-88c655071445-kube-api-access-p49mq\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.849410 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.849204    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/771bbef7-a312-4d6f-9091-4a339e0876e5-sys-fs\") pod \"aws-ebs-csi-driver-node-95vkq\" (UID: \"771bbef7-a312-4d6f-9091-4a339e0876e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95vkq"
Jul 21 01:36:22.849410 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.849229    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-system-cni-dir\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.849410 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.849251    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-os-release\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.849410 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.849270    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 21 01:36:22.849410 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.849289    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/40c4613a-d37f-4b0d-bdfe-88c655071445-env-overrides\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.849410 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.849315    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/771bbef7-a312-4d6f-9091-4a339e0876e5-socket-dir\") pod \"aws-ebs-csi-driver-node-95vkq\" (UID: \"771bbef7-a312-4d6f-9091-4a339e0876e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95vkq"
Jul 21 01:36:22.849410 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.849340    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-host-run-multus-certs\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.849410 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.849391    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb395f84-ef3c-486c-852f-19c9954a6715-host-slash\") pod \"iptables-alerter-dr894\" (UID: \"cb395f84-ef3c-486c-852f-19c9954a6715\") " pod="openshift-network-operator/iptables-alerter-dr894"
Jul 21 01:36:22.849832 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.849415    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/771bbef7-a312-4d6f-9091-4a339e0876e5-registration-dir\") pod \"aws-ebs-csi-driver-node-95vkq\" (UID: \"771bbef7-a312-4d6f-9091-4a339e0876e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95vkq"
Jul 21 01:36:22.849832 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.849412    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 21 01:36:22.849832 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.849440    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-host-var-lib-cni-bin\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.849832 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.849470    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-run-systemd\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.849832 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.849516    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-var-lib-openvswitch\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.849832 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.849545    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-etc-openvswitch\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.849832 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.849571    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-run-openvswitch\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.849832 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.849615    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-run-ovn\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.849832 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.849737    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-host-cni-netd\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.849832 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.849778    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.849832 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.849803    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-host-run-netns\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.849832 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.849828    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-host-run-ovn-kubernetes\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.850423 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.849871    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/40c4613a-d37f-4b0d-bdfe-88c655071445-ovn-node-metrics-cert\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.850423 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.849931    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/40c4613a-d37f-4b0d-bdfe-88c655071445-ovnkube-script-lib\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.850423 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.849959    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ee75f94-10ca-4a6a-a2f4-077eb597a94f-kubelet-config\") pod \"global-pull-secret-syncer-pmhdx\" (UID: \"5ee75f94-10ca-4a6a-a2f4-077eb597a94f\") " pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:36:22.850423 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.849983    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ee75f94-10ca-4a6a-a2f4-077eb597a94f-dbus\") pod \"global-pull-secret-syncer-pmhdx\" (UID: \"5ee75f94-10ca-4a6a-a2f4-077eb597a94f\") " pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:36:22.850423 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.850005    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a54c5fd7-029d-4f15-89ee-e46e339b8312-konnectivity-ca\") pod \"konnectivity-agent-25bgn\" (UID: \"a54c5fd7-029d-4f15-89ee-e46e339b8312\") " pod="kube-system/konnectivity-agent-25bgn"
Jul 21 01:36:22.850423 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.850030    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/771bbef7-a312-4d6f-9091-4a339e0876e5-device-dir\") pod \"aws-ebs-csi-driver-node-95vkq\" (UID: \"771bbef7-a312-4d6f-9091-4a339e0876e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95vkq"
Jul 21 01:36:22.850423 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.850054    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbzgc\" (UniqueName: \"kubernetes.io/projected/cb395f84-ef3c-486c-852f-19c9954a6715-kube-api-access-dbzgc\") pod \"iptables-alerter-dr894\" (UID: \"cb395f84-ef3c-486c-852f-19c9954a6715\") " pod="openshift-network-operator/iptables-alerter-dr894"
Jul 21 01:36:22.850423 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.850079    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/40c4613a-d37f-4b0d-bdfe-88c655071445-ovnkube-config\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.850423 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.850104    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ee75f94-10ca-4a6a-a2f4-077eb597a94f-original-pull-secret\") pod \"global-pull-secret-syncer-pmhdx\" (UID: \"5ee75f94-10ca-4a6a-a2f4-077eb597a94f\") " pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:36:22.850423 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.850130    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-cnibin\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.850423 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.850287    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b40e983b-2755-4993-abe2-30e2fc3cec40-cni-binary-copy\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.850423 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.850318    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-multus-socket-dir-parent\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.850423 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.850343    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-host-var-lib-kubelet\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.850423 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.850378    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-hostroot\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.850423 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.850402    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cb395f84-ef3c-486c-852f-19c9954a6715-iptables-alerter-script\") pod \"iptables-alerter-dr894\" (UID: \"cb395f84-ef3c-486c-852f-19c9954a6715\") " pod="openshift-network-operator/iptables-alerter-dr894"
Jul 21 01:36:22.851028 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.850448    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-systemd-units\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.851028 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.850501    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-node-log\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.851028 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.850525    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/771bbef7-a312-4d6f-9091-4a339e0876e5-etc-selinux\") pod \"aws-ebs-csi-driver-node-95vkq\" (UID: \"771bbef7-a312-4d6f-9091-4a339e0876e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95vkq"
Jul 21 01:36:22.851028 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.850561    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f92sr\" (UniqueName: \"kubernetes.io/projected/771bbef7-a312-4d6f-9091-4a339e0876e5-kube-api-access-f92sr\") pod \"aws-ebs-csi-driver-node-95vkq\" (UID: \"771bbef7-a312-4d6f-9091-4a339e0876e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95vkq"
Jul 21 01:36:22.851028 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.850597    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-multus-cni-dir\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.851028 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.850622    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-host-run-netns\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.851028 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.850655    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-host-var-lib-cni-multus\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.851028 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.850714    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a54c5fd7-029d-4f15-89ee-e46e339b8312-agent-certs\") pod \"konnectivity-agent-25bgn\" (UID: \"a54c5fd7-029d-4f15-89ee-e46e339b8312\") " pod="kube-system/konnectivity-agent-25bgn"
Jul 21 01:36:22.851028 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.850743    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/771bbef7-a312-4d6f-9091-4a339e0876e5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-95vkq\" (UID: \"771bbef7-a312-4d6f-9091-4a339e0876e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95vkq"
Jul 21 01:36:22.851028 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.850784    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-host-run-k8s-cni-cncf-io\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.851028 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.850827    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-multus-conf-dir\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.851028 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.850855    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b40e983b-2755-4993-abe2-30e2fc3cec40-multus-daemon-config\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.851028 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.850888    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-etc-kubernetes\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.851028 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.850913    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgxp5\" (UniqueName: \"kubernetes.io/projected/b40e983b-2755-4993-abe2-30e2fc3cec40-kube-api-access-cgxp5\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.851028 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.850942    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-host-kubelet\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.851028 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.850973    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-host-slash\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.851815 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.851334    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gl7hj"
Jul 21 01:36:22.853825 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.853800    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 21 01:36:22.854087 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.854070    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 21 01:36:22.854465 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.854444    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g855s\""
Jul 21 01:36:22.854554 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.854516    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:22.856455 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.856434    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245g5"
Jul 21 01:36:22.856558 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:22.856500    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-245g5" podUID="12e288ad-0d62-4e00-9a2a-1de85626abb9"
Jul 21 01:36:22.856682 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.856629    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-c5k5h\""
Jul 21 01:36:22.856682 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.856629    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 21 01:36:22.856682 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.856661    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 21 01:36:22.872543 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.872522    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jkckg"
Jul 21 01:36:22.881129 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.881105    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jkckg"
Jul 21 01:36:22.935991 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.935899    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 21 01:36:22.951268 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.951229    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b40e983b-2755-4993-abe2-30e2fc3cec40-multus-daemon-config\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.951268 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.951278    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ee75f94-10ca-4a6a-a2f4-077eb597a94f-original-pull-secret\") pod \"global-pull-secret-syncer-pmhdx\" (UID: \"5ee75f94-10ca-4a6a-a2f4-077eb597a94f\") " pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:36:22.951530 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.951316    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-etc-sysconfig\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:22.951530 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.951341    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-var-lib-kubelet\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:22.951530 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.951363    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-etc-tuned\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:22.951530 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.951390    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cb395f84-ef3c-486c-852f-19c9954a6715-iptables-alerter-script\") pod \"iptables-alerter-dr894\" (UID: \"cb395f84-ef3c-486c-852f-19c9954a6715\") " pod="openshift-network-operator/iptables-alerter-dr894"
Jul 21 01:36:22.951530 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.951416    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-node-log\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.951530 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.951447    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-multus-conf-dir\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.951530 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.951472    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed12def9-d42e-4d3e-98b2-d870a1c43231-hosts-file\") pod \"node-resolver-fw8b6\" (UID: \"ed12def9-d42e-4d3e-98b2-d870a1c43231\") " pod="openshift-dns/node-resolver-fw8b6"
Jul 21 01:36:22.951530 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.951497    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed12def9-d42e-4d3e-98b2-d870a1c43231-tmp-dir\") pod \"node-resolver-fw8b6\" (UID: \"ed12def9-d42e-4d3e-98b2-d870a1c43231\") " pod="openshift-dns/node-resolver-fw8b6"
Jul 21 01:36:22.951530 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.951521    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7073f999-4de0-4a60-b3a2-58e9e527ffea-serviceca\") pod \"node-ca-kzvp4\" (UID: \"7073f999-4de0-4a60-b3a2-58e9e527ffea\") " pod="openshift-image-registry/node-ca-kzvp4"
Jul 21 01:36:22.952015 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.951546    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/84e8ef89-8a9f-4f94-9d58-3f650d213260-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gl7hj\" (UID: \"84e8ef89-8a9f-4f94-9d58-3f650d213260\") " pod="openshift-multus/multus-additional-cni-plugins-gl7hj"
Jul 21 01:36:22.952015 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.951575    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/771bbef7-a312-4d6f-9091-4a339e0876e5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-95vkq\" (UID: \"771bbef7-a312-4d6f-9091-4a339e0876e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95vkq"
Jul 21 01:36:22.952015 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.951609    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-sys\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:22.952015 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.951631    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-host-kubelet\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.952015 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.951668    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-host-slash\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.952015 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.951689    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-host-cni-bin\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.952015 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.951710    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-system-cni-dir\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.952015 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.951735    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9c5p\" (UniqueName: \"kubernetes.io/projected/84e8ef89-8a9f-4f94-9d58-3f650d213260-kube-api-access-n9c5p\") pod \"multus-additional-cni-plugins-gl7hj\" (UID: \"84e8ef89-8a9f-4f94-9d58-3f650d213260\") " pod="openshift-multus/multus-additional-cni-plugins-gl7hj"
Jul 21 01:36:22.952015 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.951745    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-multus-conf-dir\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.952015 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.951761    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-etc-kubernetes\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:22.952015 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.951786    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnp9g\" (UniqueName: \"kubernetes.io/projected/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-kube-api-access-wnp9g\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:22.952015 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.951812    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/771bbef7-a312-4d6f-9091-4a339e0876e5-socket-dir\") pod \"aws-ebs-csi-driver-node-95vkq\" (UID: \"771bbef7-a312-4d6f-9091-4a339e0876e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95vkq"
Jul 21 01:36:22.952015 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.951836    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdk4s\" (UniqueName: \"kubernetes.io/projected/ed12def9-d42e-4d3e-98b2-d870a1c43231-kube-api-access-fdk4s\") pod \"node-resolver-fw8b6\" (UID: \"ed12def9-d42e-4d3e-98b2-d870a1c43231\") " pod="openshift-dns/node-resolver-fw8b6"
Jul 21 01:36:22.952015 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.951863    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/84e8ef89-8a9f-4f94-9d58-3f650d213260-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gl7hj\" (UID: \"84e8ef89-8a9f-4f94-9d58-3f650d213260\") " pod="openshift-multus/multus-additional-cni-plugins-gl7hj"
Jul 21 01:36:22.952015 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.951892    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb395f84-ef3c-486c-852f-19c9954a6715-host-slash\") pod \"iptables-alerter-dr894\" (UID: \"cb395f84-ef3c-486c-852f-19c9954a6715\") " pod="openshift-network-operator/iptables-alerter-dr894"
Jul 21 01:36:22.952015 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.951916    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/771bbef7-a312-4d6f-9091-4a339e0876e5-registration-dir\") pod \"aws-ebs-csi-driver-node-95vkq\" (UID: \"771bbef7-a312-4d6f-9091-4a339e0876e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95vkq"
Jul 21 01:36:22.952805 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.951940    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b97gz\" (UniqueName: \"kubernetes.io/projected/7073f999-4de0-4a60-b3a2-58e9e527ffea-kube-api-access-b97gz\") pod \"node-ca-kzvp4\" (UID: \"7073f999-4de0-4a60-b3a2-58e9e527ffea\") " pod="openshift-image-registry/node-ca-kzvp4"
Jul 21 01:36:22.952805 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.951963    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-etc-sysctl-conf\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:22.952805 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.951988    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-var-lib-openvswitch\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.952805 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.952024    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-host-cni-netd\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.952805 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.952057    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/40c4613a-d37f-4b0d-bdfe-88c655071445-ovnkube-script-lib\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.952805 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.952085    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a54c5fd7-029d-4f15-89ee-e46e339b8312-konnectivity-ca\") pod \"konnectivity-agent-25bgn\" (UID: \"a54c5fd7-029d-4f15-89ee-e46e339b8312\") " pod="kube-system/konnectivity-agent-25bgn"
Jul 21 01:36:22.952805 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.952165    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/771bbef7-a312-4d6f-9091-4a339e0876e5-device-dir\") pod \"aws-ebs-csi-driver-node-95vkq\" (UID: \"771bbef7-a312-4d6f-9091-4a339e0876e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95vkq"
Jul 21 01:36:22.952805 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.952251    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/771bbef7-a312-4d6f-9091-4a339e0876e5-device-dir\") pod \"aws-ebs-csi-driver-node-95vkq\" (UID: \"771bbef7-a312-4d6f-9091-4a339e0876e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95vkq"
Jul 21 01:36:22.952805 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.952248    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/771bbef7-a312-4d6f-9091-4a339e0876e5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-95vkq\" (UID: \"771bbef7-a312-4d6f-9091-4a339e0876e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95vkq"
Jul 21 01:36:22.952805 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.952287    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-host-kubelet\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.952805 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.952306    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-host-slash\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.952805 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.952341    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb395f84-ef3c-486c-852f-19c9954a6715-host-slash\") pod \"iptables-alerter-dr894\" (UID: \"cb395f84-ef3c-486c-852f-19c9954a6715\") " pod="openshift-network-operator/iptables-alerter-dr894"
Jul 21 01:36:22.952805 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.952371    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b40e983b-2755-4993-abe2-30e2fc3cec40-multus-daemon-config\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.952805 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.952385    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-system-cni-dir\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.952805 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.952426    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/771bbef7-a312-4d6f-9091-4a339e0876e5-registration-dir\") pod \"aws-ebs-csi-driver-node-95vkq\" (UID: \"771bbef7-a312-4d6f-9091-4a339e0876e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95vkq"
Jul 21 01:36:22.952805 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.952438    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/771bbef7-a312-4d6f-9091-4a339e0876e5-socket-dir\") pod \"aws-ebs-csi-driver-node-95vkq\" (UID: \"771bbef7-a312-4d6f-9091-4a339e0876e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95vkq"
Jul 21 01:36:22.952805 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.952507    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/771bbef7-a312-4d6f-9091-4a339e0876e5-etc-selinux\") pod \"aws-ebs-csi-driver-node-95vkq\" (UID: \"771bbef7-a312-4d6f-9091-4a339e0876e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95vkq"
Jul 21 01:36:22.953352 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.952548    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/84e8ef89-8a9f-4f94-9d58-3f650d213260-cni-binary-copy\") pod \"multus-additional-cni-plugins-gl7hj\" (UID: \"84e8ef89-8a9f-4f94-9d58-3f650d213260\") " pod="openshift-multus/multus-additional-cni-plugins-gl7hj"
Jul 21 01:36:22.953352 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.952587    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-run\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:22.953352 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.952616    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-host-run-netns\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.953352 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.952709    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a54c5fd7-029d-4f15-89ee-e46e339b8312-konnectivity-ca\") pod \"konnectivity-agent-25bgn\" (UID: \"a54c5fd7-029d-4f15-89ee-e46e339b8312\") " pod="kube-system/konnectivity-agent-25bgn"
Jul 21 01:36:22.953352 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.952651    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b40e983b-2755-4993-abe2-30e2fc3cec40-cni-binary-copy\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.953352 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.952772    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-multus-socket-dir-parent\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.953352 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.952800    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7073f999-4de0-4a60-b3a2-58e9e527ffea-host\") pod \"node-ca-kzvp4\" (UID: \"7073f999-4de0-4a60-b3a2-58e9e527ffea\") " pod="openshift-image-registry/node-ca-kzvp4"
Jul 21 01:36:22.953352 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:22.952814    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 21 01:36:22.953352 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.952825    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-etc-sysctl-d\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:22.953352 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.952851    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12e288ad-0d62-4e00-9a2a-1de85626abb9-metrics-certs\") pod \"network-metrics-daemon-245g5\" (UID: \"12e288ad-0d62-4e00-9a2a-1de85626abb9\") " pod="openshift-multus/network-metrics-daemon-245g5"
Jul 21 01:36:22.953352 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:22.952912    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ee75f94-10ca-4a6a-a2f4-077eb597a94f-original-pull-secret podName:5ee75f94-10ca-4a6a-a2f4-077eb597a94f nodeName:}" failed. No retries permitted until 2026-07-21 01:36:23.452870949 +0000 UTC m=+3.050511294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ee75f94-10ca-4a6a-a2f4-077eb597a94f-original-pull-secret") pod "global-pull-secret-syncer-pmhdx" (UID: "5ee75f94-10ca-4a6a-a2f4-077eb597a94f") : object "kube-system"/"original-pull-secret" not registered
Jul 21 01:36:22.953352 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.952929    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbzgc\" (UniqueName: \"kubernetes.io/projected/cb395f84-ef3c-486c-852f-19c9954a6715-kube-api-access-dbzgc\") pod \"iptables-alerter-dr894\" (UID: \"cb395f84-ef3c-486c-852f-19c9954a6715\") " pod="openshift-network-operator/iptables-alerter-dr894"
Jul 21 01:36:22.953352 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.952937    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cb395f84-ef3c-486c-852f-19c9954a6715-iptables-alerter-script\") pod \"iptables-alerter-dr894\" (UID: \"cb395f84-ef3c-486c-852f-19c9954a6715\") " pod="openshift-network-operator/iptables-alerter-dr894"
Jul 21 01:36:22.953352 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.952956    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/40c4613a-d37f-4b0d-bdfe-88c655071445-ovnkube-config\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.953352 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.952997    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-etc-kubernetes\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.953352 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.953002    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-host-cni-bin\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.953352 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.953024    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/84e8ef89-8a9f-4f94-9d58-3f650d213260-cnibin\") pod \"multus-additional-cni-plugins-gl7hj\" (UID: \"84e8ef89-8a9f-4f94-9d58-3f650d213260\") " pod="openshift-multus/multus-additional-cni-plugins-gl7hj"
Jul 21 01:36:22.953966 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.953131    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/40c4613a-d37f-4b0d-bdfe-88c655071445-ovnkube-script-lib\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.953966 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.953190    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-var-lib-openvswitch\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.953966 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.953294    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/771bbef7-a312-4d6f-9091-4a339e0876e5-etc-selinux\") pod \"aws-ebs-csi-driver-node-95vkq\" (UID: \"771bbef7-a312-4d6f-9091-4a339e0876e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95vkq"
Jul 21 01:36:22.953966 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.953335    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-node-log\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.953966 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.953374    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-host-cni-netd\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.953966 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.953399    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-multus-socket-dir-parent\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.953966 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.953481    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-etc-kubernetes\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.953966 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.953498    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-host-run-netns\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.953966 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.953520    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-etc-systemd\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:22.953966 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.953538    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b40e983b-2755-4993-abe2-30e2fc3cec40-cni-binary-copy\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.953966 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.953546    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/40c4613a-d37f-4b0d-bdfe-88c655071445-ovnkube-config\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.953966 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.953583    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rqcn\" (UniqueName: \"kubernetes.io/projected/12e288ad-0d62-4e00-9a2a-1de85626abb9-kube-api-access-5rqcn\") pod \"network-metrics-daemon-245g5\" (UID: \"12e288ad-0d62-4e00-9a2a-1de85626abb9\") " pod="openshift-multus/network-metrics-daemon-245g5"
Jul 21 01:36:22.953966 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.953625    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-systemd-units\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.953966 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.953668    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-systemd-units\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.953966 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.953729    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f92sr\" (UniqueName: \"kubernetes.io/projected/771bbef7-a312-4d6f-9091-4a339e0876e5-kube-api-access-f92sr\") pod \"aws-ebs-csi-driver-node-95vkq\" (UID: \"771bbef7-a312-4d6f-9091-4a339e0876e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95vkq"
Jul 21 01:36:22.953966 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.953836    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-host-run-netns\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.953966 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.953899    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-host-run-netns\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.954725 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.953938    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/84e8ef89-8a9f-4f94-9d58-3f650d213260-system-cni-dir\") pod \"multus-additional-cni-plugins-gl7hj\" (UID: \"84e8ef89-8a9f-4f94-9d58-3f650d213260\") " pod="openshift-multus/multus-additional-cni-plugins-gl7hj"
Jul 21 01:36:22.954725 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.953968    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-etc-modprobe-d\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:22.954725 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.954010    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-tmp\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:22.954725 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.954054    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a54c5fd7-029d-4f15-89ee-e46e339b8312-agent-certs\") pod \"konnectivity-agent-25bgn\" (UID: \"a54c5fd7-029d-4f15-89ee-e46e339b8312\") " pod="kube-system/konnectivity-agent-25bgn"
Jul 21 01:36:22.954725 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.954092    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-host-run-k8s-cni-cncf-io\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.954725 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.954119    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgxp5\" (UniqueName: \"kubernetes.io/projected/b40e983b-2755-4993-abe2-30e2fc3cec40-kube-api-access-cgxp5\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.954725 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.954206    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpxtj\" (UniqueName: \"kubernetes.io/projected/26238877-47e2-4ed6-8df9-c21f38f7578c-kube-api-access-dpxtj\") pod \"network-check-target-xbk6c\" (UID: \"26238877-47e2-4ed6-8df9-c21f38f7578c\") " pod="openshift-network-diagnostics/network-check-target-xbk6c"
Jul 21 01:36:22.954725 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.954276    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-lib-modules\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:22.954725 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.954267    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-host-run-k8s-cni-cncf-io\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.954725 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.954322    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-log-socket\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.954725 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.954352    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p49mq\" (UniqueName: \"kubernetes.io/projected/40c4613a-d37f-4b0d-bdfe-88c655071445-kube-api-access-p49mq\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.954725 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.954395    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/771bbef7-a312-4d6f-9091-4a339e0876e5-sys-fs\") pod \"aws-ebs-csi-driver-node-95vkq\" (UID: \"771bbef7-a312-4d6f-9091-4a339e0876e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95vkq"
Jul 21 01:36:22.954725 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.954419    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-os-release\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.954725 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.954429    2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 21 01:36:22.954725 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.954492    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/771bbef7-a312-4d6f-9091-4a339e0876e5-sys-fs\") pod \"aws-ebs-csi-driver-node-95vkq\" (UID: \"771bbef7-a312-4d6f-9091-4a339e0876e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95vkq"
Jul 21 01:36:22.954725 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.954543    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-host-run-multus-certs\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.954725 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.954568    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-os-release\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.954725 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.954611    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-log-socket\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.955297 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.954662    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/84e8ef89-8a9f-4f94-9d58-3f650d213260-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gl7hj\" (UID: \"84e8ef89-8a9f-4f94-9d58-3f650d213260\") " pod="openshift-multus/multus-additional-cni-plugins-gl7hj"
Jul 21 01:36:22.955297 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.954719    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-host\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:22.955297 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.954766    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-host-run-multus-certs\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.955297 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.954820    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/40c4613a-d37f-4b0d-bdfe-88c655071445-env-overrides\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.955297 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.954862    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-host-var-lib-cni-bin\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.955297 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.954922    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/84e8ef89-8a9f-4f94-9d58-3f650d213260-os-release\") pod \"multus-additional-cni-plugins-gl7hj\" (UID: \"84e8ef89-8a9f-4f94-9d58-3f650d213260\") " pod="openshift-multus/multus-additional-cni-plugins-gl7hj"
Jul 21 01:36:22.955297 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.954971    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-host-var-lib-cni-bin\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.955297 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.955011    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-run-systemd\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.955297 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.955057    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-etc-openvswitch\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.955297 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.955074    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-run-systemd\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.955297 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.955092    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-run-openvswitch\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.955297 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.955123    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-run-ovn\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.955297 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.955148    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.955297 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.955158    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-etc-openvswitch\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.955297 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.955167    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ee75f94-10ca-4a6a-a2f4-077eb597a94f-kubelet-config\") pod \"global-pull-secret-syncer-pmhdx\" (UID: \"5ee75f94-10ca-4a6a-a2f4-077eb597a94f\") " pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:36:22.955297 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.955187    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ee75f94-10ca-4a6a-a2f4-077eb597a94f-dbus\") pod \"global-pull-secret-syncer-pmhdx\" (UID: \"5ee75f94-10ca-4a6a-a2f4-077eb597a94f\") " pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:36:22.955297 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.955219    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-host-run-ovn-kubernetes\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.956064 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.955224    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/40c4613a-d37f-4b0d-bdfe-88c655071445-env-overrides\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.956064 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.955271    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/40c4613a-d37f-4b0d-bdfe-88c655071445-ovn-node-metrics-cert\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.956064 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.955288    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-run-openvswitch\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.956064 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.955304    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ee75f94-10ca-4a6a-a2f4-077eb597a94f-dbus\") pod \"global-pull-secret-syncer-pmhdx\" (UID: \"5ee75f94-10ca-4a6a-a2f4-077eb597a94f\") " pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:36:22.956064 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.955304    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-cnibin\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.956064 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.955340    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-host-var-lib-kubelet\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.956064 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.955343    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-run-ovn\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.956064 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.955357    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-hostroot\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.956064 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.955354    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.956064 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.955375    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-multus-cni-dir\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.956064 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.955394    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ee75f94-10ca-4a6a-a2f4-077eb597a94f-kubelet-config\") pod \"global-pull-secret-syncer-pmhdx\" (UID: \"5ee75f94-10ca-4a6a-a2f4-077eb597a94f\") " pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:36:22.956064 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.955400    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40c4613a-d37f-4b0d-bdfe-88c655071445-host-run-ovn-kubernetes\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.956064 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.955405    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-hostroot\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.956064 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.955379    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-cnibin\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.956064 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.955424    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-host-var-lib-cni-multus\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.956064 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.955420    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-host-var-lib-kubelet\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.956064 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.955485    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-multus-cni-dir\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.956064 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.955534    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b40e983b-2755-4993-abe2-30e2fc3cec40-host-var-lib-cni-multus\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.959247 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.959217    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/40c4613a-d37f-4b0d-bdfe-88c655071445-ovn-node-metrics-cert\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:22.959421 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.959265    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a54c5fd7-029d-4f15-89ee-e46e339b8312-agent-certs\") pod \"konnectivity-agent-25bgn\" (UID: \"a54c5fd7-029d-4f15-89ee-e46e339b8312\") " pod="kube-system/konnectivity-agent-25bgn"
Jul 21 01:36:22.961232 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.961213    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbzgc\" (UniqueName: \"kubernetes.io/projected/cb395f84-ef3c-486c-852f-19c9954a6715-kube-api-access-dbzgc\") pod \"iptables-alerter-dr894\" (UID: \"cb395f84-ef3c-486c-852f-19c9954a6715\") " pod="openshift-network-operator/iptables-alerter-dr894"
Jul 21 01:36:22.962182 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.962130    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f92sr\" (UniqueName: \"kubernetes.io/projected/771bbef7-a312-4d6f-9091-4a339e0876e5-kube-api-access-f92sr\") pod \"aws-ebs-csi-driver-node-95vkq\" (UID: \"771bbef7-a312-4d6f-9091-4a339e0876e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95vkq"
Jul 21 01:36:22.962182 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.962146    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgxp5\" (UniqueName: \"kubernetes.io/projected/b40e983b-2755-4993-abe2-30e2fc3cec40-kube-api-access-cgxp5\") pod \"multus-kr6h6\" (UID: \"b40e983b-2755-4993-abe2-30e2fc3cec40\") " pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:22.963143 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:22.963116    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p49mq\" (UniqueName: \"kubernetes.io/projected/40c4613a-d37f-4b0d-bdfe-88c655071445-kube-api-access-p49mq\") pod \"ovnkube-node-vzgqj\" (UID: \"40c4613a-d37f-4b0d-bdfe-88c655071445\") " pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:23.024127 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.024074    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal" event={"ID":"4f87f057fc96d679b5a4e615c23da522","Type":"ContainerStarted","Data":"2e9b7c27c92c6c631c7aaff702d61a3a6384a8862e04d0952a08f64ae79c0759"}
Jul 21 01:36:23.025401 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.025376    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-219.ec2.internal" event={"ID":"8bb34cb64ea3f44a8e9569f1b50f1dce","Type":"ContainerStarted","Data":"66cc896543964c0d5a91da9928c012b8a7a18b53f4b31bbb968df6e2a36ef6be"}
Jul 21 01:36:23.056517 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.056481    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/84e8ef89-8a9f-4f94-9d58-3f650d213260-os-release\") pod \"multus-additional-cni-plugins-gl7hj\" (UID: \"84e8ef89-8a9f-4f94-9d58-3f650d213260\") " pod="openshift-multus/multus-additional-cni-plugins-gl7hj"
Jul 21 01:36:23.056720 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.056538    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-etc-sysconfig\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:23.056720 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.056557    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-var-lib-kubelet\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:23.056720 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.056577    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-etc-tuned\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:23.056720 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.056598    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed12def9-d42e-4d3e-98b2-d870a1c43231-hosts-file\") pod \"node-resolver-fw8b6\" (UID: \"ed12def9-d42e-4d3e-98b2-d870a1c43231\") " pod="openshift-dns/node-resolver-fw8b6"
Jul 21 01:36:23.056720 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.056617    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed12def9-d42e-4d3e-98b2-d870a1c43231-tmp-dir\") pod \"node-resolver-fw8b6\" (UID: \"ed12def9-d42e-4d3e-98b2-d870a1c43231\") " pod="openshift-dns/node-resolver-fw8b6"
Jul 21 01:36:23.056720 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.056659    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7073f999-4de0-4a60-b3a2-58e9e527ffea-serviceca\") pod \"node-ca-kzvp4\" (UID: \"7073f999-4de0-4a60-b3a2-58e9e527ffea\") " pod="openshift-image-registry/node-ca-kzvp4"
Jul 21 01:36:23.056720 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.056686    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/84e8ef89-8a9f-4f94-9d58-3f650d213260-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gl7hj\" (UID: \"84e8ef89-8a9f-4f94-9d58-3f650d213260\") " pod="openshift-multus/multus-additional-cni-plugins-gl7hj"
Jul 21 01:36:23.056720 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.056672    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/84e8ef89-8a9f-4f94-9d58-3f650d213260-os-release\") pod \"multus-additional-cni-plugins-gl7hj\" (UID: \"84e8ef89-8a9f-4f94-9d58-3f650d213260\") " pod="openshift-multus/multus-additional-cni-plugins-gl7hj"
Jul 21 01:36:23.056720 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.056702    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-etc-sysconfig\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:23.056720 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.056714    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-sys\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:23.056720 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.056658    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-var-lib-kubelet\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:23.057262 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.056750    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed12def9-d42e-4d3e-98b2-d870a1c43231-hosts-file\") pod \"node-resolver-fw8b6\" (UID: \"ed12def9-d42e-4d3e-98b2-d870a1c43231\") " pod="openshift-dns/node-resolver-fw8b6"
Jul 21 01:36:23.057262 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.056763    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9c5p\" (UniqueName: \"kubernetes.io/projected/84e8ef89-8a9f-4f94-9d58-3f650d213260-kube-api-access-n9c5p\") pod \"multus-additional-cni-plugins-gl7hj\" (UID: \"84e8ef89-8a9f-4f94-9d58-3f650d213260\") " pod="openshift-multus/multus-additional-cni-plugins-gl7hj"
Jul 21 01:36:23.057262 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.056785    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-sys\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:23.057262 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.056805    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-etc-kubernetes\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:23.057262 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.056840    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnp9g\" (UniqueName: \"kubernetes.io/projected/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-kube-api-access-wnp9g\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:23.057262 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.056871    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdk4s\" (UniqueName: \"kubernetes.io/projected/ed12def9-d42e-4d3e-98b2-d870a1c43231-kube-api-access-fdk4s\") pod \"node-resolver-fw8b6\" (UID: \"ed12def9-d42e-4d3e-98b2-d870a1c43231\") " pod="openshift-dns/node-resolver-fw8b6"
Jul 21 01:36:23.057262 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.056899    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/84e8ef89-8a9f-4f94-9d58-3f650d213260-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gl7hj\" (UID: \"84e8ef89-8a9f-4f94-9d58-3f650d213260\") " pod="openshift-multus/multus-additional-cni-plugins-gl7hj"
Jul 21 01:36:23.057262 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.056919    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-etc-kubernetes\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:23.057262 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.056930    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b97gz\" (UniqueName: \"kubernetes.io/projected/7073f999-4de0-4a60-b3a2-58e9e527ffea-kube-api-access-b97gz\") pod \"node-ca-kzvp4\" (UID: \"7073f999-4de0-4a60-b3a2-58e9e527ffea\") " pod="openshift-image-registry/node-ca-kzvp4"
Jul 21 01:36:23.057262 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.056958    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-etc-sysctl-conf\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:23.057262 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.056991    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/84e8ef89-8a9f-4f94-9d58-3f650d213260-cni-binary-copy\") pod \"multus-additional-cni-plugins-gl7hj\" (UID: \"84e8ef89-8a9f-4f94-9d58-3f650d213260\") " pod="openshift-multus/multus-additional-cni-plugins-gl7hj"
Jul 21 01:36:23.057262 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.057016    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-run\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:23.057262 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.057043    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7073f999-4de0-4a60-b3a2-58e9e527ffea-host\") pod \"node-ca-kzvp4\" (UID: \"7073f999-4de0-4a60-b3a2-58e9e527ffea\") " pod="openshift-image-registry/node-ca-kzvp4"
Jul 21 01:36:23.057262 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.057067    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-etc-sysctl-d\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:23.057262 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.057094    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12e288ad-0d62-4e00-9a2a-1de85626abb9-metrics-certs\") pod \"network-metrics-daemon-245g5\" (UID: \"12e288ad-0d62-4e00-9a2a-1de85626abb9\") " pod="openshift-multus/network-metrics-daemon-245g5"
Jul 21 01:36:23.057262 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.057124    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/84e8ef89-8a9f-4f94-9d58-3f650d213260-cnibin\") pod \"multus-additional-cni-plugins-gl7hj\" (UID: \"84e8ef89-8a9f-4f94-9d58-3f650d213260\") " pod="openshift-multus/multus-additional-cni-plugins-gl7hj"
Jul 21 01:36:23.057262 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.057148    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-etc-systemd\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:23.058064 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.057174    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rqcn\" (UniqueName: \"kubernetes.io/projected/12e288ad-0d62-4e00-9a2a-1de85626abb9-kube-api-access-5rqcn\") pod \"network-metrics-daemon-245g5\" (UID: \"12e288ad-0d62-4e00-9a2a-1de85626abb9\") " pod="openshift-multus/network-metrics-daemon-245g5"
Jul 21 01:36:23.058064 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.057186    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-etc-sysctl-conf\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:23.058064 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.057203    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/84e8ef89-8a9f-4f94-9d58-3f650d213260-system-cni-dir\") pod \"multus-additional-cni-plugins-gl7hj\" (UID: \"84e8ef89-8a9f-4f94-9d58-3f650d213260\") " pod="openshift-multus/multus-additional-cni-plugins-gl7hj"
Jul 21 01:36:23.058064 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.057286    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/84e8ef89-8a9f-4f94-9d58-3f650d213260-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gl7hj\" (UID: \"84e8ef89-8a9f-4f94-9d58-3f650d213260\") " pod="openshift-multus/multus-additional-cni-plugins-gl7hj"
Jul 21 01:36:23.058064 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.057651    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/84e8ef89-8a9f-4f94-9d58-3f650d213260-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gl7hj\" (UID: \"84e8ef89-8a9f-4f94-9d58-3f650d213260\") " pod="openshift-multus/multus-additional-cni-plugins-gl7hj"
Jul 21 01:36:23.058064 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.057227    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7073f999-4de0-4a60-b3a2-58e9e527ffea-serviceca\") pod \"node-ca-kzvp4\" (UID: \"7073f999-4de0-4a60-b3a2-58e9e527ffea\") " pod="openshift-image-registry/node-ca-kzvp4"
Jul 21 01:36:23.058064 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.057290    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-etc-modprobe-d\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:23.058064 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.057770    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-etc-modprobe-d\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:23.058064 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.057782    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-tmp\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:23.058064 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.057859    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpxtj\" (UniqueName: \"kubernetes.io/projected/26238877-47e2-4ed6-8df9-c21f38f7578c-kube-api-access-dpxtj\") pod \"network-check-target-xbk6c\" (UID: \"26238877-47e2-4ed6-8df9-c21f38f7578c\") " pod="openshift-network-diagnostics/network-check-target-xbk6c"
Jul 21 01:36:23.058064 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.057862    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-run\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:23.058064 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.057902    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-lib-modules\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:23.058064 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.057912    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-etc-systemd\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:23.058064 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.057961    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/84e8ef89-8a9f-4f94-9d58-3f650d213260-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gl7hj\" (UID: \"84e8ef89-8a9f-4f94-9d58-3f650d213260\") " pod="openshift-multus/multus-additional-cni-plugins-gl7hj"
Jul 21 01:36:23.058064 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.058026    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-host\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:23.058064 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.058064    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7073f999-4de0-4a60-b3a2-58e9e527ffea-host\") pod \"node-ca-kzvp4\" (UID: \"7073f999-4de0-4a60-b3a2-58e9e527ffea\") " pod="openshift-image-registry/node-ca-kzvp4"
Jul 21 01:36:23.058824 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.058353    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-host\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:23.058824 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.058408    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed12def9-d42e-4d3e-98b2-d870a1c43231-tmp-dir\") pod \"node-resolver-fw8b6\" (UID: \"ed12def9-d42e-4d3e-98b2-d870a1c43231\") " pod="openshift-dns/node-resolver-fw8b6"
Jul 21 01:36:23.058824 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.058515    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/84e8ef89-8a9f-4f94-9d58-3f650d213260-cnibin\") pod \"multus-additional-cni-plugins-gl7hj\" (UID: \"84e8ef89-8a9f-4f94-9d58-3f650d213260\") " pod="openshift-multus/multus-additional-cni-plugins-gl7hj"
Jul 21 01:36:23.058824 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:23.058650    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 01:36:23.059015 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:23.058862    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12e288ad-0d62-4e00-9a2a-1de85626abb9-metrics-certs podName:12e288ad-0d62-4e00-9a2a-1de85626abb9 nodeName:}" failed. No retries permitted until 2026-07-21 01:36:23.55883944 +0000 UTC m=+3.156479786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12e288ad-0d62-4e00-9a2a-1de85626abb9-metrics-certs") pod "network-metrics-daemon-245g5" (UID: "12e288ad-0d62-4e00-9a2a-1de85626abb9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 01:36:23.061868 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.059070    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/84e8ef89-8a9f-4f94-9d58-3f650d213260-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gl7hj\" (UID: \"84e8ef89-8a9f-4f94-9d58-3f650d213260\") " pod="openshift-multus/multus-additional-cni-plugins-gl7hj"
Jul 21 01:36:23.061868 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.059238    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/84e8ef89-8a9f-4f94-9d58-3f650d213260-system-cni-dir\") pod \"multus-additional-cni-plugins-gl7hj\" (UID: \"84e8ef89-8a9f-4f94-9d58-3f650d213260\") " pod="openshift-multus/multus-additional-cni-plugins-gl7hj"
Jul 21 01:36:23.061868 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.059363    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-etc-sysctl-d\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:23.061868 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.059461    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-lib-modules\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:23.061868 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.059543    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/84e8ef89-8a9f-4f94-9d58-3f650d213260-cni-binary-copy\") pod \"multus-additional-cni-plugins-gl7hj\" (UID: \"84e8ef89-8a9f-4f94-9d58-3f650d213260\") " pod="openshift-multus/multus-additional-cni-plugins-gl7hj"
Jul 21 01:36:23.061868 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.059664    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-etc-tuned\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:23.064521 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.064467    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-tmp\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:23.068213 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.068188    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdk4s\" (UniqueName: \"kubernetes.io/projected/ed12def9-d42e-4d3e-98b2-d870a1c43231-kube-api-access-fdk4s\") pod \"node-resolver-fw8b6\" (UID: \"ed12def9-d42e-4d3e-98b2-d870a1c43231\") " pod="openshift-dns/node-resolver-fw8b6"
Jul 21 01:36:23.068361 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.068347    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9c5p\" (UniqueName: \"kubernetes.io/projected/84e8ef89-8a9f-4f94-9d58-3f650d213260-kube-api-access-n9c5p\") pod \"multus-additional-cni-plugins-gl7hj\" (UID: \"84e8ef89-8a9f-4f94-9d58-3f650d213260\") " pod="openshift-multus/multus-additional-cni-plugins-gl7hj"
Jul 21 01:36:23.070354 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:23.070331    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 21 01:36:23.070354 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:23.070356    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 21 01:36:23.070544 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:23.070371    2575 projected.go:196] Error preparing data for projected volume kube-api-access-dpxtj for pod openshift-network-diagnostics/network-check-target-xbk6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 01:36:23.070544 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:23.070419    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26238877-47e2-4ed6-8df9-c21f38f7578c-kube-api-access-dpxtj podName:26238877-47e2-4ed6-8df9-c21f38f7578c nodeName:}" failed. No retries permitted until 2026-07-21 01:36:23.570406941 +0000 UTC m=+3.168047276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dpxtj" (UniqueName: "kubernetes.io/projected/26238877-47e2-4ed6-8df9-c21f38f7578c-kube-api-access-dpxtj") pod "network-check-target-xbk6c" (UID: "26238877-47e2-4ed6-8df9-c21f38f7578c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 01:36:23.071893 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.071846    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b97gz\" (UniqueName: \"kubernetes.io/projected/7073f999-4de0-4a60-b3a2-58e9e527ffea-kube-api-access-b97gz\") pod \"node-ca-kzvp4\" (UID: \"7073f999-4de0-4a60-b3a2-58e9e527ffea\") " pod="openshift-image-registry/node-ca-kzvp4"
Jul 21 01:36:23.072435 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.072415    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnp9g\" (UniqueName: \"kubernetes.io/projected/486d1dc5-263d-4bcf-a2c6-2882c5c7bd40-kube-api-access-wnp9g\") pod \"tuned-wl7t9\" (UID: \"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40\") " pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:23.073114 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.073095    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rqcn\" (UniqueName: \"kubernetes.io/projected/12e288ad-0d62-4e00-9a2a-1de85626abb9-kube-api-access-5rqcn\") pod \"network-metrics-daemon-245g5\" (UID: \"12e288ad-0d62-4e00-9a2a-1de85626abb9\") " pod="openshift-multus/network-metrics-daemon-245g5"
Jul 21 01:36:23.140722 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.140685    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95vkq"
Jul 21 01:36:23.147694 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.147667    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kr6h6"
Jul 21 01:36:23.160837 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.160813    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dr894"
Jul 21 01:36:23.167059 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.167039    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:23.174902 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.174885    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-25bgn"
Jul 21 01:36:23.183715 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.183693    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fw8b6"
Jul 21 01:36:23.191779 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.191757    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kzvp4"
Jul 21 01:36:23.203519 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.203476    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gl7hj"
Jul 21 01:36:23.208448 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.208429    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wl7t9"
Jul 21 01:36:23.343528 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.343454    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 21 01:36:23.461198 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.461152    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ee75f94-10ca-4a6a-a2f4-077eb597a94f-original-pull-secret\") pod \"global-pull-secret-syncer-pmhdx\" (UID: \"5ee75f94-10ca-4a6a-a2f4-077eb597a94f\") " pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:36:23.461368 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:23.461286    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 21 01:36:23.461368 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:23.461358    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ee75f94-10ca-4a6a-a2f4-077eb597a94f-original-pull-secret podName:5ee75f94-10ca-4a6a-a2f4-077eb597a94f nodeName:}" failed. No retries permitted until 2026-07-21 01:36:24.461335804 +0000 UTC m=+4.058976137 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ee75f94-10ca-4a6a-a2f4-077eb597a94f-original-pull-secret") pod "global-pull-secret-syncer-pmhdx" (UID: "5ee75f94-10ca-4a6a-a2f4-077eb597a94f") : object "kube-system"/"original-pull-secret" not registered
Jul 21 01:36:23.561760 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.561720    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12e288ad-0d62-4e00-9a2a-1de85626abb9-metrics-certs\") pod \"network-metrics-daemon-245g5\" (UID: \"12e288ad-0d62-4e00-9a2a-1de85626abb9\") " pod="openshift-multus/network-metrics-daemon-245g5"
Jul 21 01:36:23.561948 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:23.561921    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 01:36:23.562021 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:23.562008    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12e288ad-0d62-4e00-9a2a-1de85626abb9-metrics-certs podName:12e288ad-0d62-4e00-9a2a-1de85626abb9 nodeName:}" failed. No retries permitted until 2026-07-21 01:36:24.561986128 +0000 UTC m=+4.159626463 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12e288ad-0d62-4e00-9a2a-1de85626abb9-metrics-certs") pod "network-metrics-daemon-245g5" (UID: "12e288ad-0d62-4e00-9a2a-1de85626abb9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 01:36:23.662215 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.662140    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpxtj\" (UniqueName: \"kubernetes.io/projected/26238877-47e2-4ed6-8df9-c21f38f7578c-kube-api-access-dpxtj\") pod \"network-check-target-xbk6c\" (UID: \"26238877-47e2-4ed6-8df9-c21f38f7578c\") " pod="openshift-network-diagnostics/network-check-target-xbk6c"
Jul 21 01:36:23.662357 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:23.662269    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 21 01:36:23.662357 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:23.662288    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 21 01:36:23.662357 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:23.662297    2575 projected.go:196] Error preparing data for projected volume kube-api-access-dpxtj for pod openshift-network-diagnostics/network-check-target-xbk6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 01:36:23.662357 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:23.662348    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26238877-47e2-4ed6-8df9-c21f38f7578c-kube-api-access-dpxtj podName:26238877-47e2-4ed6-8df9-c21f38f7578c nodeName:}" failed. No retries permitted until 2026-07-21 01:36:24.66233531 +0000 UTC m=+4.259975641 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dpxtj" (UniqueName: "kubernetes.io/projected/26238877-47e2-4ed6-8df9-c21f38f7578c-kube-api-access-dpxtj") pod "network-check-target-xbk6c" (UID: "26238877-47e2-4ed6-8df9-c21f38f7578c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 01:36:23.884115 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.884055    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 01:31:23 +0000 UTC" deadline="2028-04-14 09:16:32.800929588 +0000 UTC"
Jul 21 01:36:23.884115 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:23.884105    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15199h40m8.916826869s"
Jul 21 01:36:24.020155 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:24.020123    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245g5"
Jul 21 01:36:24.020317 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:24.020246    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-245g5" podUID="12e288ad-0d62-4e00-9a2a-1de85626abb9"
Jul 21 01:36:24.039972 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:24.039929    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e8ef89_8a9f_4f94_9d58_3f650d213260.slice/crio-50504b2896df63e7d261919e8abf7a3e0725f7018e951ee3942fb8e22413ea6f WatchSource:0}: Error finding container 50504b2896df63e7d261919e8abf7a3e0725f7018e951ee3942fb8e22413ea6f: Status 404 returned error can't find the container with id 50504b2896df63e7d261919e8abf7a3e0725f7018e951ee3942fb8e22413ea6f
Jul 21 01:36:24.048327 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:24.047327    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7073f999_4de0_4a60_b3a2_58e9e527ffea.slice/crio-76327c846ae2985c8d707d9415fafcc862c0abc9edf2527ffba737cd7f81faa1 WatchSource:0}: Error finding container 76327c846ae2985c8d707d9415fafcc862c0abc9edf2527ffba737cd7f81faa1: Status 404 returned error can't find the container with id 76327c846ae2985c8d707d9415fafcc862c0abc9edf2527ffba737cd7f81faa1
Jul 21 01:36:24.048327 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:24.047760    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb395f84_ef3c_486c_852f_19c9954a6715.slice/crio-401fd7ba658a1bd162fd42a45b59e8d4947f0c166d8e28c707a36ac80c13ab7a WatchSource:0}: Error finding container 401fd7ba658a1bd162fd42a45b59e8d4947f0c166d8e28c707a36ac80c13ab7a: Status 404 returned error can't find the container with id 401fd7ba658a1bd162fd42a45b59e8d4947f0c166d8e28c707a36ac80c13ab7a
Jul 21 01:36:24.049303 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:24.049275    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda54c5fd7_029d_4f15_89ee_e46e339b8312.slice/crio-8b84c37c5dd8931884cc1dc824c9b77aea04b685c94dc57b584dcca265ad8851 WatchSource:0}: Error finding container 8b84c37c5dd8931884cc1dc824c9b77aea04b685c94dc57b584dcca265ad8851: Status 404 returned error can't find the container with id 8b84c37c5dd8931884cc1dc824c9b77aea04b685c94dc57b584dcca265ad8851
Jul 21 01:36:24.051253 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:24.051227    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod771bbef7_a312_4d6f_9091_4a339e0876e5.slice/crio-98b7ea05dd88d68fe7c2f579613b81be460657969ddd1ff56ada3b3926aa7478 WatchSource:0}: Error finding container 98b7ea05dd88d68fe7c2f579613b81be460657969ddd1ff56ada3b3926aa7478: Status 404 returned error can't find the container with id 98b7ea05dd88d68fe7c2f579613b81be460657969ddd1ff56ada3b3926aa7478
Jul 21 01:36:24.469146 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:24.468834    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ee75f94-10ca-4a6a-a2f4-077eb597a94f-original-pull-secret\") pod \"global-pull-secret-syncer-pmhdx\" (UID: \"5ee75f94-10ca-4a6a-a2f4-077eb597a94f\") " pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:36:24.469319 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:24.469304    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 21 01:36:24.469424 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:24.469379    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ee75f94-10ca-4a6a-a2f4-077eb597a94f-original-pull-secret podName:5ee75f94-10ca-4a6a-a2f4-077eb597a94f nodeName:}" failed. No retries permitted until 2026-07-21 01:36:26.469359602 +0000 UTC m=+6.066999935 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ee75f94-10ca-4a6a-a2f4-077eb597a94f-original-pull-secret") pod "global-pull-secret-syncer-pmhdx" (UID: "5ee75f94-10ca-4a6a-a2f4-077eb597a94f") : object "kube-system"/"original-pull-secret" not registered
Jul 21 01:36:24.570916 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:24.570238    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12e288ad-0d62-4e00-9a2a-1de85626abb9-metrics-certs\") pod \"network-metrics-daemon-245g5\" (UID: \"12e288ad-0d62-4e00-9a2a-1de85626abb9\") " pod="openshift-multus/network-metrics-daemon-245g5"
Jul 21 01:36:24.570916 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:24.570460    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 01:36:24.570916 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:24.570532    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12e288ad-0d62-4e00-9a2a-1de85626abb9-metrics-certs podName:12e288ad-0d62-4e00-9a2a-1de85626abb9 nodeName:}" failed. No retries permitted until 2026-07-21 01:36:26.570510665 +0000 UTC m=+6.168150997 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12e288ad-0d62-4e00-9a2a-1de85626abb9-metrics-certs") pod "network-metrics-daemon-245g5" (UID: "12e288ad-0d62-4e00-9a2a-1de85626abb9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 01:36:24.672105 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:24.671312    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpxtj\" (UniqueName: \"kubernetes.io/projected/26238877-47e2-4ed6-8df9-c21f38f7578c-kube-api-access-dpxtj\") pod \"network-check-target-xbk6c\" (UID: \"26238877-47e2-4ed6-8df9-c21f38f7578c\") " pod="openshift-network-diagnostics/network-check-target-xbk6c"
Jul 21 01:36:24.672105 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:24.671510    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 21 01:36:24.672105 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:24.671534    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 21 01:36:24.672105 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:24.671548    2575 projected.go:196] Error preparing data for projected volume kube-api-access-dpxtj for pod openshift-network-diagnostics/network-check-target-xbk6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 01:36:24.672105 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:24.671650    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26238877-47e2-4ed6-8df9-c21f38f7578c-kube-api-access-dpxtj podName:26238877-47e2-4ed6-8df9-c21f38f7578c nodeName:}" failed. No retries permitted until 2026-07-21 01:36:26.671629225 +0000 UTC m=+6.269269565 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dpxtj" (UniqueName: "kubernetes.io/projected/26238877-47e2-4ed6-8df9-c21f38f7578c-kube-api-access-dpxtj") pod "network-check-target-xbk6c" (UID: "26238877-47e2-4ed6-8df9-c21f38f7578c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 01:36:24.885286 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:24.885236    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 01:31:23 +0000 UTC" deadline="2028-01-03 11:53:06.866527824 +0000 UTC"
Jul 21 01:36:24.885286 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:24.885278    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12754h16m41.981255285s"
Jul 21 01:36:25.026632 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:25.026597    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:36:25.026848 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:25.026794    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmhdx" podUID="5ee75f94-10ca-4a6a-a2f4-077eb597a94f"
Jul 21 01:36:25.027704 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:25.027348    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xbk6c"
Jul 21 01:36:25.027704 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:25.027476    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xbk6c" podUID="26238877-47e2-4ed6-8df9-c21f38f7578c"
Jul 21 01:36:25.057861 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:25.057792    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wl7t9" event={"ID":"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40","Type":"ContainerStarted","Data":"703525078c1d45b612eadb9f3a60acea880ca3b9570d42a667f3e45e73c22b87"}
Jul 21 01:36:25.062348 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:25.062315    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kr6h6" event={"ID":"b40e983b-2755-4993-abe2-30e2fc3cec40","Type":"ContainerStarted","Data":"5f67a574683ba93165fa1c6f0332ad3066b3ed017ba43f73e2f497e6b10c85f6"}
Jul 21 01:36:25.067180 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:25.067106    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fw8b6" event={"ID":"ed12def9-d42e-4d3e-98b2-d870a1c43231","Type":"ContainerStarted","Data":"a0188320a5a89b451230817e49e3d8061255316f35de24c1b76ce06a92872cac"}
Jul 21 01:36:25.086566 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:25.086539    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-219.ec2.internal" event={"ID":"8bb34cb64ea3f44a8e9569f1b50f1dce","Type":"ContainerStarted","Data":"5ce1223ed7c8b3e9b4dcdb3765a16aa6a86d0b5d063053570d7af23349d88868"}
Jul 21 01:36:25.096610 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:25.096581    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-25bgn" event={"ID":"a54c5fd7-029d-4f15-89ee-e46e339b8312","Type":"ContainerStarted","Data":"8b84c37c5dd8931884cc1dc824c9b77aea04b685c94dc57b584dcca265ad8851"}
Jul 21 01:36:25.103236 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:25.103172    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-219.ec2.internal" podStartSLOduration=3.103152585 podStartE2EDuration="3.103152585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 01:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:36:25.102199892 +0000 UTC m=+4.699840248" watchObservedRunningTime="2026-07-21 01:36:25.103152585 +0000 UTC m=+4.700792940"
Jul 21 01:36:25.110044 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:25.110017    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kzvp4" event={"ID":"7073f999-4de0-4a60-b3a2-58e9e527ffea","Type":"ContainerStarted","Data":"76327c846ae2985c8d707d9415fafcc862c0abc9edf2527ffba737cd7f81faa1"}
Jul 21 01:36:25.124842 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:25.124814    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj" event={"ID":"40c4613a-d37f-4b0d-bdfe-88c655071445","Type":"ContainerStarted","Data":"51f89d0d5c179c2992afc84405a82966cdf8b6eecbdb476a530d564b4f26aae2"}
Jul 21 01:36:25.133700 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:25.133654    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gl7hj" event={"ID":"84e8ef89-8a9f-4f94-9d58-3f650d213260","Type":"ContainerStarted","Data":"50504b2896df63e7d261919e8abf7a3e0725f7018e951ee3942fb8e22413ea6f"}
Jul 21 01:36:25.155517 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:25.155437    2575 generic.go:356] "Generic (PLEG): container finished" podID="4f87f057fc96d679b5a4e615c23da522" containerID="8388ec6e234f628378ca8984346f02672792caf1564467a4244bef4293228cf2" exitCode=0
Jul 21 01:36:25.155670 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:25.155529    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal" event={"ID":"4f87f057fc96d679b5a4e615c23da522","Type":"ContainerDied","Data":"8388ec6e234f628378ca8984346f02672792caf1564467a4244bef4293228cf2"}
Jul 21 01:36:25.171777 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:25.171747    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95vkq" event={"ID":"771bbef7-a312-4d6f-9091-4a339e0876e5","Type":"ContainerStarted","Data":"98b7ea05dd88d68fe7c2f579613b81be460657969ddd1ff56ada3b3926aa7478"}
Jul 21 01:36:25.180218 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:25.180192    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dr894" event={"ID":"cb395f84-ef3c-486c-852f-19c9954a6715","Type":"ContainerStarted","Data":"401fd7ba658a1bd162fd42a45b59e8d4947f0c166d8e28c707a36ac80c13ab7a"}
Jul 21 01:36:26.020523 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:26.020486    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245g5"
Jul 21 01:36:26.021044 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:26.020665    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-245g5" podUID="12e288ad-0d62-4e00-9a2a-1de85626abb9"
Jul 21 01:36:26.196665 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:26.196630    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal" event={"ID":"4f87f057fc96d679b5a4e615c23da522","Type":"ContainerStarted","Data":"1b1e0f861661143dfe01aadef105cddc202a0a1e20ef896f6e04295ffeb7ded1"}
Jul 21 01:36:26.211928 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:26.211869    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-219.ec2.internal" podStartSLOduration=4.211846745 podStartE2EDuration="4.211846745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 01:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:36:26.211263235 +0000 UTC m=+5.808903589" watchObservedRunningTime="2026-07-21 01:36:26.211846745 +0000 UTC m=+5.809487096"
Jul 21 01:36:26.493088 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:26.492999    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ee75f94-10ca-4a6a-a2f4-077eb597a94f-original-pull-secret\") pod \"global-pull-secret-syncer-pmhdx\" (UID: \"5ee75f94-10ca-4a6a-a2f4-077eb597a94f\") " pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:36:26.493255 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:26.493226    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 21 01:36:26.493325 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:26.493293    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ee75f94-10ca-4a6a-a2f4-077eb597a94f-original-pull-secret podName:5ee75f94-10ca-4a6a-a2f4-077eb597a94f nodeName:}" failed. No retries permitted until 2026-07-21 01:36:30.493272834 +0000 UTC m=+10.090913168 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ee75f94-10ca-4a6a-a2f4-077eb597a94f-original-pull-secret") pod "global-pull-secret-syncer-pmhdx" (UID: "5ee75f94-10ca-4a6a-a2f4-077eb597a94f") : object "kube-system"/"original-pull-secret" not registered
Jul 21 01:36:26.593722 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:26.593684    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12e288ad-0d62-4e00-9a2a-1de85626abb9-metrics-certs\") pod \"network-metrics-daemon-245g5\" (UID: \"12e288ad-0d62-4e00-9a2a-1de85626abb9\") " pod="openshift-multus/network-metrics-daemon-245g5"
Jul 21 01:36:26.593922 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:26.593874    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 01:36:26.593982 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:26.593937    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12e288ad-0d62-4e00-9a2a-1de85626abb9-metrics-certs podName:12e288ad-0d62-4e00-9a2a-1de85626abb9 nodeName:}" failed. No retries permitted until 2026-07-21 01:36:30.593918828 +0000 UTC m=+10.191559163 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12e288ad-0d62-4e00-9a2a-1de85626abb9-metrics-certs") pod "network-metrics-daemon-245g5" (UID: "12e288ad-0d62-4e00-9a2a-1de85626abb9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 01:36:26.695642 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:26.695011    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpxtj\" (UniqueName: \"kubernetes.io/projected/26238877-47e2-4ed6-8df9-c21f38f7578c-kube-api-access-dpxtj\") pod \"network-check-target-xbk6c\" (UID: \"26238877-47e2-4ed6-8df9-c21f38f7578c\") " pod="openshift-network-diagnostics/network-check-target-xbk6c"
Jul 21 01:36:26.695642 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:26.695173    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 21 01:36:26.695642 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:26.695196    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 21 01:36:26.695642 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:26.695210    2575 projected.go:196] Error preparing data for projected volume kube-api-access-dpxtj for pod openshift-network-diagnostics/network-check-target-xbk6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 01:36:26.695642 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:26.695276    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26238877-47e2-4ed6-8df9-c21f38f7578c-kube-api-access-dpxtj podName:26238877-47e2-4ed6-8df9-c21f38f7578c nodeName:}" failed. No retries permitted until 2026-07-21 01:36:30.695252484 +0000 UTC m=+10.292892850 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dpxtj" (UniqueName: "kubernetes.io/projected/26238877-47e2-4ed6-8df9-c21f38f7578c-kube-api-access-dpxtj") pod "network-check-target-xbk6c" (UID: "26238877-47e2-4ed6-8df9-c21f38f7578c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 01:36:27.023500 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:27.022842    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:36:27.023500 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:27.022975    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmhdx" podUID="5ee75f94-10ca-4a6a-a2f4-077eb597a94f"
Jul 21 01:36:27.023500 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:27.023355    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xbk6c"
Jul 21 01:36:27.023500 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:27.023446    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xbk6c" podUID="26238877-47e2-4ed6-8df9-c21f38f7578c"
Jul 21 01:36:28.019675 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:28.019636    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245g5"
Jul 21 01:36:28.019960 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:28.019799    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-245g5" podUID="12e288ad-0d62-4e00-9a2a-1de85626abb9"
Jul 21 01:36:29.021054 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:29.020783    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xbk6c"
Jul 21 01:36:29.021054 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:29.020935    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xbk6c" podUID="26238877-47e2-4ed6-8df9-c21f38f7578c"
Jul 21 01:36:29.021941 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:29.021915    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:36:29.022133 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:29.022106    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmhdx" podUID="5ee75f94-10ca-4a6a-a2f4-077eb597a94f"
Jul 21 01:36:30.020034 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:30.019995    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245g5"
Jul 21 01:36:30.020219 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:30.020157    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-245g5" podUID="12e288ad-0d62-4e00-9a2a-1de85626abb9"
Jul 21 01:36:30.529202 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:30.529162    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ee75f94-10ca-4a6a-a2f4-077eb597a94f-original-pull-secret\") pod \"global-pull-secret-syncer-pmhdx\" (UID: \"5ee75f94-10ca-4a6a-a2f4-077eb597a94f\") " pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:36:30.529699 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:30.529298    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 21 01:36:30.529699 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:30.529365    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ee75f94-10ca-4a6a-a2f4-077eb597a94f-original-pull-secret podName:5ee75f94-10ca-4a6a-a2f4-077eb597a94f nodeName:}" failed. No retries permitted until 2026-07-21 01:36:38.529348349 +0000 UTC m=+18.126988701 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ee75f94-10ca-4a6a-a2f4-077eb597a94f-original-pull-secret") pod "global-pull-secret-syncer-pmhdx" (UID: "5ee75f94-10ca-4a6a-a2f4-077eb597a94f") : object "kube-system"/"original-pull-secret" not registered
Jul 21 01:36:30.630353 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:30.630242    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12e288ad-0d62-4e00-9a2a-1de85626abb9-metrics-certs\") pod \"network-metrics-daemon-245g5\" (UID: \"12e288ad-0d62-4e00-9a2a-1de85626abb9\") " pod="openshift-multus/network-metrics-daemon-245g5"
Jul 21 01:36:30.630527 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:30.630406    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 01:36:30.630527 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:30.630491    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12e288ad-0d62-4e00-9a2a-1de85626abb9-metrics-certs podName:12e288ad-0d62-4e00-9a2a-1de85626abb9 nodeName:}" failed. No retries permitted until 2026-07-21 01:36:38.630468193 +0000 UTC m=+18.228108526 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12e288ad-0d62-4e00-9a2a-1de85626abb9-metrics-certs") pod "network-metrics-daemon-245g5" (UID: "12e288ad-0d62-4e00-9a2a-1de85626abb9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 01:36:30.731193 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:30.731148    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpxtj\" (UniqueName: \"kubernetes.io/projected/26238877-47e2-4ed6-8df9-c21f38f7578c-kube-api-access-dpxtj\") pod \"network-check-target-xbk6c\" (UID: \"26238877-47e2-4ed6-8df9-c21f38f7578c\") " pod="openshift-network-diagnostics/network-check-target-xbk6c"
Jul 21 01:36:30.731436 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:30.731414    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 21 01:36:30.731498 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:30.731443    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 21 01:36:30.731498 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:30.731459    2575 projected.go:196] Error preparing data for projected volume kube-api-access-dpxtj for pod openshift-network-diagnostics/network-check-target-xbk6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 01:36:30.731605 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:30.731534    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26238877-47e2-4ed6-8df9-c21f38f7578c-kube-api-access-dpxtj podName:26238877-47e2-4ed6-8df9-c21f38f7578c nodeName:}" failed. No retries permitted until 2026-07-21 01:36:38.73151009 +0000 UTC m=+18.329150424 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dpxtj" (UniqueName: "kubernetes.io/projected/26238877-47e2-4ed6-8df9-c21f38f7578c-kube-api-access-dpxtj") pod "network-check-target-xbk6c" (UID: "26238877-47e2-4ed6-8df9-c21f38f7578c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 01:36:31.021717 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:31.021626    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:36:31.022474 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:31.021979    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xbk6c"
Jul 21 01:36:31.022474 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:31.022104    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xbk6c" podUID="26238877-47e2-4ed6-8df9-c21f38f7578c"
Jul 21 01:36:31.022811 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:31.022780    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmhdx" podUID="5ee75f94-10ca-4a6a-a2f4-077eb597a94f"
Jul 21 01:36:32.019574 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:32.019541    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245g5"
Jul 21 01:36:32.019983 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:32.019706    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-245g5" podUID="12e288ad-0d62-4e00-9a2a-1de85626abb9"
Jul 21 01:36:33.020057 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:33.019968    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xbk6c"
Jul 21 01:36:33.020522 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:33.020104    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xbk6c" podUID="26238877-47e2-4ed6-8df9-c21f38f7578c"
Jul 21 01:36:33.020522 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:33.020144    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:36:33.020522 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:33.020208    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmhdx" podUID="5ee75f94-10ca-4a6a-a2f4-077eb597a94f"
Jul 21 01:36:34.020171 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:34.020137    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245g5"
Jul 21 01:36:34.020597 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:34.020265    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-245g5" podUID="12e288ad-0d62-4e00-9a2a-1de85626abb9"
Jul 21 01:36:35.019770 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:35.019734    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:36:35.019965 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:35.019838    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmhdx" podUID="5ee75f94-10ca-4a6a-a2f4-077eb597a94f"
Jul 21 01:36:35.019965 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:35.019867    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xbk6c"
Jul 21 01:36:35.019965 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:35.019913    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xbk6c" podUID="26238877-47e2-4ed6-8df9-c21f38f7578c"
Jul 21 01:36:36.020438 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:36.020404    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245g5"
Jul 21 01:36:36.020899 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:36.020541    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-245g5" podUID="12e288ad-0d62-4e00-9a2a-1de85626abb9"
Jul 21 01:36:37.020005 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:37.019974    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xbk6c"
Jul 21 01:36:37.020005 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:37.019974    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:36:37.020211 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:37.020151    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xbk6c" podUID="26238877-47e2-4ed6-8df9-c21f38f7578c"
Jul 21 01:36:37.020211 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:37.020202    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmhdx" podUID="5ee75f94-10ca-4a6a-a2f4-077eb597a94f"
Jul 21 01:36:38.020161 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:38.020122    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245g5"
Jul 21 01:36:38.020696 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:38.020273    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-245g5" podUID="12e288ad-0d62-4e00-9a2a-1de85626abb9"
Jul 21 01:36:38.588922 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:38.588886    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ee75f94-10ca-4a6a-a2f4-077eb597a94f-original-pull-secret\") pod \"global-pull-secret-syncer-pmhdx\" (UID: \"5ee75f94-10ca-4a6a-a2f4-077eb597a94f\") " pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:36:38.589071 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:38.589049    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 21 01:36:38.589125 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:38.589113    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ee75f94-10ca-4a6a-a2f4-077eb597a94f-original-pull-secret podName:5ee75f94-10ca-4a6a-a2f4-077eb597a94f nodeName:}" failed. No retries permitted until 2026-07-21 01:36:54.589099537 +0000 UTC m=+34.186739865 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ee75f94-10ca-4a6a-a2f4-077eb597a94f-original-pull-secret") pod "global-pull-secret-syncer-pmhdx" (UID: "5ee75f94-10ca-4a6a-a2f4-077eb597a94f") : object "kube-system"/"original-pull-secret" not registered
Jul 21 01:36:38.689549 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:38.689499    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12e288ad-0d62-4e00-9a2a-1de85626abb9-metrics-certs\") pod \"network-metrics-daemon-245g5\" (UID: \"12e288ad-0d62-4e00-9a2a-1de85626abb9\") " pod="openshift-multus/network-metrics-daemon-245g5"
Jul 21 01:36:38.689723 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:38.689662    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 01:36:38.689766 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:38.689743    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12e288ad-0d62-4e00-9a2a-1de85626abb9-metrics-certs podName:12e288ad-0d62-4e00-9a2a-1de85626abb9 nodeName:}" failed. No retries permitted until 2026-07-21 01:36:54.689722629 +0000 UTC m=+34.287362977 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12e288ad-0d62-4e00-9a2a-1de85626abb9-metrics-certs") pod "network-metrics-daemon-245g5" (UID: "12e288ad-0d62-4e00-9a2a-1de85626abb9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 01:36:38.790311 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:38.790275    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpxtj\" (UniqueName: \"kubernetes.io/projected/26238877-47e2-4ed6-8df9-c21f38f7578c-kube-api-access-dpxtj\") pod \"network-check-target-xbk6c\" (UID: \"26238877-47e2-4ed6-8df9-c21f38f7578c\") " pod="openshift-network-diagnostics/network-check-target-xbk6c"
Jul 21 01:36:38.790494 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:38.790471    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 21 01:36:38.790547 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:38.790502    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 21 01:36:38.790547 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:38.790517    2575 projected.go:196] Error preparing data for projected volume kube-api-access-dpxtj for pod openshift-network-diagnostics/network-check-target-xbk6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 01:36:38.790660 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:38.790585    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26238877-47e2-4ed6-8df9-c21f38f7578c-kube-api-access-dpxtj podName:26238877-47e2-4ed6-8df9-c21f38f7578c nodeName:}" failed. No retries permitted until 2026-07-21 01:36:54.790566367 +0000 UTC m=+34.388206698 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dpxtj" (UniqueName: "kubernetes.io/projected/26238877-47e2-4ed6-8df9-c21f38f7578c-kube-api-access-dpxtj") pod "network-check-target-xbk6c" (UID: "26238877-47e2-4ed6-8df9-c21f38f7578c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 01:36:39.019857 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:39.019822    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:36:39.020033 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:39.019859    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xbk6c"
Jul 21 01:36:39.020033 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:39.019960    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmhdx" podUID="5ee75f94-10ca-4a6a-a2f4-077eb597a94f"
Jul 21 01:36:39.020161 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:39.020103    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xbk6c" podUID="26238877-47e2-4ed6-8df9-c21f38f7578c"
Jul 21 01:36:40.020224 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:40.020187    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245g5"
Jul 21 01:36:40.020677 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:40.020317    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-245g5" podUID="12e288ad-0d62-4e00-9a2a-1de85626abb9"
Jul 21 01:36:41.020622 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:41.020575    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:36:41.021119 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:41.020705    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmhdx" podUID="5ee75f94-10ca-4a6a-a2f4-077eb597a94f"
Jul 21 01:36:41.021119 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:41.020774    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xbk6c"
Jul 21 01:36:41.021119 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:41.020892    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xbk6c" podUID="26238877-47e2-4ed6-8df9-c21f38f7578c"
Jul 21 01:36:42.020536 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:42.020492    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245g5"
Jul 21 01:36:42.020730 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:42.020674    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-245g5" podUID="12e288ad-0d62-4e00-9a2a-1de85626abb9"
Jul 21 01:36:43.020502 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:43.020462    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:36:43.020804 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:43.020591    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmhdx" podUID="5ee75f94-10ca-4a6a-a2f4-077eb597a94f"
Jul 21 01:36:43.020804 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:43.020624    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xbk6c"
Jul 21 01:36:43.020804 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:43.020733    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xbk6c" podUID="26238877-47e2-4ed6-8df9-c21f38f7578c"
Jul 21 01:36:44.021003 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:44.020583    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245g5"
Jul 21 01:36:44.021789 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:44.021123    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-245g5" podUID="12e288ad-0d62-4e00-9a2a-1de85626abb9"
Jul 21 01:36:44.238244 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:44.238122    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95vkq" event={"ID":"771bbef7-a312-4d6f-9091-4a339e0876e5","Type":"ContainerStarted","Data":"abe9b1979368a98f6f974e112d24f95d74e16c87f75435921ef6ca2481f12e3e"}
Jul 21 01:36:44.239662 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:44.239624    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wl7t9" event={"ID":"486d1dc5-263d-4bcf-a2c6-2882c5c7bd40","Type":"ContainerStarted","Data":"5abe8b303f4323f90a20da6d6c15472c01b4e2244cbc5632e82bc08a7d8fbc43"}
Jul 21 01:36:44.241275 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:44.241244    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kr6h6" event={"ID":"b40e983b-2755-4993-abe2-30e2fc3cec40","Type":"ContainerStarted","Data":"ef48056fbd10eff89debd3b00e0aa3708d864fc8d81febcc341d5c07acf4b409"}
Jul 21 01:36:44.242786 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:44.242760    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fw8b6" event={"ID":"ed12def9-d42e-4d3e-98b2-d870a1c43231","Type":"ContainerStarted","Data":"910a7619ed27521ffb35dbae0bf64f38677ad0da7ae6a68b5e4a51519418d4ba"}
Jul 21 01:36:44.244291 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:44.244266    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-25bgn" event={"ID":"a54c5fd7-029d-4f15-89ee-e46e339b8312","Type":"ContainerStarted","Data":"d861cef51ed7e95bb479349c74bf99d00e4def5ec5d8b0e65c9f89fa6461e415"}
Jul 21 01:36:44.245808 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:44.245779    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kzvp4" event={"ID":"7073f999-4de0-4a60-b3a2-58e9e527ffea","Type":"ContainerStarted","Data":"3e823f48050b6c2219390f55219658203a4a2ccf6695c1aecc2b37210062afcf"}
Jul 21 01:36:44.248947 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:44.248925    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj" event={"ID":"40c4613a-d37f-4b0d-bdfe-88c655071445","Type":"ContainerStarted","Data":"01e5f27fa00e5c0d11536da252ba7e359a3e1bf19e4c1661cadefc1f397d19fc"}
Jul 21 01:36:44.249057 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:44.249000    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj" event={"ID":"40c4613a-d37f-4b0d-bdfe-88c655071445","Type":"ContainerStarted","Data":"94fdfbac3dc9bb47d4a8f0566aaf43b2b2071e968bc342907ec02d6a2397cc9e"}
Jul 21 01:36:44.249057 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:44.249052    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj" event={"ID":"40c4613a-d37f-4b0d-bdfe-88c655071445","Type":"ContainerStarted","Data":"99e3ff38a65d6560ea743b661b57fbddf8fbf21a1a8998126fd3bb05b8ce3353"}
Jul 21 01:36:44.249168 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:44.249099    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj" event={"ID":"40c4613a-d37f-4b0d-bdfe-88c655071445","Type":"ContainerStarted","Data":"03ea66259713b6afcbc294fefec30c93055f664575fca49d515e3eca31244e0f"}
Jul 21 01:36:44.249168 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:44.249147    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj" event={"ID":"40c4613a-d37f-4b0d-bdfe-88c655071445","Type":"ContainerStarted","Data":"367d27a45b3ac90a661689462533f208e65a7bf06a1f8bee7045d58c112ef240"}
Jul 21 01:36:44.249265 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:44.249194    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj" event={"ID":"40c4613a-d37f-4b0d-bdfe-88c655071445","Type":"ContainerStarted","Data":"fa7b2d362543ad091f39cb1111ecd455ba266cddfb8a75c02f8ca3d5daae56fb"}
Jul 21 01:36:44.250527 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:44.250505    2575 generic.go:356] "Generic (PLEG): container finished" podID="84e8ef89-8a9f-4f94-9d58-3f650d213260" containerID="e3b66366b48eb5bd29ed70c2140e17aadfd306325fc3962edf379b2984ef30f7" exitCode=0
Jul 21 01:36:44.250640 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:44.250537    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gl7hj" event={"ID":"84e8ef89-8a9f-4f94-9d58-3f650d213260","Type":"ContainerDied","Data":"e3b66366b48eb5bd29ed70c2140e17aadfd306325fc3962edf379b2984ef30f7"}
Jul 21 01:36:44.255600 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:44.255556    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wl7t9" podStartSLOduration=3.9443064249999997 podStartE2EDuration="23.25554193s" totalImagesPullingTime="19.311235505s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 01:36:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:36:44.255209116 +0000 UTC m=+23.852849460" watchObservedRunningTime="2026-07-21 01:36:44.25554193 +0000 UTC m=+23.853182280"
Jul 21 01:36:44.270135 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:44.270091    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-kr6h6" podStartSLOduration=3.921329097 podStartE2EDuration="23.270075864s" totalImagesPullingTime="19.348746767s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 01:36:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:36:44.270037373 +0000 UTC m=+23.867677724" watchObservedRunningTime="2026-07-21 01:36:44.270075864 +0000 UTC m=+23.867716214"
Jul 21 01:36:44.282730 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:44.282676    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-fw8b6" podStartSLOduration=3.994995975 podStartE2EDuration="23.282659224s" totalImagesPullingTime="19.287663249s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 01:36:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:36:44.282518977 +0000 UTC m=+23.880159330" watchObservedRunningTime="2026-07-21 01:36:44.282659224 +0000 UTC m=+23.880299576"
Jul 21 01:36:44.295159 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:44.295113    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kzvp4" podStartSLOduration=4.015033726 podStartE2EDuration="23.295098256s" totalImagesPullingTime="19.28006453s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 01:36:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:36:44.29508537 +0000 UTC m=+23.892725721" watchObservedRunningTime="2026-07-21 01:36:44.295098256 +0000 UTC m=+23.892738604"
Jul 21 01:36:44.326587 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:44.326532    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-25bgn" podStartSLOduration=4.073013724 podStartE2EDuration="23.326517716s" totalImagesPullingTime="19.253503992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:36:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:36:44.307324361 +0000 UTC m=+23.904964712" watchObservedRunningTime="2026-07-21 01:36:44.326517716 +0000 UTC m=+23.924158068"
Jul 21 01:36:44.826847 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:44.826818    2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 21 01:36:44.913060 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:44.912930    2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-21T01:36:44.826843128Z","UUID":"b3c5b623-8350-484b-87bd-b7f3673a6c1c","Handler":null,"Name":"","Endpoint":""}
Jul 21 01:36:44.914833 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:44.914806    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 21 01:36:44.914980 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:44.914849    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 21 01:36:45.020085 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:45.020048    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:36:45.020324 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:45.020188    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmhdx" podUID="5ee75f94-10ca-4a6a-a2f4-077eb597a94f"
Jul 21 01:36:45.020324 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:45.020234    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xbk6c"
Jul 21 01:36:45.020436 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:45.020329    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xbk6c" podUID="26238877-47e2-4ed6-8df9-c21f38f7578c"
Jul 21 01:36:45.215876 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:45.215805    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-25bgn"
Jul 21 01:36:45.216452 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:45.216434    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-25bgn"
Jul 21 01:36:45.254730 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:45.254702    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95vkq" event={"ID":"771bbef7-a312-4d6f-9091-4a339e0876e5","Type":"ContainerStarted","Data":"a2ec770eefb4fdfcf90f10da4b5628970a98268ecdb12b1c4e1178def941d2a3"}
Jul 21 01:36:45.256089 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:45.256063    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dr894" event={"ID":"cb395f84-ef3c-486c-852f-19c9954a6715","Type":"ContainerStarted","Data":"86fb0c255f7a9b5b5da706c78f82347693f56822972595b530a0fdb62f93c0a5"}
Jul 21 01:36:45.256835 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:45.256809    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-25bgn"
Jul 21 01:36:45.257243 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:45.257219    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-25bgn"
Jul 21 01:36:45.273476 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:45.273429    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dr894" podStartSLOduration=4.985448378 podStartE2EDuration="24.273410608s" totalImagesPullingTime="19.28796223s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 01:36:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:36:45.273348912 +0000 UTC m=+24.870989263" watchObservedRunningTime="2026-07-21 01:36:45.273410608 +0000 UTC m=+24.871050962"
Jul 21 01:36:46.019703 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:46.019675    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245g5"
Jul 21 01:36:46.019844 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:46.019819    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-245g5" podUID="12e288ad-0d62-4e00-9a2a-1de85626abb9"
Jul 21 01:36:46.264495 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:46.264422    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj" event={"ID":"40c4613a-d37f-4b0d-bdfe-88c655071445","Type":"ContainerStarted","Data":"b1d6195e2f424683165b813b9336b840b8f24f8bcce0867a83aaac4290e19b72"}
Jul 21 01:36:46.266749 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:46.266708    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95vkq" event={"ID":"771bbef7-a312-4d6f-9091-4a339e0876e5","Type":"ContainerStarted","Data":"f9356ee4d9ce1cffd56d2d4c1892468e972d6db815e56900b3ccdff81856ec9f"}
Jul 21 01:36:46.284724 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:46.284683    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95vkq" podStartSLOduration=3.762632782 podStartE2EDuration="25.284664796s" totalImagesPullingTime="21.522032014s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 01:36:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:36:46.283423969 +0000 UTC m=+25.881064324" watchObservedRunningTime="2026-07-21 01:36:46.284664796 +0000 UTC m=+25.882305148"
Jul 21 01:36:47.020018 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:47.019988    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xbk6c"
Jul 21 01:36:47.020290 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:47.019988    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:36:47.020290 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:47.020116    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xbk6c" podUID="26238877-47e2-4ed6-8df9-c21f38f7578c"
Jul 21 01:36:47.020290 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:47.020199    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmhdx" podUID="5ee75f94-10ca-4a6a-a2f4-077eb597a94f"
Jul 21 01:36:48.019579 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:48.019547    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245g5"
Jul 21 01:36:48.020035 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:48.019701    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-245g5" podUID="12e288ad-0d62-4e00-9a2a-1de85626abb9"
Jul 21 01:36:49.020747 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:49.020570    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:36:49.021674 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:49.020577    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xbk6c"
Jul 21 01:36:49.021674 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:49.020824    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmhdx" podUID="5ee75f94-10ca-4a6a-a2f4-077eb597a94f"
Jul 21 01:36:49.021674 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:49.020904    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xbk6c" podUID="26238877-47e2-4ed6-8df9-c21f38f7578c"
Jul 21 01:36:49.277175 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:49.277087    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj" event={"ID":"40c4613a-d37f-4b0d-bdfe-88c655071445","Type":"ContainerStarted","Data":"e46002c2f60b46b4502a629b780aa118d99e0172c1e596a6e597c879544fbdcd"}
Jul 21 01:36:49.277412 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:49.277393    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:49.277476 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:49.277467    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:49.279101 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:49.279074    2575 generic.go:356] "Generic (PLEG): container finished" podID="84e8ef89-8a9f-4f94-9d58-3f650d213260" containerID="c9b93838862c0edaa606763ebb0b0ffab3ecfe4f5f1fec2ead12b3a6485b78fe" exitCode=0
Jul 21 01:36:49.279192 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:49.279139    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gl7hj" event={"ID":"84e8ef89-8a9f-4f94-9d58-3f650d213260","Type":"ContainerDied","Data":"c9b93838862c0edaa606763ebb0b0ffab3ecfe4f5f1fec2ead12b3a6485b78fe"}
Jul 21 01:36:49.292580 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:49.292562    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:49.347956 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:49.347909    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj" podStartSLOduration=8.967605884 podStartE2EDuration="28.347896485s" totalImagesPullingTime="19.380290601s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 01:36:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:36:49.31139772 +0000 UTC m=+28.909038072" watchObservedRunningTime="2026-07-21 01:36:49.347896485 +0000 UTC m=+28.945536836"
Jul 21 01:36:50.019701 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:50.019540    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245g5"
Jul 21 01:36:50.019952 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:50.019811    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-245g5" podUID="12e288ad-0d62-4e00-9a2a-1de85626abb9"
Jul 21 01:36:50.283679 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:50.283539    2575 generic.go:356] "Generic (PLEG): container finished" podID="84e8ef89-8a9f-4f94-9d58-3f650d213260" containerID="27bb3a6f882dad3e4b3b43dbb158818023aafc5af9a4639258c395a3c112bbc9" exitCode=0
Jul 21 01:36:50.283679 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:50.283638    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gl7hj" event={"ID":"84e8ef89-8a9f-4f94-9d58-3f650d213260","Type":"ContainerDied","Data":"27bb3a6f882dad3e4b3b43dbb158818023aafc5af9a4639258c395a3c112bbc9"}
Jul 21 01:36:50.284851 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:50.284339    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:50.299322 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:50.299297    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:36:50.589893 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:50.589806    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pmhdx"]
Jul 21 01:36:50.590121 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:50.589925    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:36:50.590121 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:50.590017    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmhdx" podUID="5ee75f94-10ca-4a6a-a2f4-077eb597a94f"
Jul 21 01:36:50.597539 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:50.597488    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-245g5"]
Jul 21 01:36:50.597658 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:50.597647    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245g5"
Jul 21 01:36:50.597792 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:50.597771    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-245g5" podUID="12e288ad-0d62-4e00-9a2a-1de85626abb9"
Jul 21 01:36:50.598164 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:50.598145    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xbk6c"]
Jul 21 01:36:50.598261 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:50.598245    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xbk6c"
Jul 21 01:36:50.598360 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:50.598336    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xbk6c" podUID="26238877-47e2-4ed6-8df9-c21f38f7578c"
Jul 21 01:36:52.019808 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:52.019761    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:36:52.020226 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:52.019877    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmhdx" podUID="5ee75f94-10ca-4a6a-a2f4-077eb597a94f"
Jul 21 01:36:52.290369 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:52.290278    2575 generic.go:356] "Generic (PLEG): container finished" podID="84e8ef89-8a9f-4f94-9d58-3f650d213260" containerID="6a24c2734e8e9cfbbcf645cff5a6dea627cea9b5a317efcb5ded3c42f4d175b9" exitCode=0
Jul 21 01:36:52.290369 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:52.290356    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gl7hj" event={"ID":"84e8ef89-8a9f-4f94-9d58-3f650d213260","Type":"ContainerDied","Data":"6a24c2734e8e9cfbbcf645cff5a6dea627cea9b5a317efcb5ded3c42f4d175b9"}
Jul 21 01:36:53.023404 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:53.023371    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245g5"
Jul 21 01:36:53.023404 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:53.023404    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xbk6c"
Jul 21 01:36:53.023887 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:53.023511    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-245g5" podUID="12e288ad-0d62-4e00-9a2a-1de85626abb9"
Jul 21 01:36:53.023887 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:53.023644    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xbk6c" podUID="26238877-47e2-4ed6-8df9-c21f38f7578c"
Jul 21 01:36:54.020330 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:54.020289    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:36:54.020519 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:54.020422    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmhdx" podUID="5ee75f94-10ca-4a6a-a2f4-077eb597a94f"
Jul 21 01:36:54.614702 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:54.614475    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ee75f94-10ca-4a6a-a2f4-077eb597a94f-original-pull-secret\") pod \"global-pull-secret-syncer-pmhdx\" (UID: \"5ee75f94-10ca-4a6a-a2f4-077eb597a94f\") " pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:36:54.615163 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:54.614664    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 21 01:36:54.615163 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:54.614808    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ee75f94-10ca-4a6a-a2f4-077eb597a94f-original-pull-secret podName:5ee75f94-10ca-4a6a-a2f4-077eb597a94f nodeName:}" failed. No retries permitted until 2026-07-21 01:37:26.614785436 +0000 UTC m=+66.212425770 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ee75f94-10ca-4a6a-a2f4-077eb597a94f-original-pull-secret") pod "global-pull-secret-syncer-pmhdx" (UID: "5ee75f94-10ca-4a6a-a2f4-077eb597a94f") : object "kube-system"/"original-pull-secret" not registered
Jul 21 01:36:54.715013 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:54.714982    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12e288ad-0d62-4e00-9a2a-1de85626abb9-metrics-certs\") pod \"network-metrics-daemon-245g5\" (UID: \"12e288ad-0d62-4e00-9a2a-1de85626abb9\") " pod="openshift-multus/network-metrics-daemon-245g5"
Jul 21 01:36:54.715155 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:54.715131    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 01:36:54.715202 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:54.715189    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12e288ad-0d62-4e00-9a2a-1de85626abb9-metrics-certs podName:12e288ad-0d62-4e00-9a2a-1de85626abb9 nodeName:}" failed. No retries permitted until 2026-07-21 01:37:26.715174686 +0000 UTC m=+66.312815038 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12e288ad-0d62-4e00-9a2a-1de85626abb9-metrics-certs") pod "network-metrics-daemon-245g5" (UID: "12e288ad-0d62-4e00-9a2a-1de85626abb9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 01:36:54.815720 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:54.815658    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpxtj\" (UniqueName: \"kubernetes.io/projected/26238877-47e2-4ed6-8df9-c21f38f7578c-kube-api-access-dpxtj\") pod \"network-check-target-xbk6c\" (UID: \"26238877-47e2-4ed6-8df9-c21f38f7578c\") " pod="openshift-network-diagnostics/network-check-target-xbk6c"
Jul 21 01:36:54.815909 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:54.815833    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 21 01:36:54.815909 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:54.815860    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 21 01:36:54.815909 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:54.815875    2575 projected.go:196] Error preparing data for projected volume kube-api-access-dpxtj for pod openshift-network-diagnostics/network-check-target-xbk6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 01:36:54.816087 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:54.815928    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26238877-47e2-4ed6-8df9-c21f38f7578c-kube-api-access-dpxtj podName:26238877-47e2-4ed6-8df9-c21f38f7578c nodeName:}" failed. No retries permitted until 2026-07-21 01:37:26.815913517 +0000 UTC m=+66.413553850 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dpxtj" (UniqueName: "kubernetes.io/projected/26238877-47e2-4ed6-8df9-c21f38f7578c-kube-api-access-dpxtj") pod "network-check-target-xbk6c" (UID: "26238877-47e2-4ed6-8df9-c21f38f7578c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 01:36:55.020436 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:55.019972    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xbk6c"
Jul 21 01:36:55.020436 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:55.019978    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245g5"
Jul 21 01:36:55.020436 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:55.020097    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xbk6c" podUID="26238877-47e2-4ed6-8df9-c21f38f7578c"
Jul 21 01:36:55.020436 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:55.020196    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-245g5" podUID="12e288ad-0d62-4e00-9a2a-1de85626abb9"
Jul 21 01:36:56.020211 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.020182    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:36:56.020629 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:56.020278    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmhdx" podUID="5ee75f94-10ca-4a6a-a2f4-077eb597a94f"
Jul 21 01:36:56.194948 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.194863    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-219.ec2.internal" event="NodeReady"
Jul 21 01:36:56.195103 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.195009    2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 21 01:36:56.225068 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.225034    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b9b98f9dc-g4qkb"]
Jul 21 01:36:56.256266 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.256240    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-twkvj"]
Jul 21 01:36:56.256426 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.256356    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:36:56.259102 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.259079    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 21 01:36:56.259228 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.259110    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-r74bp\""
Jul 21 01:36:56.259376 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.259357    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 21 01:36:56.259593 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.259578    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 21 01:36:56.264127 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.264105    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 21 01:36:56.272055 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.272036    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-drc4p"]
Jul 21 01:36:56.272130 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.272100    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-twkvj"
Jul 21 01:36:56.274255 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.274238    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 21 01:36:56.274350 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.274266    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wcxpb\""
Jul 21 01:36:56.274350 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.274284    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 21 01:36:56.290739 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.290713    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b9b98f9dc-g4qkb"]
Jul 21 01:36:56.290835 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.290749    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-drc4p"]
Jul 21 01:36:56.290835 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.290769    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-twkvj"]
Jul 21 01:36:56.290835 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.290797    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-drc4p"
Jul 21 01:36:56.293338 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.293318    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 21 01:36:56.293422 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.293327    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wrwpn\""
Jul 21 01:36:56.293670 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.293657    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 21 01:36:56.293715 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.293660    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 21 01:36:56.428190 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.428158    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/122f620b-920e-4174-8658-489db0d018ad-ca-trust-extracted\") pod \"image-registry-b9b98f9dc-g4qkb\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") " pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:36:56.428329 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.428195    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s24pv\" (UniqueName: \"kubernetes.io/projected/86db8e5d-3869-4ad0-8d3d-989e132f6038-kube-api-access-s24pv\") pod \"ingress-canary-drc4p\" (UID: \"86db8e5d-3869-4ad0-8d3d-989e132f6038\") " pod="openshift-ingress-canary/ingress-canary-drc4p"
Jul 21 01:36:56.428329 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.428219    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-registry-tls\") pod \"image-registry-b9b98f9dc-g4qkb\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") " pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:36:56.428329 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.428260    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/06885d56-7129-4e19-9911-5720b9668058-config-volume\") pod \"dns-default-twkvj\" (UID: \"06885d56-7129-4e19-9911-5720b9668058\") " pod="openshift-dns/dns-default-twkvj"
Jul 21 01:36:56.428329 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.428287    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/122f620b-920e-4174-8658-489db0d018ad-image-registry-private-configuration\") pod \"image-registry-b9b98f9dc-g4qkb\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") " pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:36:56.428329 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.428305    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/122f620b-920e-4174-8658-489db0d018ad-installation-pull-secrets\") pod \"image-registry-b9b98f9dc-g4qkb\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") " pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:36:56.428505 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.428343    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06885d56-7129-4e19-9911-5720b9668058-metrics-tls\") pod \"dns-default-twkvj\" (UID: \"06885d56-7129-4e19-9911-5720b9668058\") " pod="openshift-dns/dns-default-twkvj"
Jul 21 01:36:56.428505 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.428361    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86db8e5d-3869-4ad0-8d3d-989e132f6038-cert\") pod \"ingress-canary-drc4p\" (UID: \"86db8e5d-3869-4ad0-8d3d-989e132f6038\") " pod="openshift-ingress-canary/ingress-canary-drc4p"
Jul 21 01:36:56.428505 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.428401    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v7cg\" (UniqueName: \"kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-kube-api-access-6v7cg\") pod \"image-registry-b9b98f9dc-g4qkb\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") " pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:36:56.428505 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.428455    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/122f620b-920e-4174-8658-489db0d018ad-registry-certificates\") pod \"image-registry-b9b98f9dc-g4qkb\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") " pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:36:56.428505 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.428472    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/122f620b-920e-4174-8658-489db0d018ad-trusted-ca\") pod \"image-registry-b9b98f9dc-g4qkb\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") " pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:36:56.428505 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.428504    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06885d56-7129-4e19-9911-5720b9668058-tmp-dir\") pod \"dns-default-twkvj\" (UID: \"06885d56-7129-4e19-9911-5720b9668058\") " pod="openshift-dns/dns-default-twkvj"
Jul 21 01:36:56.428717 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.428520    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94tkd\" (UniqueName: \"kubernetes.io/projected/06885d56-7129-4e19-9911-5720b9668058-kube-api-access-94tkd\") pod \"dns-default-twkvj\" (UID: \"06885d56-7129-4e19-9911-5720b9668058\") " pod="openshift-dns/dns-default-twkvj"
Jul 21 01:36:56.428717 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.428565    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-bound-sa-token\") pod \"image-registry-b9b98f9dc-g4qkb\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") " pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:36:56.529506 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.529475    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v7cg\" (UniqueName: \"kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-kube-api-access-6v7cg\") pod \"image-registry-b9b98f9dc-g4qkb\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") " pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:36:56.529672 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.529513    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/122f620b-920e-4174-8658-489db0d018ad-registry-certificates\") pod \"image-registry-b9b98f9dc-g4qkb\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") " pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:36:56.529672 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.529539    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/122f620b-920e-4174-8658-489db0d018ad-trusted-ca\") pod \"image-registry-b9b98f9dc-g4qkb\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") " pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:36:56.529672 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.529585    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06885d56-7129-4e19-9911-5720b9668058-tmp-dir\") pod \"dns-default-twkvj\" (UID: \"06885d56-7129-4e19-9911-5720b9668058\") " pod="openshift-dns/dns-default-twkvj"
Jul 21 01:36:56.529672 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.529628    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94tkd\" (UniqueName: \"kubernetes.io/projected/06885d56-7129-4e19-9911-5720b9668058-kube-api-access-94tkd\") pod \"dns-default-twkvj\" (UID: \"06885d56-7129-4e19-9911-5720b9668058\") " pod="openshift-dns/dns-default-twkvj"
Jul 21 01:36:56.529672 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.529655    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-bound-sa-token\") pod \"image-registry-b9b98f9dc-g4qkb\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") " pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:36:56.529930 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.529701    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/122f620b-920e-4174-8658-489db0d018ad-ca-trust-extracted\") pod \"image-registry-b9b98f9dc-g4qkb\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") " pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:36:56.529930 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.529778    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s24pv\" (UniqueName: \"kubernetes.io/projected/86db8e5d-3869-4ad0-8d3d-989e132f6038-kube-api-access-s24pv\") pod \"ingress-canary-drc4p\" (UID: \"86db8e5d-3869-4ad0-8d3d-989e132f6038\") " pod="openshift-ingress-canary/ingress-canary-drc4p"
Jul 21 01:36:56.529930 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.529808    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-registry-tls\") pod \"image-registry-b9b98f9dc-g4qkb\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") " pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:36:56.529930 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.529834    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/06885d56-7129-4e19-9911-5720b9668058-config-volume\") pod \"dns-default-twkvj\" (UID: \"06885d56-7129-4e19-9911-5720b9668058\") " pod="openshift-dns/dns-default-twkvj"
Jul 21 01:36:56.529930 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.529860    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/122f620b-920e-4174-8658-489db0d018ad-image-registry-private-configuration\") pod \"image-registry-b9b98f9dc-g4qkb\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") " pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:36:56.529930 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.529887    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/122f620b-920e-4174-8658-489db0d018ad-installation-pull-secrets\") pod \"image-registry-b9b98f9dc-g4qkb\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") " pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:36:56.530205 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.529940    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06885d56-7129-4e19-9911-5720b9668058-metrics-tls\") pod \"dns-default-twkvj\" (UID: \"06885d56-7129-4e19-9911-5720b9668058\") " pod="openshift-dns/dns-default-twkvj"
Jul 21 01:36:56.530205 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:56.529951    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 21 01:36:56.530205 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:56.529969    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9b98f9dc-g4qkb: secret "image-registry-tls" not found
Jul 21 01:36:56.530205 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.529972    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86db8e5d-3869-4ad0-8d3d-989e132f6038-cert\") pod \"ingress-canary-drc4p\" (UID: \"86db8e5d-3869-4ad0-8d3d-989e132f6038\") " pod="openshift-ingress-canary/ingress-canary-drc4p"
Jul 21 01:36:56.530205 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.529971    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06885d56-7129-4e19-9911-5720b9668058-tmp-dir\") pod \"dns-default-twkvj\" (UID: \"06885d56-7129-4e19-9911-5720b9668058\") " pod="openshift-dns/dns-default-twkvj"
Jul 21 01:36:56.530205 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:56.530024    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-registry-tls podName:122f620b-920e-4174-8658-489db0d018ad nodeName:}" failed. No retries permitted until 2026-07-21 01:36:57.030006034 +0000 UTC m=+36.627646366 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-registry-tls") pod "image-registry-b9b98f9dc-g4qkb" (UID: "122f620b-920e-4174-8658-489db0d018ad") : secret "image-registry-tls" not found
Jul 21 01:36:56.530205 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:56.530103    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 21 01:36:56.530205 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.530116    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/122f620b-920e-4174-8658-489db0d018ad-registry-certificates\") pod \"image-registry-b9b98f9dc-g4qkb\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") " pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:36:56.530205 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.530116    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/122f620b-920e-4174-8658-489db0d018ad-ca-trust-extracted\") pod \"image-registry-b9b98f9dc-g4qkb\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") " pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:36:56.530205 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:56.530158    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06885d56-7129-4e19-9911-5720b9668058-metrics-tls podName:06885d56-7129-4e19-9911-5720b9668058 nodeName:}" failed. No retries permitted until 2026-07-21 01:36:57.030143243 +0000 UTC m=+36.627783593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06885d56-7129-4e19-9911-5720b9668058-metrics-tls") pod "dns-default-twkvj" (UID: "06885d56-7129-4e19-9911-5720b9668058") : secret "dns-default-metrics-tls" not found
Jul 21 01:36:56.530586 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.530559    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/122f620b-920e-4174-8658-489db0d018ad-trusted-ca\") pod \"image-registry-b9b98f9dc-g4qkb\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") " pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:36:56.530716 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.530650    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/06885d56-7129-4e19-9911-5720b9668058-config-volume\") pod \"dns-default-twkvj\" (UID: \"06885d56-7129-4e19-9911-5720b9668058\") " pod="openshift-dns/dns-default-twkvj"
Jul 21 01:36:56.533993 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.533968    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/122f620b-920e-4174-8658-489db0d018ad-installation-pull-secrets\") pod \"image-registry-b9b98f9dc-g4qkb\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") " pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:36:56.534078 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.533991    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/122f620b-920e-4174-8658-489db0d018ad-image-registry-private-configuration\") pod \"image-registry-b9b98f9dc-g4qkb\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") " pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:36:56.534078 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.533996    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86db8e5d-3869-4ad0-8d3d-989e132f6038-cert\") pod \"ingress-canary-drc4p\" (UID: \"86db8e5d-3869-4ad0-8d3d-989e132f6038\") " pod="openshift-ingress-canary/ingress-canary-drc4p"
Jul 21 01:36:56.537960 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.537935    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94tkd\" (UniqueName: \"kubernetes.io/projected/06885d56-7129-4e19-9911-5720b9668058-kube-api-access-94tkd\") pod \"dns-default-twkvj\" (UID: \"06885d56-7129-4e19-9911-5720b9668058\") " pod="openshift-dns/dns-default-twkvj"
Jul 21 01:36:56.538453 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.538429    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v7cg\" (UniqueName: \"kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-kube-api-access-6v7cg\") pod \"image-registry-b9b98f9dc-g4qkb\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") " pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:36:56.538551 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.538532    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-bound-sa-token\") pod \"image-registry-b9b98f9dc-g4qkb\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") " pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:36:56.538689 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.538670    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s24pv\" (UniqueName: \"kubernetes.io/projected/86db8e5d-3869-4ad0-8d3d-989e132f6038-kube-api-access-s24pv\") pod \"ingress-canary-drc4p\" (UID: \"86db8e5d-3869-4ad0-8d3d-989e132f6038\") " pod="openshift-ingress-canary/ingress-canary-drc4p"
Jul 21 01:36:56.600127 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.600103    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-drc4p"
Jul 21 01:36:56.790050 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:56.789912    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-drc4p"]
Jul 21 01:36:56.793062 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:36:56.793018    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86db8e5d_3869_4ad0_8d3d_989e132f6038.slice/crio-a8b60b949ef7ced1b6a2948b9b69e3394017af2f99f95da0db7ef73fd5d488f6 WatchSource:0}: Error finding container a8b60b949ef7ced1b6a2948b9b69e3394017af2f99f95da0db7ef73fd5d488f6: Status 404 returned error can't find the container with id a8b60b949ef7ced1b6a2948b9b69e3394017af2f99f95da0db7ef73fd5d488f6
Jul 21 01:36:57.020058 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:57.020029    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xbk6c"
Jul 21 01:36:57.020249 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:57.020229    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245g5"
Jul 21 01:36:57.023894 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:57.023869    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 21 01:36:57.023894 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:57.023885    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 21 01:36:57.024076 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:57.023912    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8s87n\""
Jul 21 01:36:57.024076 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:57.024032    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5qdkq\""
Jul 21 01:36:57.024257 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:57.024241    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 21 01:36:57.028287 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:57.028258    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:36:57.034273 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:57.034256    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06885d56-7129-4e19-9911-5720b9668058-metrics-tls\") pod \"dns-default-twkvj\" (UID: \"06885d56-7129-4e19-9911-5720b9668058\") " pod="openshift-dns/dns-default-twkvj"
Jul 21 01:36:57.034367 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:57.034322    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-registry-tls\") pod \"image-registry-b9b98f9dc-g4qkb\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") " pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:36:57.034413 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:57.034399    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 21 01:36:57.034462 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:57.034453    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06885d56-7129-4e19-9911-5720b9668058-metrics-tls podName:06885d56-7129-4e19-9911-5720b9668058 nodeName:}" failed. No retries permitted until 2026-07-21 01:36:58.0344406 +0000 UTC m=+37.632080931 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06885d56-7129-4e19-9911-5720b9668058-metrics-tls") pod "dns-default-twkvj" (UID: "06885d56-7129-4e19-9911-5720b9668058") : secret "dns-default-metrics-tls" not found
Jul 21 01:36:57.034499 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:57.034402    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 21 01:36:57.034499 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:57.034497    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9b98f9dc-g4qkb: secret "image-registry-tls" not found
Jul 21 01:36:57.034560 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:57.034547    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-registry-tls podName:122f620b-920e-4174-8658-489db0d018ad nodeName:}" failed. No retries permitted until 2026-07-21 01:36:58.034534789 +0000 UTC m=+37.632175136 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-registry-tls") pod "image-registry-b9b98f9dc-g4qkb" (UID: "122f620b-920e-4174-8658-489db0d018ad") : secret "image-registry-tls" not found
Jul 21 01:36:57.159525 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:57.159496    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:36:57.306081 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:57.306015    2575 generic.go:356] "Generic (PLEG): container finished" podID="84e8ef89-8a9f-4f94-9d58-3f650d213260" containerID="321d78c9cb741b42e900c23b600fd93fb0bc83b8bc4acaec2f84f1c9846d1db7" exitCode=0
Jul 21 01:36:57.306214 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:57.306087    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gl7hj" event={"ID":"84e8ef89-8a9f-4f94-9d58-3f650d213260","Type":"ContainerDied","Data":"321d78c9cb741b42e900c23b600fd93fb0bc83b8bc4acaec2f84f1c9846d1db7"}
Jul 21 01:36:57.307185 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:57.307161    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-drc4p" event={"ID":"86db8e5d-3869-4ad0-8d3d-989e132f6038","Type":"ContainerStarted","Data":"a8b60b949ef7ced1b6a2948b9b69e3394017af2f99f95da0db7ef73fd5d488f6"}
Jul 21 01:36:57.331119 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:57.331088    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:36:57.391727 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:57.391704    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:36:57.491796 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:57.491770    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:36:58.019717 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:58.019687    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:36:58.022602 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:58.022578    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 21 01:36:58.042331 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:58.042302    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-registry-tls\") pod \"image-registry-b9b98f9dc-g4qkb\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") " pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:36:58.042459 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:58.042364    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06885d56-7129-4e19-9911-5720b9668058-metrics-tls\") pod \"dns-default-twkvj\" (UID: \"06885d56-7129-4e19-9911-5720b9668058\") " pod="openshift-dns/dns-default-twkvj"
Jul 21 01:36:58.042515 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:58.042459    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 21 01:36:58.042515 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:58.042477    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9b98f9dc-g4qkb: secret "image-registry-tls" not found
Jul 21 01:36:58.042515 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:58.042509    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 21 01:36:58.042655 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:58.042535    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-registry-tls podName:122f620b-920e-4174-8658-489db0d018ad nodeName:}" failed. No retries permitted until 2026-07-21 01:37:00.042515076 +0000 UTC m=+39.640155422 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-registry-tls") pod "image-registry-b9b98f9dc-g4qkb" (UID: "122f620b-920e-4174-8658-489db0d018ad") : secret "image-registry-tls" not found
Jul 21 01:36:58.042655 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:36:58.042554    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06885d56-7129-4e19-9911-5720b9668058-metrics-tls podName:06885d56-7129-4e19-9911-5720b9668058 nodeName:}" failed. No retries permitted until 2026-07-21 01:37:00.042544086 +0000 UTC m=+39.640184417 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06885d56-7129-4e19-9911-5720b9668058-metrics-tls") pod "dns-default-twkvj" (UID: "06885d56-7129-4e19-9911-5720b9668058") : secret "dns-default-metrics-tls" not found
Jul 21 01:36:58.312810 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:58.312729    2575 generic.go:356] "Generic (PLEG): container finished" podID="84e8ef89-8a9f-4f94-9d58-3f650d213260" containerID="87d0e147e673c715c17436794793ebdad0d37e3fb215c18cb23957e2aa3f60d3" exitCode=0
Jul 21 01:36:58.312810 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:58.312787    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gl7hj" event={"ID":"84e8ef89-8a9f-4f94-9d58-3f650d213260","Type":"ContainerDied","Data":"87d0e147e673c715c17436794793ebdad0d37e3fb215c18cb23957e2aa3f60d3"}
Jul 21 01:36:59.320208 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:59.320036    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gl7hj" event={"ID":"84e8ef89-8a9f-4f94-9d58-3f650d213260","Type":"ContainerStarted","Data":"4502a5c9470fc7ff2434281c5d4167fe2eb3e3d09f9e4b4bb77bab6694608242"}
Jul 21 01:36:59.321953 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:59.321924    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-drc4p" event={"ID":"86db8e5d-3869-4ad0-8d3d-989e132f6038","Type":"ContainerStarted","Data":"f24f89455a7a4f98d480ed6308c30813c2f343cfa652aa967f49504f8970d81f"}
Jul 21 01:36:59.365642 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:59.365515    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gl7hj" podStartSLOduration=8.717284206 podStartE2EDuration="38.365497151s" totalImagesPullingTime="29.648212945s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 01:36:21 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:36:59.348092872 +0000 UTC m=+38.945733224" watchObservedRunningTime="2026-07-21 01:36:59.365497151 +0000 UTC m=+38.963137502"
Jul 21 01:36:59.366019 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:59.365988    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-drc4p" podStartSLOduration=1.068862177 podStartE2EDuration="3.365978285s" totalImagesPullingTime="2.297116108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:36:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:36:59.364844484 +0000 UTC m=+38.962484829" watchObservedRunningTime="2026-07-21 01:36:59.365978285 +0000 UTC m=+38.963618636"
Jul 21 01:36:59.484852 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:59.484822    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:36:59.554722 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:36:59.554695    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:00.057768 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:00.057726    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-registry-tls\") pod \"image-registry-b9b98f9dc-g4qkb\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") " pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:37:00.057889 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:00.057783    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06885d56-7129-4e19-9911-5720b9668058-metrics-tls\") pod \"dns-default-twkvj\" (UID: \"06885d56-7129-4e19-9911-5720b9668058\") " pod="openshift-dns/dns-default-twkvj"
Jul 21 01:37:00.057889 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:00.057870    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 21 01:37:00.057889 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:00.057876    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 21 01:37:00.057983 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:00.057894    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9b98f9dc-g4qkb: secret "image-registry-tls" not found
Jul 21 01:37:00.057983 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:00.057922    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06885d56-7129-4e19-9911-5720b9668058-metrics-tls podName:06885d56-7129-4e19-9911-5720b9668058 nodeName:}" failed. No retries permitted until 2026-07-21 01:37:04.05790831 +0000 UTC m=+43.655548652 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06885d56-7129-4e19-9911-5720b9668058-metrics-tls") pod "dns-default-twkvj" (UID: "06885d56-7129-4e19-9911-5720b9668058") : secret "dns-default-metrics-tls" not found
Jul 21 01:37:00.057983 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:00.057947    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-registry-tls podName:122f620b-920e-4174-8658-489db0d018ad nodeName:}" failed. No retries permitted until 2026-07-21 01:37:04.057934227 +0000 UTC m=+43.655574556 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-registry-tls") pod "image-registry-b9b98f9dc-g4qkb" (UID: "122f620b-920e-4174-8658-489db0d018ad") : secret "image-registry-tls" not found
Jul 21 01:37:00.796097 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:00.795928    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:00.854804 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:00.854772    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:02.020316 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:02.020285    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:02.076784 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:02.076758    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:02.578164 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:02.578130    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:02.678997 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:02.678967    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:03.215387 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:03.215359    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:03.318487 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:03.318455    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:03.403063 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:03.403032    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:04.085772 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:04.085736    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06885d56-7129-4e19-9911-5720b9668058-metrics-tls\") pod \"dns-default-twkvj\" (UID: \"06885d56-7129-4e19-9911-5720b9668058\") " pod="openshift-dns/dns-default-twkvj"
Jul 21 01:37:04.085934 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:04.085806    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-registry-tls\") pod \"image-registry-b9b98f9dc-g4qkb\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") " pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:37:04.085934 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:04.085901    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 21 01:37:04.085997 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:04.085970    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06885d56-7129-4e19-9911-5720b9668058-metrics-tls podName:06885d56-7129-4e19-9911-5720b9668058 nodeName:}" failed. No retries permitted until 2026-07-21 01:37:12.085955374 +0000 UTC m=+51.683595702 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06885d56-7129-4e19-9911-5720b9668058-metrics-tls") pod "dns-default-twkvj" (UID: "06885d56-7129-4e19-9911-5720b9668058") : secret "dns-default-metrics-tls" not found
Jul 21 01:37:04.086038 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:04.085904    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 21 01:37:04.086038 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:04.086015    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9b98f9dc-g4qkb: secret "image-registry-tls" not found
Jul 21 01:37:04.086097 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:04.086071    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-registry-tls podName:122f620b-920e-4174-8658-489db0d018ad nodeName:}" failed. No retries permitted until 2026-07-21 01:37:12.086058958 +0000 UTC m=+51.683699302 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-registry-tls") pod "image-registry-b9b98f9dc-g4qkb" (UID: "122f620b-920e-4174-8658-489db0d018ad") : secret "image-registry-tls" not found
Jul 21 01:37:04.261807 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:04.261771    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:04.359038 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:04.358972    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:04.458848 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:04.458823    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:04.520976 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:04.520954    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:04.760761 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:04.760688    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2v6dg"]
Jul 21 01:37:04.781563 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:04.781538    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2v6dg"]
Jul 21 01:37:04.781691 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:04.781657    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2v6dg"
Jul 21 01:37:04.781892 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:04.781869    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:04.783942 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:04.783923    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 21 01:37:04.784092 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:04.784063    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5pn52\""
Jul 21 01:37:04.784168 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:04.784148    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 21 01:37:04.784842 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:04.784822    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 21 01:37:04.784842 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:04.784831    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 21 01:37:04.789843 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:04.789822    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7971e50e-1094-40f1-a8e4-56eb51aa83d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2v6dg\" (UID: \"7971e50e-1094-40f1-a8e4-56eb51aa83d4\") " pod="openshift-insights/insights-runtime-extractor-2v6dg"
Jul 21 01:37:04.789964 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:04.789946    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7971e50e-1094-40f1-a8e4-56eb51aa83d4-crio-socket\") pod \"insights-runtime-extractor-2v6dg\" (UID: \"7971e50e-1094-40f1-a8e4-56eb51aa83d4\") " pod="openshift-insights/insights-runtime-extractor-2v6dg"
Jul 21 01:37:04.790004 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:04.789989    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7971e50e-1094-40f1-a8e4-56eb51aa83d4-data-volume\") pod \"insights-runtime-extractor-2v6dg\" (UID: \"7971e50e-1094-40f1-a8e4-56eb51aa83d4\") " pod="openshift-insights/insights-runtime-extractor-2v6dg"
Jul 21 01:37:04.790037 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:04.790017    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7971e50e-1094-40f1-a8e4-56eb51aa83d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2v6dg\" (UID: \"7971e50e-1094-40f1-a8e4-56eb51aa83d4\") " pod="openshift-insights/insights-runtime-extractor-2v6dg"
Jul 21 01:37:04.790069 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:04.790040    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzt2c\" (UniqueName: \"kubernetes.io/projected/7971e50e-1094-40f1-a8e4-56eb51aa83d4-kube-api-access-hzt2c\") pod \"insights-runtime-extractor-2v6dg\" (UID: \"7971e50e-1094-40f1-a8e4-56eb51aa83d4\") " pod="openshift-insights/insights-runtime-extractor-2v6dg"
Jul 21 01:37:04.842883 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:04.842860    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:04.891031 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:04.891007    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7971e50e-1094-40f1-a8e4-56eb51aa83d4-crio-socket\") pod \"insights-runtime-extractor-2v6dg\" (UID: \"7971e50e-1094-40f1-a8e4-56eb51aa83d4\") " pod="openshift-insights/insights-runtime-extractor-2v6dg"
Jul 21 01:37:04.891121 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:04.891035    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7971e50e-1094-40f1-a8e4-56eb51aa83d4-data-volume\") pod \"insights-runtime-extractor-2v6dg\" (UID: \"7971e50e-1094-40f1-a8e4-56eb51aa83d4\") " pod="openshift-insights/insights-runtime-extractor-2v6dg"
Jul 21 01:37:04.891121 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:04.891054    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7971e50e-1094-40f1-a8e4-56eb51aa83d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2v6dg\" (UID: \"7971e50e-1094-40f1-a8e4-56eb51aa83d4\") " pod="openshift-insights/insights-runtime-extractor-2v6dg"
Jul 21 01:37:04.891121 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:04.891071    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzt2c\" (UniqueName: \"kubernetes.io/projected/7971e50e-1094-40f1-a8e4-56eb51aa83d4-kube-api-access-hzt2c\") pod \"insights-runtime-extractor-2v6dg\" (UID: \"7971e50e-1094-40f1-a8e4-56eb51aa83d4\") " pod="openshift-insights/insights-runtime-extractor-2v6dg"
Jul 21 01:37:04.891264 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:04.891121    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7971e50e-1094-40f1-a8e4-56eb51aa83d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2v6dg\" (UID: \"7971e50e-1094-40f1-a8e4-56eb51aa83d4\") " pod="openshift-insights/insights-runtime-extractor-2v6dg"
Jul 21 01:37:04.891264 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:04.891185    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 21 01:37:04.891264 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:04.891219    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 21 01:37:04.891264 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:04.891226    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7971e50e-1094-40f1-a8e4-56eb51aa83d4-crio-socket\") pod \"insights-runtime-extractor-2v6dg\" (UID: \"7971e50e-1094-40f1-a8e4-56eb51aa83d4\") " pod="openshift-insights/insights-runtime-extractor-2v6dg"
Jul 21 01:37:04.891264 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:04.891246    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7971e50e-1094-40f1-a8e4-56eb51aa83d4-insights-runtime-extractor-tls podName:7971e50e-1094-40f1-a8e4-56eb51aa83d4 nodeName:}" failed. No retries permitted until 2026-07-21 01:37:05.391230502 +0000 UTC m=+44.988870845 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7971e50e-1094-40f1-a8e4-56eb51aa83d4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2v6dg" (UID: "7971e50e-1094-40f1-a8e4-56eb51aa83d4") : secret "insights-runtime-extractor-tls" not found
Jul 21 01:37:04.891264 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:04.891263    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7971e50e-1094-40f1-a8e4-56eb51aa83d4-kube-rbac-proxy-cm podName:7971e50e-1094-40f1-a8e4-56eb51aa83d4 nodeName:}" failed. No retries permitted until 2026-07-21 01:37:05.391254423 +0000 UTC m=+44.988894754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7971e50e-1094-40f1-a8e4-56eb51aa83d4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2v6dg" (UID: "7971e50e-1094-40f1-a8e4-56eb51aa83d4") : configmap "kube-rbac-proxy" not found
Jul 21 01:37:04.891458 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:04.891352    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7971e50e-1094-40f1-a8e4-56eb51aa83d4-data-volume\") pod \"insights-runtime-extractor-2v6dg\" (UID: \"7971e50e-1094-40f1-a8e4-56eb51aa83d4\") " pod="openshift-insights/insights-runtime-extractor-2v6dg"
Jul 21 01:37:04.903002 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:04.902978    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzt2c\" (UniqueName: \"kubernetes.io/projected/7971e50e-1094-40f1-a8e4-56eb51aa83d4-kube-api-access-hzt2c\") pod \"insights-runtime-extractor-2v6dg\" (UID: \"7971e50e-1094-40f1-a8e4-56eb51aa83d4\") " pod="openshift-insights/insights-runtime-extractor-2v6dg"
Jul 21 01:37:04.950433 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:04.950406    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:05.048981 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:05.048952    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:05.319422 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:05.319343    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:05.377231 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:05.377204    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:05.394471 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:05.394449    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7971e50e-1094-40f1-a8e4-56eb51aa83d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2v6dg\" (UID: \"7971e50e-1094-40f1-a8e4-56eb51aa83d4\") " pod="openshift-insights/insights-runtime-extractor-2v6dg"
Jul 21 01:37:05.394564 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:05.394492    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7971e50e-1094-40f1-a8e4-56eb51aa83d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2v6dg\" (UID: \"7971e50e-1094-40f1-a8e4-56eb51aa83d4\") " pod="openshift-insights/insights-runtime-extractor-2v6dg"
Jul 21 01:37:05.394647 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:05.394630    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 21 01:37:05.394709 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:05.394644    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 21 01:37:05.394709 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:05.394697    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7971e50e-1094-40f1-a8e4-56eb51aa83d4-kube-rbac-proxy-cm podName:7971e50e-1094-40f1-a8e4-56eb51aa83d4 nodeName:}" failed. No retries permitted until 2026-07-21 01:37:06.394683734 +0000 UTC m=+45.992324063 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7971e50e-1094-40f1-a8e4-56eb51aa83d4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2v6dg" (UID: "7971e50e-1094-40f1-a8e4-56eb51aa83d4") : configmap "kube-rbac-proxy" not found
Jul 21 01:37:05.394815 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:05.394711    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7971e50e-1094-40f1-a8e4-56eb51aa83d4-insights-runtime-extractor-tls podName:7971e50e-1094-40f1-a8e4-56eb51aa83d4 nodeName:}" failed. No retries permitted until 2026-07-21 01:37:06.394703774 +0000 UTC m=+45.992344102 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7971e50e-1094-40f1-a8e4-56eb51aa83d4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2v6dg" (UID: "7971e50e-1094-40f1-a8e4-56eb51aa83d4") : secret "insights-runtime-extractor-tls" not found
Jul 21 01:37:06.402740 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:06.402704    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7971e50e-1094-40f1-a8e4-56eb51aa83d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2v6dg\" (UID: \"7971e50e-1094-40f1-a8e4-56eb51aa83d4\") " pod="openshift-insights/insights-runtime-extractor-2v6dg"
Jul 21 01:37:06.403187 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:06.402765    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7971e50e-1094-40f1-a8e4-56eb51aa83d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2v6dg\" (UID: \"7971e50e-1094-40f1-a8e4-56eb51aa83d4\") " pod="openshift-insights/insights-runtime-extractor-2v6dg"
Jul 21 01:37:06.403187 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:06.402836    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 21 01:37:06.403187 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:06.402884    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 21 01:37:06.403187 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:06.402892    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7971e50e-1094-40f1-a8e4-56eb51aa83d4-insights-runtime-extractor-tls podName:7971e50e-1094-40f1-a8e4-56eb51aa83d4 nodeName:}" failed. No retries permitted until 2026-07-21 01:37:08.402878049 +0000 UTC m=+48.000518378 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7971e50e-1094-40f1-a8e4-56eb51aa83d4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2v6dg" (UID: "7971e50e-1094-40f1-a8e4-56eb51aa83d4") : secret "insights-runtime-extractor-tls" not found
Jul 21 01:37:06.403187 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:06.402929    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7971e50e-1094-40f1-a8e4-56eb51aa83d4-kube-rbac-proxy-cm podName:7971e50e-1094-40f1-a8e4-56eb51aa83d4 nodeName:}" failed. No retries permitted until 2026-07-21 01:37:08.402916458 +0000 UTC m=+48.000556791 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7971e50e-1094-40f1-a8e4-56eb51aa83d4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2v6dg" (UID: "7971e50e-1094-40f1-a8e4-56eb51aa83d4") : configmap "kube-rbac-proxy" not found
Jul 21 01:37:06.537776 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:06.537749    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-x7gxz"]
Jul 21 01:37:06.559303 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:06.559277    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-x7gxz"]
Jul 21 01:37:06.559420 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:06.559360    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-x7gxz"
Jul 21 01:37:06.561564 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:06.561542    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 21 01:37:06.561774 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:06.561762    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 21 01:37:06.562527 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:06.562508    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-gn59x\""
Jul 21 01:37:06.562527 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:06.562518    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 21 01:37:06.562692 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:06.562514    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 21 01:37:06.576022 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:06.576002    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:06.605332 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:06.605311    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgpvv\" (UniqueName: \"kubernetes.io/projected/4388371b-ede8-4dcc-954f-59e2fb8c5e02-kube-api-access-hgpvv\") pod \"service-ca-58598695f7-x7gxz\" (UID: \"4388371b-ede8-4dcc-954f-59e2fb8c5e02\") " pod="openshift-service-ca/service-ca-58598695f7-x7gxz"
Jul 21 01:37:06.605405 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:06.605372    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4388371b-ede8-4dcc-954f-59e2fb8c5e02-signing-cabundle\") pod \"service-ca-58598695f7-x7gxz\" (UID: \"4388371b-ede8-4dcc-954f-59e2fb8c5e02\") " pod="openshift-service-ca/service-ca-58598695f7-x7gxz"
Jul 21 01:37:06.605444 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:06.605418    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4388371b-ede8-4dcc-954f-59e2fb8c5e02-signing-key\") pod \"service-ca-58598695f7-x7gxz\" (UID: \"4388371b-ede8-4dcc-954f-59e2fb8c5e02\") " pod="openshift-service-ca/service-ca-58598695f7-x7gxz"
Jul 21 01:37:06.673135 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:06.673071    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:06.706683 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:06.706661    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgpvv\" (UniqueName: \"kubernetes.io/projected/4388371b-ede8-4dcc-954f-59e2fb8c5e02-kube-api-access-hgpvv\") pod \"service-ca-58598695f7-x7gxz\" (UID: \"4388371b-ede8-4dcc-954f-59e2fb8c5e02\") " pod="openshift-service-ca/service-ca-58598695f7-x7gxz"
Jul 21 01:37:06.706766 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:06.706723    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4388371b-ede8-4dcc-954f-59e2fb8c5e02-signing-cabundle\") pod \"service-ca-58598695f7-x7gxz\" (UID: \"4388371b-ede8-4dcc-954f-59e2fb8c5e02\") " pod="openshift-service-ca/service-ca-58598695f7-x7gxz"
Jul 21 01:37:06.706766 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:06.706756    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4388371b-ede8-4dcc-954f-59e2fb8c5e02-signing-key\") pod \"service-ca-58598695f7-x7gxz\" (UID: \"4388371b-ede8-4dcc-954f-59e2fb8c5e02\") " pod="openshift-service-ca/service-ca-58598695f7-x7gxz"
Jul 21 01:37:06.707353 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:06.707336    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4388371b-ede8-4dcc-954f-59e2fb8c5e02-signing-cabundle\") pod \"service-ca-58598695f7-x7gxz\" (UID: \"4388371b-ede8-4dcc-954f-59e2fb8c5e02\") " pod="openshift-service-ca/service-ca-58598695f7-x7gxz"
Jul 21 01:37:06.709050 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:06.709034    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4388371b-ede8-4dcc-954f-59e2fb8c5e02-signing-key\") pod \"service-ca-58598695f7-x7gxz\" (UID: \"4388371b-ede8-4dcc-954f-59e2fb8c5e02\") " pod="openshift-service-ca/service-ca-58598695f7-x7gxz"
Jul 21 01:37:06.713781 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:06.713757    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgpvv\" (UniqueName: \"kubernetes.io/projected/4388371b-ede8-4dcc-954f-59e2fb8c5e02-kube-api-access-hgpvv\") pod \"service-ca-58598695f7-x7gxz\" (UID: \"4388371b-ede8-4dcc-954f-59e2fb8c5e02\") " pod="openshift-service-ca/service-ca-58598695f7-x7gxz"
Jul 21 01:37:06.867384 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:06.867345    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-x7gxz"
Jul 21 01:37:06.944828 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:06.944799    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fw8b6_ed12def9-d42e-4d3e-98b2-d870a1c43231/dns-node-resolver/0.log"
Jul 21 01:37:07.008394 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:07.008351    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-x7gxz"]
Jul 21 01:37:07.012038 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:37:07.012013    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4388371b_ede8_4dcc_954f_59e2fb8c5e02.slice/crio-d2550d45e0be46a4e069c4f528f23d60008724d59bba61d3215198be0d9d8f9a WatchSource:0}: Error finding container d2550d45e0be46a4e069c4f528f23d60008724d59bba61d3215198be0d9d8f9a: Status 404 returned error can't find the container with id d2550d45e0be46a4e069c4f528f23d60008724d59bba61d3215198be0d9d8f9a
Jul 21 01:37:07.340005 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:07.339977    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-x7gxz" event={"ID":"4388371b-ede8-4dcc-954f-59e2fb8c5e02","Type":"ContainerStarted","Data":"d2550d45e0be46a4e069c4f528f23d60008724d59bba61d3215198be0d9d8f9a"}
Jul 21 01:37:07.478512 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:07.478483    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:07.745660 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:07.745577    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kzvp4_7073f999-4de0-4a60-b3a2-58e9e527ffea/node-ca/0.log"
Jul 21 01:37:08.420684 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:08.420646    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7971e50e-1094-40f1-a8e4-56eb51aa83d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2v6dg\" (UID: \"7971e50e-1094-40f1-a8e4-56eb51aa83d4\") " pod="openshift-insights/insights-runtime-extractor-2v6dg"
Jul 21 01:37:08.420865 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:08.420729    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7971e50e-1094-40f1-a8e4-56eb51aa83d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2v6dg\" (UID: \"7971e50e-1094-40f1-a8e4-56eb51aa83d4\") " pod="openshift-insights/insights-runtime-extractor-2v6dg"
Jul 21 01:37:08.420865 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:08.420811    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 21 01:37:08.420979 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:08.420858    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 21 01:37:08.420979 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:08.420892    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7971e50e-1094-40f1-a8e4-56eb51aa83d4-insights-runtime-extractor-tls podName:7971e50e-1094-40f1-a8e4-56eb51aa83d4 nodeName:}" failed. No retries permitted until 2026-07-21 01:37:12.420868333 +0000 UTC m=+52.018508662 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7971e50e-1094-40f1-a8e4-56eb51aa83d4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2v6dg" (UID: "7971e50e-1094-40f1-a8e4-56eb51aa83d4") : secret "insights-runtime-extractor-tls" not found
Jul 21 01:37:08.420979 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:08.420923    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7971e50e-1094-40f1-a8e4-56eb51aa83d4-kube-rbac-proxy-cm podName:7971e50e-1094-40f1-a8e4-56eb51aa83d4 nodeName:}" failed. No retries permitted until 2026-07-21 01:37:12.420909921 +0000 UTC m=+52.018550249 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7971e50e-1094-40f1-a8e4-56eb51aa83d4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2v6dg" (UID: "7971e50e-1094-40f1-a8e4-56eb51aa83d4") : configmap "kube-rbac-proxy" not found
Jul 21 01:37:08.745043 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:08.744926    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-drc4p_86db8e5d-3869-4ad0-8d3d-989e132f6038/serve-healthcheck-canary/0.log"
Jul 21 01:37:08.796007 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:08.795973    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:10.347848 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:10.347814    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-x7gxz" event={"ID":"4388371b-ede8-4dcc-954f-59e2fb8c5e02","Type":"ContainerStarted","Data":"4427892f92fc0eee61a6b0f07d60d69e41137612d18bb81318559b36efc95a25"}
Jul 21 01:37:10.364492 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:10.364373    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-x7gxz" podStartSLOduration=2.112777408 podStartE2EDuration="4.36435483s" totalImagesPullingTime="2.251577422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:37:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:37:10.363969964 +0000 UTC m=+49.961610319" watchObservedRunningTime="2026-07-21 01:37:10.36435483 +0000 UTC m=+49.961995185"
Jul 21 01:37:10.552834 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:10.552805    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:10.875515 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:10.875452    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:10.999002 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:10.998968    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:11.135821 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:11.135721    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:11.228829 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:11.228797    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:11.293973 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:11.293942    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:11.966516 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:11.966485    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:12.150896 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:12.150863    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-registry-tls\") pod \"image-registry-b9b98f9dc-g4qkb\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") " pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:37:12.151066 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:12.150914    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06885d56-7129-4e19-9911-5720b9668058-metrics-tls\") pod \"dns-default-twkvj\" (UID: \"06885d56-7129-4e19-9911-5720b9668058\") " pod="openshift-dns/dns-default-twkvj"
Jul 21 01:37:12.151066 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:12.151046    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 21 01:37:12.151066 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:12.151062    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9b98f9dc-g4qkb: secret "image-registry-tls" not found
Jul 21 01:37:12.151222 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:12.151113    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-registry-tls podName:122f620b-920e-4174-8658-489db0d018ad nodeName:}" failed. No retries permitted until 2026-07-21 01:37:28.151099547 +0000 UTC m=+67.748739876 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-registry-tls") pod "image-registry-b9b98f9dc-g4qkb" (UID: "122f620b-920e-4174-8658-489db0d018ad") : secret "image-registry-tls" not found
Jul 21 01:37:12.151222 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:12.151129    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 21 01:37:12.151222 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:12.151183    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06885d56-7129-4e19-9911-5720b9668058-metrics-tls podName:06885d56-7129-4e19-9911-5720b9668058 nodeName:}" failed. No retries permitted until 2026-07-21 01:37:28.151169908 +0000 UTC m=+67.748810242 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06885d56-7129-4e19-9911-5720b9668058-metrics-tls") pod "dns-default-twkvj" (UID: "06885d56-7129-4e19-9911-5720b9668058") : secret "dns-default-metrics-tls" not found
Jul 21 01:37:12.362888 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:12.362854    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:12.452011 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:12.451982    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7971e50e-1094-40f1-a8e4-56eb51aa83d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2v6dg\" (UID: \"7971e50e-1094-40f1-a8e4-56eb51aa83d4\") " pod="openshift-insights/insights-runtime-extractor-2v6dg"
Jul 21 01:37:12.452147 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:12.452037    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7971e50e-1094-40f1-a8e4-56eb51aa83d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2v6dg\" (UID: \"7971e50e-1094-40f1-a8e4-56eb51aa83d4\") " pod="openshift-insights/insights-runtime-extractor-2v6dg"
Jul 21 01:37:12.452186 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:12.452145    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 21 01:37:12.452217 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:12.452194    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7971e50e-1094-40f1-a8e4-56eb51aa83d4-insights-runtime-extractor-tls podName:7971e50e-1094-40f1-a8e4-56eb51aa83d4 nodeName:}" failed. No retries permitted until 2026-07-21 01:37:20.452180924 +0000 UTC m=+60.049821255 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7971e50e-1094-40f1-a8e4-56eb51aa83d4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2v6dg" (UID: "7971e50e-1094-40f1-a8e4-56eb51aa83d4") : secret "insights-runtime-extractor-tls" not found
Jul 21 01:37:12.452588 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:12.452562    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7971e50e-1094-40f1-a8e4-56eb51aa83d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2v6dg\" (UID: \"7971e50e-1094-40f1-a8e4-56eb51aa83d4\") " pod="openshift-insights/insights-runtime-extractor-2v6dg"
Jul 21 01:37:12.769027 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:12.768991    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:12.964998 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:12.964967    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:13.374705 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:13.374672    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:13.945402 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:13.945365    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:14.003688 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:14.003646    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:14.369114 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:14.369071    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:14.568734 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:14.568707    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:14.829335 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:14.829276    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:14.883760 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:14.883722    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:15.166044 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:15.165970    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:15.569562 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:15.569532    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:15.766316 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:15.766286    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:16.173837 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:16.173806    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:16.275149 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:16.275122    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:16.374651 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:16.374625    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:16.768248 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:16.768219    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:17.070625 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:17.070540    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:17.170931 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:17.170904    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:17.572552 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:17.572523    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:17.777846 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:17.777815    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:18.371343 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:18.371315    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:18.771648 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:18.771600    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:18.968925 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:18.968872    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:19.421282 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:19.421256    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:19.977317 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:19.977284    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:20.392166 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:20.392135    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:20.514935 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:20.514891    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7971e50e-1094-40f1-a8e4-56eb51aa83d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2v6dg\" (UID: \"7971e50e-1094-40f1-a8e4-56eb51aa83d4\") " pod="openshift-insights/insights-runtime-extractor-2v6dg"
Jul 21 01:37:20.517112 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:20.517089    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7971e50e-1094-40f1-a8e4-56eb51aa83d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2v6dg\" (UID: \"7971e50e-1094-40f1-a8e4-56eb51aa83d4\") " pod="openshift-insights/insights-runtime-extractor-2v6dg"
Jul 21 01:37:20.690547 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:20.690451    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2v6dg"
Jul 21 01:37:20.785040 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:20.785011    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:20.809592 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:20.809574    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2v6dg"]
Jul 21 01:37:20.960770 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:20.960689    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:21.061812 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:21.061784    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:21.133547 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:21.133521    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:21.375924 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:21.375883    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:21.376130 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:21.376103    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2v6dg" event={"ID":"7971e50e-1094-40f1-a8e4-56eb51aa83d4","Type":"ContainerStarted","Data":"a5a7f6fd54bf743c04b0b2f7aefb6c2e783460511861adfa310f1e97be9333e3"}
Jul 21 01:37:21.570064 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:21.570025    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:22.302467 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:22.302441    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vzgqj"
Jul 21 01:37:26.391500 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:26.391457    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2v6dg" event={"ID":"7971e50e-1094-40f1-a8e4-56eb51aa83d4","Type":"ContainerStarted","Data":"fe1ae8ea429f47e8c64f5f7b323d86f84eaaef47748493e67bc61a5f714f0308"}
Jul 21 01:37:26.663604 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:26.663516    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ee75f94-10ca-4a6a-a2f4-077eb597a94f-original-pull-secret\") pod \"global-pull-secret-syncer-pmhdx\" (UID: \"5ee75f94-10ca-4a6a-a2f4-077eb597a94f\") " pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:37:26.666077 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:26.666045    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 21 01:37:26.677134 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:26.677108    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ee75f94-10ca-4a6a-a2f4-077eb597a94f-original-pull-secret\") pod \"global-pull-secret-syncer-pmhdx\" (UID: \"5ee75f94-10ca-4a6a-a2f4-077eb597a94f\") " pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:37:26.764904 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:26.764869    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12e288ad-0d62-4e00-9a2a-1de85626abb9-metrics-certs\") pod \"network-metrics-daemon-245g5\" (UID: \"12e288ad-0d62-4e00-9a2a-1de85626abb9\") " pod="openshift-multus/network-metrics-daemon-245g5"
Jul 21 01:37:26.767136 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:26.767114    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 21 01:37:26.777946 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:26.777922    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12e288ad-0d62-4e00-9a2a-1de85626abb9-metrics-certs\") pod \"network-metrics-daemon-245g5\" (UID: \"12e288ad-0d62-4e00-9a2a-1de85626abb9\") " pod="openshift-multus/network-metrics-daemon-245g5"
Jul 21 01:37:26.829264 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:26.829232    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmhdx"
Jul 21 01:37:26.865987 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:26.865926    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpxtj\" (UniqueName: \"kubernetes.io/projected/26238877-47e2-4ed6-8df9-c21f38f7578c-kube-api-access-dpxtj\") pod \"network-check-target-xbk6c\" (UID: \"26238877-47e2-4ed6-8df9-c21f38f7578c\") " pod="openshift-network-diagnostics/network-check-target-xbk6c"
Jul 21 01:37:26.868102 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:26.868081    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 21 01:37:26.878577 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:26.878488    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 21 01:37:26.890798 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:26.890771    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpxtj\" (UniqueName: \"kubernetes.io/projected/26238877-47e2-4ed6-8df9-c21f38f7578c-kube-api-access-dpxtj\") pod \"network-check-target-xbk6c\" (UID: \"26238877-47e2-4ed6-8df9-c21f38f7578c\") " pod="openshift-network-diagnostics/network-check-target-xbk6c"
Jul 21 01:37:26.966313 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:26.966285    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pmhdx"]
Jul 21 01:37:26.970287 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:37:26.970258    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ee75f94_10ca_4a6a_a2f4_077eb597a94f.slice/crio-c0df9a8c0270a2959208bfada69610793a9212ee367050413bd710b553a2bd69 WatchSource:0}: Error finding container c0df9a8c0270a2959208bfada69610793a9212ee367050413bd710b553a2bd69: Status 404 returned error can't find the container with id c0df9a8c0270a2959208bfada69610793a9212ee367050413bd710b553a2bd69
Jul 21 01:37:27.032960 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:27.032932    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8s87n\""
Jul 21 01:37:27.036306 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:27.036280    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5qdkq\""
Jul 21 01:37:27.040872 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:27.040857    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xbk6c"
Jul 21 01:37:27.045527 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:27.045509    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-245g5"
Jul 21 01:37:27.395411 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:27.395373    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pmhdx" event={"ID":"5ee75f94-10ca-4a6a-a2f4-077eb597a94f","Type":"ContainerStarted","Data":"c0df9a8c0270a2959208bfada69610793a9212ee367050413bd710b553a2bd69"}
Jul 21 01:37:27.920013 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:27.919982    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:28.003042 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:28.003011    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:28.108939 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:28.108906    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:28.176206 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:28.176127    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06885d56-7129-4e19-9911-5720b9668058-metrics-tls\") pod \"dns-default-twkvj\" (UID: \"06885d56-7129-4e19-9911-5720b9668058\") " pod="openshift-dns/dns-default-twkvj"
Jul 21 01:37:28.176368 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:28.176213    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-registry-tls\") pod \"image-registry-b9b98f9dc-g4qkb\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") " pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:37:28.178409 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:28.178384    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-registry-tls\") pod \"image-registry-b9b98f9dc-g4qkb\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") " pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:37:28.178409 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:28.178409    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06885d56-7129-4e19-9911-5720b9668058-metrics-tls\") pod \"dns-default-twkvj\" (UID: \"06885d56-7129-4e19-9911-5720b9668058\") " pod="openshift-dns/dns-default-twkvj"
Jul 21 01:37:28.210386 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:28.210354    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:28.369915 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:28.369881    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-r74bp\""
Jul 21 01:37:28.377750 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:28.377716    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:37:28.382205 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:28.382179    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wcxpb\""
Jul 21 01:37:28.391081 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:28.391053    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-twkvj"
Jul 21 01:37:28.581708 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:28.581674    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:28.977905 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:28.977831    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:29.078729 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:29.078632    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:29.147973 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:29.147906    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xbk6c"]
Jul 21 01:37:29.153367 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:37:29.153324    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26238877_47e2_4ed6_8df9_c21f38f7578c.slice/crio-79e2e49cb657ae17bd5e6bffdff6d935dcd38b35d1384df0eb5a9f6d7899ab3f WatchSource:0}: Error finding container 79e2e49cb657ae17bd5e6bffdff6d935dcd38b35d1384df0eb5a9f6d7899ab3f: Status 404 returned error can't find the container with id 79e2e49cb657ae17bd5e6bffdff6d935dcd38b35d1384df0eb5a9f6d7899ab3f
Jul 21 01:37:29.178172 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:29.178133    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-twkvj"]
Jul 21 01:37:29.181943 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:37:29.181901    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06885d56_7129_4e19_9911_5720b9668058.slice/crio-a21c68b7f8ed0581ceb89b19f04143a8af161e0ae9096385042436d40250055c WatchSource:0}: Error finding container a21c68b7f8ed0581ceb89b19f04143a8af161e0ae9096385042436d40250055c: Status 404 returned error can't find the container with id a21c68b7f8ed0581ceb89b19f04143a8af161e0ae9096385042436d40250055c
Jul 21 01:37:29.379812 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:29.379776    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:29.402626 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:29.402335    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-245g5"]
Jul 21 01:37:29.403439 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:29.403410    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-twkvj" event={"ID":"06885d56-7129-4e19-9911-5720b9668058","Type":"ContainerStarted","Data":"a21c68b7f8ed0581ceb89b19f04143a8af161e0ae9096385042436d40250055c"}
Jul 21 01:37:29.405590 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:29.405528    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b9b98f9dc-g4qkb"]
Jul 21 01:37:29.406761 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:29.406698    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2v6dg" event={"ID":"7971e50e-1094-40f1-a8e4-56eb51aa83d4","Type":"ContainerStarted","Data":"d8764bda1198e8be8d69c0394631bed3532a04685619cd6caf485511be72412a"}
Jul 21 01:37:29.408154 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:37:29.407668    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12e288ad_0d62_4e00_9a2a_1de85626abb9.slice/crio-de9c6bad0d550d5b06f366d42aea7b219649a4fc3e8a8b35931676a599368ba6 WatchSource:0}: Error finding container de9c6bad0d550d5b06f366d42aea7b219649a4fc3e8a8b35931676a599368ba6: Status 404 returned error can't find the container with id de9c6bad0d550d5b06f366d42aea7b219649a4fc3e8a8b35931676a599368ba6
Jul 21 01:37:29.408763 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:29.408739    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xbk6c" event={"ID":"26238877-47e2-4ed6-8df9-c21f38f7578c","Type":"ContainerStarted","Data":"79e2e49cb657ae17bd5e6bffdff6d935dcd38b35d1384df0eb5a9f6d7899ab3f"}
Jul 21 01:37:29.409912 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:37:29.409894    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod122f620b_920e_4174_8658_489db0d018ad.slice/crio-4715ed7f47ad0922994b37d8aee5cdbfe2478f0df80f7afb53d91e053821c298 WatchSource:0}: Error finding container 4715ed7f47ad0922994b37d8aee5cdbfe2478f0df80f7afb53d91e053821c298: Status 404 returned error can't find the container with id 4715ed7f47ad0922994b37d8aee5cdbfe2478f0df80f7afb53d91e053821c298
Jul 21 01:37:29.479936 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:29.479904    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:29.593923 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:29.593857    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:29.667796 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:29.667672    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:29.814132 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:29.814034    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:29.920433 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:29.919359    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-75fd5554cc-88fnw"]
Jul 21 01:37:29.922219 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:29.921623    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75fd5554cc-88fnw"
Jul 21 01:37:29.927289 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:29.926801    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 21 01:37:29.931834 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:29.929848    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 21 01:37:29.931834 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:29.930094    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 21 01:37:29.931834 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:29.930312    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 21 01:37:29.949655 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:29.949627    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-75fd5554cc-88fnw"]
Jul 21 01:37:29.989288 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:29.989184    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b9b98f9dc-g4qkb"]
Jul 21 01:37:29.992566 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:29.992376    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4fb11a5e-fad0-4c5e-a935-87bf77622cd2-klusterlet-config\") pod \"klusterlet-addon-workmgr-75fd5554cc-88fnw\" (UID: \"4fb11a5e-fad0-4c5e-a935-87bf77622cd2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75fd5554cc-88fnw"
Jul 21 01:37:29.992566 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:29.992448    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fb11a5e-fad0-4c5e-a935-87bf77622cd2-tmp\") pod \"klusterlet-addon-workmgr-75fd5554cc-88fnw\" (UID: \"4fb11a5e-fad0-4c5e-a935-87bf77622cd2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75fd5554cc-88fnw"
Jul 21 01:37:29.992566 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:29.992484    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzlqs\" (UniqueName: \"kubernetes.io/projected/4fb11a5e-fad0-4c5e-a935-87bf77622cd2-kube-api-access-kzlqs\") pod \"klusterlet-addon-workmgr-75fd5554cc-88fnw\" (UID: \"4fb11a5e-fad0-4c5e-a935-87bf77622cd2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75fd5554cc-88fnw"
Jul 21 01:37:29.998605 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:29.998036    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-q6t9r"]
Jul 21 01:37:30.008592 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.007889    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-q6t9r"
Jul 21 01:37:30.015949 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.015564    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 21 01:37:30.015949 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.015832    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-crrgt\""
Jul 21 01:37:30.016442 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.016276    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 21 01:37:30.040396 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.040335    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-q6t9r"]
Jul 21 01:37:30.045138 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.045073    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:30.078716 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.077699    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79db7cc8cd-f6q5z"]
Jul 21 01:37:30.082991 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.082967    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79db7cc8cd-f6q5z"
Jul 21 01:37:30.093726 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.093212    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4fb11a5e-fad0-4c5e-a935-87bf77622cd2-klusterlet-config\") pod \"klusterlet-addon-workmgr-75fd5554cc-88fnw\" (UID: \"4fb11a5e-fad0-4c5e-a935-87bf77622cd2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75fd5554cc-88fnw"
Jul 21 01:37:30.093726 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.093275    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fb11a5e-fad0-4c5e-a935-87bf77622cd2-tmp\") pod \"klusterlet-addon-workmgr-75fd5554cc-88fnw\" (UID: \"4fb11a5e-fad0-4c5e-a935-87bf77622cd2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75fd5554cc-88fnw"
Jul 21 01:37:30.093726 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.093315    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzlqs\" (UniqueName: \"kubernetes.io/projected/4fb11a5e-fad0-4c5e-a935-87bf77622cd2-kube-api-access-kzlqs\") pod \"klusterlet-addon-workmgr-75fd5554cc-88fnw\" (UID: \"4fb11a5e-fad0-4c5e-a935-87bf77622cd2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75fd5554cc-88fnw"
Jul 21 01:37:30.093726 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.093352    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqdvf\" (UniqueName: \"kubernetes.io/projected/0937c807-6c37-4807-8499-e71c8f8ceb02-kube-api-access-pqdvf\") pod \"downloads-5bc67dcd46-q6t9r\" (UID: \"0937c807-6c37-4807-8499-e71c8f8ceb02\") " pod="openshift-console/downloads-5bc67dcd46-q6t9r"
Jul 21 01:37:30.095962 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.095911    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fb11a5e-fad0-4c5e-a935-87bf77622cd2-tmp\") pod \"klusterlet-addon-workmgr-75fd5554cc-88fnw\" (UID: \"4fb11a5e-fad0-4c5e-a935-87bf77622cd2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75fd5554cc-88fnw"
Jul 21 01:37:30.099583 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.099533    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79db7cc8cd-f6q5z"]
Jul 21 01:37:30.101850 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.101806    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4fb11a5e-fad0-4c5e-a935-87bf77622cd2-klusterlet-config\") pod \"klusterlet-addon-workmgr-75fd5554cc-88fnw\" (UID: \"4fb11a5e-fad0-4c5e-a935-87bf77622cd2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75fd5554cc-88fnw"
Jul 21 01:37:30.115678 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.115658    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzlqs\" (UniqueName: \"kubernetes.io/projected/4fb11a5e-fad0-4c5e-a935-87bf77622cd2-kube-api-access-kzlqs\") pod \"klusterlet-addon-workmgr-75fd5554cc-88fnw\" (UID: \"4fb11a5e-fad0-4c5e-a935-87bf77622cd2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75fd5554cc-88fnw"
Jul 21 01:37:30.170598 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.170467    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:30.194662 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.194605    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f034e8e1-3fdf-4257-986b-34784be403cf-installation-pull-secrets\") pod \"image-registry-79db7cc8cd-f6q5z\" (UID: \"f034e8e1-3fdf-4257-986b-34784be403cf\") " pod="openshift-image-registry/image-registry-79db7cc8cd-f6q5z"
Jul 21 01:37:30.194773 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.194665    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f034e8e1-3fdf-4257-986b-34784be403cf-ca-trust-extracted\") pod \"image-registry-79db7cc8cd-f6q5z\" (UID: \"f034e8e1-3fdf-4257-986b-34784be403cf\") " pod="openshift-image-registry/image-registry-79db7cc8cd-f6q5z"
Jul 21 01:37:30.194773 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.194692    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f034e8e1-3fdf-4257-986b-34784be403cf-registry-tls\") pod \"image-registry-79db7cc8cd-f6q5z\" (UID: \"f034e8e1-3fdf-4257-986b-34784be403cf\") " pod="openshift-image-registry/image-registry-79db7cc8cd-f6q5z"
Jul 21 01:37:30.194773 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.194716    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f034e8e1-3fdf-4257-986b-34784be403cf-trusted-ca\") pod \"image-registry-79db7cc8cd-f6q5z\" (UID: \"f034e8e1-3fdf-4257-986b-34784be403cf\") " pod="openshift-image-registry/image-registry-79db7cc8cd-f6q5z"
Jul 21 01:37:30.194773 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.194749    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f034e8e1-3fdf-4257-986b-34784be403cf-image-registry-private-configuration\") pod \"image-registry-79db7cc8cd-f6q5z\" (UID: \"f034e8e1-3fdf-4257-986b-34784be403cf\") " pod="openshift-image-registry/image-registry-79db7cc8cd-f6q5z"
Jul 21 01:37:30.195182 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.194777    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f034e8e1-3fdf-4257-986b-34784be403cf-bound-sa-token\") pod \"image-registry-79db7cc8cd-f6q5z\" (UID: \"f034e8e1-3fdf-4257-986b-34784be403cf\") " pod="openshift-image-registry/image-registry-79db7cc8cd-f6q5z"
Jul 21 01:37:30.195182 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.194805    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqdvf\" (UniqueName: \"kubernetes.io/projected/0937c807-6c37-4807-8499-e71c8f8ceb02-kube-api-access-pqdvf\") pod \"downloads-5bc67dcd46-q6t9r\" (UID: \"0937c807-6c37-4807-8499-e71c8f8ceb02\") " pod="openshift-console/downloads-5bc67dcd46-q6t9r"
Jul 21 01:37:30.195182 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.194828    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f034e8e1-3fdf-4257-986b-34784be403cf-registry-certificates\") pod \"image-registry-79db7cc8cd-f6q5z\" (UID: \"f034e8e1-3fdf-4257-986b-34784be403cf\") " pod="openshift-image-registry/image-registry-79db7cc8cd-f6q5z"
Jul 21 01:37:30.195182 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.195024    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqzk4\" (UniqueName: \"kubernetes.io/projected/f034e8e1-3fdf-4257-986b-34784be403cf-kube-api-access-zqzk4\") pod \"image-registry-79db7cc8cd-f6q5z\" (UID: \"f034e8e1-3fdf-4257-986b-34784be403cf\") " pod="openshift-image-registry/image-registry-79db7cc8cd-f6q5z"
Jul 21 01:37:30.212893 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.212844    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqdvf\" (UniqueName: \"kubernetes.io/projected/0937c807-6c37-4807-8499-e71c8f8ceb02-kube-api-access-pqdvf\") pod \"downloads-5bc67dcd46-q6t9r\" (UID: \"0937c807-6c37-4807-8499-e71c8f8ceb02\") " pod="openshift-console/downloads-5bc67dcd46-q6t9r"
Jul 21 01:37:30.237110 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.236947    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75fd5554cc-88fnw"
Jul 21 01:37:30.267917 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.267850    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:30.298083 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.296152    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqzk4\" (UniqueName: \"kubernetes.io/projected/f034e8e1-3fdf-4257-986b-34784be403cf-kube-api-access-zqzk4\") pod \"image-registry-79db7cc8cd-f6q5z\" (UID: \"f034e8e1-3fdf-4257-986b-34784be403cf\") " pod="openshift-image-registry/image-registry-79db7cc8cd-f6q5z"
Jul 21 01:37:30.298083 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.296213    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f034e8e1-3fdf-4257-986b-34784be403cf-installation-pull-secrets\") pod \"image-registry-79db7cc8cd-f6q5z\" (UID: \"f034e8e1-3fdf-4257-986b-34784be403cf\") " pod="openshift-image-registry/image-registry-79db7cc8cd-f6q5z"
Jul 21 01:37:30.298083 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.296250    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f034e8e1-3fdf-4257-986b-34784be403cf-ca-trust-extracted\") pod \"image-registry-79db7cc8cd-f6q5z\" (UID: \"f034e8e1-3fdf-4257-986b-34784be403cf\") " pod="openshift-image-registry/image-registry-79db7cc8cd-f6q5z"
Jul 21 01:37:30.298083 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.296279    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f034e8e1-3fdf-4257-986b-34784be403cf-registry-tls\") pod \"image-registry-79db7cc8cd-f6q5z\" (UID: \"f034e8e1-3fdf-4257-986b-34784be403cf\") " pod="openshift-image-registry/image-registry-79db7cc8cd-f6q5z"
Jul 21 01:37:30.298083 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.296302    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f034e8e1-3fdf-4257-986b-34784be403cf-trusted-ca\") pod \"image-registry-79db7cc8cd-f6q5z\" (UID: \"f034e8e1-3fdf-4257-986b-34784be403cf\") " pod="openshift-image-registry/image-registry-79db7cc8cd-f6q5z"
Jul 21 01:37:30.298083 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.296339    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f034e8e1-3fdf-4257-986b-34784be403cf-image-registry-private-configuration\") pod \"image-registry-79db7cc8cd-f6q5z\" (UID: \"f034e8e1-3fdf-4257-986b-34784be403cf\") " pod="openshift-image-registry/image-registry-79db7cc8cd-f6q5z"
Jul 21 01:37:30.298083 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.296367    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f034e8e1-3fdf-4257-986b-34784be403cf-bound-sa-token\") pod \"image-registry-79db7cc8cd-f6q5z\" (UID: \"f034e8e1-3fdf-4257-986b-34784be403cf\") " pod="openshift-image-registry/image-registry-79db7cc8cd-f6q5z"
Jul 21 01:37:30.298083 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.296391    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f034e8e1-3fdf-4257-986b-34784be403cf-registry-certificates\") pod \"image-registry-79db7cc8cd-f6q5z\" (UID: \"f034e8e1-3fdf-4257-986b-34784be403cf\") " pod="openshift-image-registry/image-registry-79db7cc8cd-f6q5z"
Jul 21 01:37:30.298083 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.297306    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f034e8e1-3fdf-4257-986b-34784be403cf-registry-certificates\") pod \"image-registry-79db7cc8cd-f6q5z\" (UID: \"f034e8e1-3fdf-4257-986b-34784be403cf\") " pod="openshift-image-registry/image-registry-79db7cc8cd-f6q5z"
Jul 21 01:37:30.298724 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.298685    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f034e8e1-3fdf-4257-986b-34784be403cf-trusted-ca\") pod \"image-registry-79db7cc8cd-f6q5z\" (UID: \"f034e8e1-3fdf-4257-986b-34784be403cf\") " pod="openshift-image-registry/image-registry-79db7cc8cd-f6q5z"
Jul 21 01:37:30.298973 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.298951    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f034e8e1-3fdf-4257-986b-34784be403cf-ca-trust-extracted\") pod \"image-registry-79db7cc8cd-f6q5z\" (UID: \"f034e8e1-3fdf-4257-986b-34784be403cf\") " pod="openshift-image-registry/image-registry-79db7cc8cd-f6q5z"
Jul 21 01:37:30.303858 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.303804    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f034e8e1-3fdf-4257-986b-34784be403cf-registry-tls\") pod \"image-registry-79db7cc8cd-f6q5z\" (UID: \"f034e8e1-3fdf-4257-986b-34784be403cf\") " pod="openshift-image-registry/image-registry-79db7cc8cd-f6q5z"
Jul 21 01:37:30.305960 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.305934    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f034e8e1-3fdf-4257-986b-34784be403cf-installation-pull-secrets\") pod \"image-registry-79db7cc8cd-f6q5z\" (UID: \"f034e8e1-3fdf-4257-986b-34784be403cf\") " pod="openshift-image-registry/image-registry-79db7cc8cd-f6q5z"
Jul 21 01:37:30.309653 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.309403    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqzk4\" (UniqueName: \"kubernetes.io/projected/f034e8e1-3fdf-4257-986b-34784be403cf-kube-api-access-zqzk4\") pod \"image-registry-79db7cc8cd-f6q5z\" (UID: \"f034e8e1-3fdf-4257-986b-34784be403cf\") " pod="openshift-image-registry/image-registry-79db7cc8cd-f6q5z"
Jul 21 01:37:30.309653 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.309472    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f034e8e1-3fdf-4257-986b-34784be403cf-image-registry-private-configuration\") pod \"image-registry-79db7cc8cd-f6q5z\" (UID: \"f034e8e1-3fdf-4257-986b-34784be403cf\") " pod="openshift-image-registry/image-registry-79db7cc8cd-f6q5z"
Jul 21 01:37:30.314397 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.314338    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f034e8e1-3fdf-4257-986b-34784be403cf-bound-sa-token\") pod \"image-registry-79db7cc8cd-f6q5z\" (UID: \"f034e8e1-3fdf-4257-986b-34784be403cf\") " pod="openshift-image-registry/image-registry-79db7cc8cd-f6q5z"
Jul 21 01:37:30.324957 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.324692    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-q6t9r"
Jul 21 01:37:30.369475 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.369406    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:30.397708 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.397386    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79db7cc8cd-f6q5z"
Jul 21 01:37:30.415993 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.415927    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-75fd5554cc-88fnw"]
Jul 21 01:37:30.417941 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.417908    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb" event={"ID":"122f620b-920e-4174-8658-489db0d018ad","Type":"ContainerStarted","Data":"541a2ce3dd9e30721c6d27ffaf2de20acce99addc1a4d84750c315618c7497a9"}
Jul 21 01:37:30.418033 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.417963    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb" event={"ID":"122f620b-920e-4174-8658-489db0d018ad","Type":"ContainerStarted","Data":"4715ed7f47ad0922994b37d8aee5cdbfe2478f0df80f7afb53d91e053821c298"}
Jul 21 01:37:30.418520 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.418473    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:37:30.420226 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.420174    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-245g5" event={"ID":"12e288ad-0d62-4e00-9a2a-1de85626abb9","Type":"ContainerStarted","Data":"de9c6bad0d550d5b06f366d42aea7b219649a4fc3e8a8b35931676a599368ba6"}
Jul 21 01:37:30.445280 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.444720    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb" podStartSLOduration=61.444702912 podStartE2EDuration="1m1.444702912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:37:30.44380826 +0000 UTC m=+70.041448607" watchObservedRunningTime="2026-07-21 01:37:30.444702912 +0000 UTC m=+70.042343269"
Jul 21 01:37:30.511113 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:30.511043    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-q6t9r"]
Jul 21 01:37:30.915318 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:37:30.915278    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0937c807_6c37_4807_8499_e71c8f8ceb02.slice/crio-d22983c66939810ac2c43f85a8d76372ccc2aa8e25853ab9dc8f5d289f6bd7bd WatchSource:0}: Error finding container d22983c66939810ac2c43f85a8d76372ccc2aa8e25853ab9dc8f5d289f6bd7bd: Status 404 returned error can't find the container with id d22983c66939810ac2c43f85a8d76372ccc2aa8e25853ab9dc8f5d289f6bd7bd
Jul 21 01:37:30.918578 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:37:30.918547    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fb11a5e_fad0_4c5e_a935_87bf77622cd2.slice/crio-13d89589e575f763b224aadb0839cefd04d42be3a0f8325b213982daddce11db WatchSource:0}: Error finding container 13d89589e575f763b224aadb0839cefd04d42be3a0f8325b213982daddce11db: Status 404 returned error can't find the container with id 13d89589e575f763b224aadb0839cefd04d42be3a0f8325b213982daddce11db
Jul 21 01:37:31.087933 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:31.087862    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79db7cc8cd-f6q5z"]
Jul 21 01:37:31.092246 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:37:31.092022    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf034e8e1_3fdf_4257_986b_34784be403cf.slice/crio-19325f1d0bed7e5c1e6ba3f547e119b1b3349fa2f600cff92e2d7514b93d3d98 WatchSource:0}: Error finding container 19325f1d0bed7e5c1e6ba3f547e119b1b3349fa2f600cff92e2d7514b93d3d98: Status 404 returned error can't find the container with id 19325f1d0bed7e5c1e6ba3f547e119b1b3349fa2f600cff92e2d7514b93d3d98
Jul 21 01:37:31.426182 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:31.426144    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-q6t9r" event={"ID":"0937c807-6c37-4807-8499-e71c8f8ceb02","Type":"ContainerStarted","Data":"d22983c66939810ac2c43f85a8d76372ccc2aa8e25853ab9dc8f5d289f6bd7bd"}
Jul 21 01:37:31.427368 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:31.427341    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79db7cc8cd-f6q5z" event={"ID":"f034e8e1-3fdf-4257-986b-34784be403cf","Type":"ContainerStarted","Data":"19325f1d0bed7e5c1e6ba3f547e119b1b3349fa2f600cff92e2d7514b93d3d98"}
Jul 21 01:37:31.428828 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:31.428522    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75fd5554cc-88fnw" event={"ID":"4fb11a5e-fad0-4c5e-a935-87bf77622cd2","Type":"ContainerStarted","Data":"13d89589e575f763b224aadb0839cefd04d42be3a0f8325b213982daddce11db"}
Jul 21 01:37:31.819965 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:31.819923    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:35.138285 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:35.138246    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:35.238773 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:35.238729    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:35.305649 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:35.305357    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-78q2q"]
Jul 21 01:37:35.308310 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:35.308247    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-78q2q"
Jul 21 01:37:35.310459 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:35.310281    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 21 01:37:35.310459 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:35.310411    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-5j5m5\""
Jul 21 01:37:35.316336 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:35.316286    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-78q2q"]
Jul 21 01:37:35.324054 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:35.324026    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:35.437133 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:35.437042    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ce2e1480-0f6e-4d72-8f0e-ccb153d25c33-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-78q2q\" (UID: \"ce2e1480-0f6e-4d72-8f0e-ccb153d25c33\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-78q2q"
Jul 21 01:37:35.470293 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:35.470264    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:35.537571 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:35.537533    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ce2e1480-0f6e-4d72-8f0e-ccb153d25c33-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-78q2q\" (UID: \"ce2e1480-0f6e-4d72-8f0e-ccb153d25c33\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-78q2q"
Jul 21 01:37:35.540364 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:35.540340    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ce2e1480-0f6e-4d72-8f0e-ccb153d25c33-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-78q2q\" (UID: \"ce2e1480-0f6e-4d72-8f0e-ccb153d25c33\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-78q2q"
Jul 21 01:37:35.548916 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:35.548890    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:35.621093 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:35.621024    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-78q2q"
Jul 21 01:37:35.648859 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:35.648831    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:35.699947 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:35.699881    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:36.048542 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:36.048462    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:36.104129 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:36.104047    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:39.207518 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:39.207472    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-78q2q"]
Jul 21 01:37:39.460629 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:39.460544    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2v6dg" event={"ID":"7971e50e-1094-40f1-a8e4-56eb51aa83d4","Type":"ContainerStarted","Data":"c8df62b9b6841607a38af38ca9fbba324d86a53606d268b997e84722a44f6213"}
Jul 21 01:37:39.463487 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:39.463410    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xbk6c" event={"ID":"26238877-47e2-4ed6-8df9-c21f38f7578c","Type":"ContainerStarted","Data":"3d664dc50aabf9f8e74fe8eb06a47b15fa3c664fd37f3854bc440b5a09d81aff"}
Jul 21 01:37:39.463862 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:39.463699    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xbk6c"
Jul 21 01:37:39.465289 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:39.465257    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pmhdx" event={"ID":"5ee75f94-10ca-4a6a-a2f4-077eb597a94f","Type":"ContainerStarted","Data":"0c3b0feaafbc8d07890743827d1ba38a4d6aa9f56cb53de653670443bb4a8e5a"}
Jul 21 01:37:39.468208 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:39.468122    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-twkvj" event={"ID":"06885d56-7129-4e19-9911-5720b9668058","Type":"ContainerStarted","Data":"8f4ca28fdc16bc945226b2cba7b02e3e3ad9a7aee99da9e05c0d27832c1a51c0"}
Jul 21 01:37:39.468208 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:39.468163    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-twkvj" event={"ID":"06885d56-7129-4e19-9911-5720b9668058","Type":"ContainerStarted","Data":"aa230701f8ee09aae0d779c8d063d888493749f24f84ccbbda13335558cc492b"}
Jul 21 01:37:39.468208 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:39.468185    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-twkvj"
Jul 21 01:37:39.469990 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:39.469935    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79db7cc8cd-f6q5z" event={"ID":"f034e8e1-3fdf-4257-986b-34784be403cf","Type":"ContainerStarted","Data":"7e6f8b5051717e4056185819d1a4434fa93a7f034ae471d565434ee9d1b07ae1"}
Jul 21 01:37:39.469990 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:39.469976    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-79db7cc8cd-f6q5z"
Jul 21 01:37:39.471910 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:39.471643    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75fd5554cc-88fnw" event={"ID":"4fb11a5e-fad0-4c5e-a935-87bf77622cd2","Type":"ContainerStarted","Data":"b10f2609dfbaff9f45ef8a86dbf2ff371c545797f467143c44f8b70bcdce089a"}
Jul 21 01:37:39.471910 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:39.471779    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75fd5554cc-88fnw"
Jul 21 01:37:39.473176 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:39.473159    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75fd5554cc-88fnw"
Jul 21 01:37:39.473865 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:39.473843    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-78q2q" event={"ID":"ce2e1480-0f6e-4d72-8f0e-ccb153d25c33","Type":"ContainerStarted","Data":"dddf350c1b6be1e899e53496d1ba18c3ab07d7f074147e26f38e041195d09545"}
Jul 21 01:37:39.475873 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:39.475843    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-245g5" event={"ID":"12e288ad-0d62-4e00-9a2a-1de85626abb9","Type":"ContainerStarted","Data":"f9bfc89819d8cede2162e8ec346b95a73e8e95eb02d776546ca0648833882b0a"}
Jul 21 01:37:39.475971 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:39.475884    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-245g5" event={"ID":"12e288ad-0d62-4e00-9a2a-1de85626abb9","Type":"ContainerStarted","Data":"3c0eab33b3f5e296a8b4e1048886db372973a7f734b9ad9ee6813875b21c2e9b"}
Jul 21 01:37:39.480056 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:39.480020    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2v6dg" podStartSLOduration=17.400817 podStartE2EDuration="35.480007231s" totalImagesPullingTime="18.079190231s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 01:37:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:37:39.479067021 +0000 UTC m=+79.076707411" watchObservedRunningTime="2026-07-21 01:37:39.480007231 +0000 UTC m=+79.077647574"
Jul 21 01:37:39.500056 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:39.500006    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-79db7cc8cd-f6q5z" podStartSLOduration=9.49998966 podStartE2EDuration="9.49998966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:37:39.499513492 +0000 UTC m=+79.097153864" watchObservedRunningTime="2026-07-21 01:37:39.49998966 +0000 UTC m=+79.097630006"
Jul 21 01:37:39.513311 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:39.513266    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pmhdx" podStartSLOduration=66.442877705 podStartE2EDuration="1m18.513247202s" totalImagesPullingTime="12.070369497s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 01:36:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:37:39.512988994 +0000 UTC m=+79.110629345" watchObservedRunningTime="2026-07-21 01:37:39.513247202 +0000 UTC m=+79.110887555"
Jul 21 01:37:39.527683 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:39.527637    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xbk6c" podStartSLOduration=68.610800656 podStartE2EDuration="1m18.527605591s" totalImagesPullingTime="9.916804935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:36:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:37:39.527351431 +0000 UTC m=+79.124991782" watchObservedRunningTime="2026-07-21 01:37:39.527605591 +0000 UTC m=+79.125245941"
Jul 21 01:37:39.543001 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:39.542931    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-245g5" podStartSLOduration=70.242658553 podStartE2EDuration="1m18.542914319s" totalImagesPullingTime="8.300255766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:36:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:37:39.542651201 +0000 UTC m=+79.140291566" watchObservedRunningTime="2026-07-21 01:37:39.542914319 +0000 UTC m=+79.140554671"
Jul 21 01:37:39.563137 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:39.563063    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-twkvj" podStartSLOduration=33.707611312 podStartE2EDuration="43.563042926s" totalImagesPullingTime="9.855431614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:36:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:37:39.562172153 +0000 UTC m=+79.159812531" watchObservedRunningTime="2026-07-21 01:37:39.563042926 +0000 UTC m=+79.160683280"
Jul 21 01:37:39.578147 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:39.578093    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75fd5554cc-88fnw" podStartSLOduration=2.399791297 podStartE2EDuration="10.578074484s" totalImagesPullingTime="8.178283187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:37:39.576552741 +0000 UTC m=+79.174193091" watchObservedRunningTime="2026-07-21 01:37:39.578074484 +0000 UTC m=+79.175714841"
Jul 21 01:37:39.637888 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:39.637782    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:39.701227 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:39.701186    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:40.164164 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:40.164099    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:40.265116 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:40.265083    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:40.456991 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:40.456900    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7ccdfdb849-jjmzc"]
Jul 21 01:37:40.459396 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:40.459372    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ccdfdb849-jjmzc"
Jul 21 01:37:40.462983 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:40.462961    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 21 01:37:40.464553 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:40.464450    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 21 01:37:40.464553 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:40.464507    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-r5vll\""
Jul 21 01:37:40.464731 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:40.464713    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 21 01:37:40.464805 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:40.464788    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 21 01:37:40.464992 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:40.464974    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 21 01:37:40.471114 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:40.471090    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7ccdfdb849-jjmzc"]
Jul 21 01:37:40.581872 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:40.581845    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3de54e9e-7461-45fb-ab72-2e66eebbf253-console-oauth-config\") pod \"console-7ccdfdb849-jjmzc\" (UID: \"3de54e9e-7461-45fb-ab72-2e66eebbf253\") " pod="openshift-console/console-7ccdfdb849-jjmzc"
Jul 21 01:37:40.581990 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:40.581918    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6qpn\" (UniqueName: \"kubernetes.io/projected/3de54e9e-7461-45fb-ab72-2e66eebbf253-kube-api-access-d6qpn\") pod \"console-7ccdfdb849-jjmzc\" (UID: \"3de54e9e-7461-45fb-ab72-2e66eebbf253\") " pod="openshift-console/console-7ccdfdb849-jjmzc"
Jul 21 01:37:40.582158 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:40.582134    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3de54e9e-7461-45fb-ab72-2e66eebbf253-console-serving-cert\") pod \"console-7ccdfdb849-jjmzc\" (UID: \"3de54e9e-7461-45fb-ab72-2e66eebbf253\") " pod="openshift-console/console-7ccdfdb849-jjmzc"
Jul 21 01:37:40.583043 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:40.582281    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3de54e9e-7461-45fb-ab72-2e66eebbf253-service-ca\") pod \"console-7ccdfdb849-jjmzc\" (UID: \"3de54e9e-7461-45fb-ab72-2e66eebbf253\") " pod="openshift-console/console-7ccdfdb849-jjmzc"
Jul 21 01:37:40.583043 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:40.582316    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3de54e9e-7461-45fb-ab72-2e66eebbf253-oauth-serving-cert\") pod \"console-7ccdfdb849-jjmzc\" (UID: \"3de54e9e-7461-45fb-ab72-2e66eebbf253\") " pod="openshift-console/console-7ccdfdb849-jjmzc"
Jul 21 01:37:40.583043 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:40.582347    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3de54e9e-7461-45fb-ab72-2e66eebbf253-console-config\") pod \"console-7ccdfdb849-jjmzc\" (UID: \"3de54e9e-7461-45fb-ab72-2e66eebbf253\") " pod="openshift-console/console-7ccdfdb849-jjmzc"
Jul 21 01:37:40.682774 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:40.682733    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3de54e9e-7461-45fb-ab72-2e66eebbf253-console-serving-cert\") pod \"console-7ccdfdb849-jjmzc\" (UID: \"3de54e9e-7461-45fb-ab72-2e66eebbf253\") " pod="openshift-console/console-7ccdfdb849-jjmzc"
Jul 21 01:37:40.682774 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:40.682781    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3de54e9e-7461-45fb-ab72-2e66eebbf253-service-ca\") pod \"console-7ccdfdb849-jjmzc\" (UID: \"3de54e9e-7461-45fb-ab72-2e66eebbf253\") " pod="openshift-console/console-7ccdfdb849-jjmzc"
Jul 21 01:37:40.683008 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:40.682799    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3de54e9e-7461-45fb-ab72-2e66eebbf253-oauth-serving-cert\") pod \"console-7ccdfdb849-jjmzc\" (UID: \"3de54e9e-7461-45fb-ab72-2e66eebbf253\") " pod="openshift-console/console-7ccdfdb849-jjmzc"
Jul 21 01:37:40.683008 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:40.682826    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3de54e9e-7461-45fb-ab72-2e66eebbf253-console-config\") pod \"console-7ccdfdb849-jjmzc\" (UID: \"3de54e9e-7461-45fb-ab72-2e66eebbf253\") " pod="openshift-console/console-7ccdfdb849-jjmzc"
Jul 21 01:37:40.683008 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:40.682861    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3de54e9e-7461-45fb-ab72-2e66eebbf253-console-oauth-config\") pod \"console-7ccdfdb849-jjmzc\" (UID: \"3de54e9e-7461-45fb-ab72-2e66eebbf253\") " pod="openshift-console/console-7ccdfdb849-jjmzc"
Jul 21 01:37:40.683008 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:40.682892    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6qpn\" (UniqueName: \"kubernetes.io/projected/3de54e9e-7461-45fb-ab72-2e66eebbf253-kube-api-access-d6qpn\") pod \"console-7ccdfdb849-jjmzc\" (UID: \"3de54e9e-7461-45fb-ab72-2e66eebbf253\") " pod="openshift-console/console-7ccdfdb849-jjmzc"
Jul 21 01:37:40.683653 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:40.683578    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3de54e9e-7461-45fb-ab72-2e66eebbf253-service-ca\") pod \"console-7ccdfdb849-jjmzc\" (UID: \"3de54e9e-7461-45fb-ab72-2e66eebbf253\") " pod="openshift-console/console-7ccdfdb849-jjmzc"
Jul 21 01:37:40.683784 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:40.683658    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3de54e9e-7461-45fb-ab72-2e66eebbf253-oauth-serving-cert\") pod \"console-7ccdfdb849-jjmzc\" (UID: \"3de54e9e-7461-45fb-ab72-2e66eebbf253\") " pod="openshift-console/console-7ccdfdb849-jjmzc"
Jul 21 01:37:40.683784 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:40.683658    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3de54e9e-7461-45fb-ab72-2e66eebbf253-console-config\") pod \"console-7ccdfdb849-jjmzc\" (UID: \"3de54e9e-7461-45fb-ab72-2e66eebbf253\") " pod="openshift-console/console-7ccdfdb849-jjmzc"
Jul 21 01:37:40.685505 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:40.685475    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3de54e9e-7461-45fb-ab72-2e66eebbf253-console-oauth-config\") pod \"console-7ccdfdb849-jjmzc\" (UID: \"3de54e9e-7461-45fb-ab72-2e66eebbf253\") " pod="openshift-console/console-7ccdfdb849-jjmzc"
Jul 21 01:37:40.685649 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:40.685608    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3de54e9e-7461-45fb-ab72-2e66eebbf253-console-serving-cert\") pod \"console-7ccdfdb849-jjmzc\" (UID: \"3de54e9e-7461-45fb-ab72-2e66eebbf253\") " pod="openshift-console/console-7ccdfdb849-jjmzc"
Jul 21 01:37:40.691034 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:40.691007    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6qpn\" (UniqueName: \"kubernetes.io/projected/3de54e9e-7461-45fb-ab72-2e66eebbf253-kube-api-access-d6qpn\") pod \"console-7ccdfdb849-jjmzc\" (UID: \"3de54e9e-7461-45fb-ab72-2e66eebbf253\") " pod="openshift-console/console-7ccdfdb849-jjmzc"
Jul 21 01:37:40.772055 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:40.772003    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ccdfdb849-jjmzc"
Jul 21 01:37:40.931809 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:40.931779    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7ccdfdb849-jjmzc"]
Jul 21 01:37:40.934829 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:37:40.934802    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de54e9e_7461_45fb_ab72_2e66eebbf253.slice/crio-96864016f1f349a26816e042241f9c5f228b25161e440e1e9c37568b148e279e WatchSource:0}: Error finding container 96864016f1f349a26816e042241f9c5f228b25161e440e1e9c37568b148e279e: Status 404 returned error can't find the container with id 96864016f1f349a26816e042241f9c5f228b25161e440e1e9c37568b148e279e
Jul 21 01:37:41.486898 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:41.486783    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-78q2q" event={"ID":"ce2e1480-0f6e-4d72-8f0e-ccb153d25c33","Type":"ContainerStarted","Data":"62115bf415fb7ee33917da717729b3a02e572886224d3c272f7c37ec2ad6435b"}
Jul 21 01:37:41.487337 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:41.487018    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-78q2q"
Jul 21 01:37:41.488531 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:41.488429    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ccdfdb849-jjmzc" event={"ID":"3de54e9e-7461-45fb-ab72-2e66eebbf253","Type":"ContainerStarted","Data":"96864016f1f349a26816e042241f9c5f228b25161e440e1e9c37568b148e279e"}
Jul 21 01:37:41.492731 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:41.492707    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-78q2q"
Jul 21 01:37:41.501125 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:41.501079    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-78q2q" podStartSLOduration=5.197648068 podStartE2EDuration="6.501063049s" totalImagesPullingTime="1.303414981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:37:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:37:41.500552032 +0000 UTC m=+81.098192383" watchObservedRunningTime="2026-07-21 01:37:41.501063049 +0000 UTC m=+81.098703401"
Jul 21 01:37:42.338946 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:42.338530    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:42.365710 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:42.365309    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-rgls6"]
Jul 21 01:37:42.367739 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:42.367714    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-rgls6"
Jul 21 01:37:42.371061 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:42.370689    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 21 01:37:42.371061 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:42.370915    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 21 01:37:42.371437 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:42.371416    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 21 01:37:42.371669 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:42.371653    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 21 01:37:42.371842 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:42.371828    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-w9xj4\""
Jul 21 01:37:42.372142 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:42.372121    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 21 01:37:42.382291 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:42.382236    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-rgls6"]
Jul 21 01:37:42.403069 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:42.403042    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:42.496240 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:42.496200    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/941d07b2-62c9-43e6-9a57-a50b65d85283-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-rgls6\" (UID: \"941d07b2-62c9-43e6-9a57-a50b65d85283\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rgls6"
Jul 21 01:37:42.496657 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:42.496245    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/941d07b2-62c9-43e6-9a57-a50b65d85283-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-rgls6\" (UID: \"941d07b2-62c9-43e6-9a57-a50b65d85283\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rgls6"
Jul 21 01:37:42.496657 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:42.496280    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/941d07b2-62c9-43e6-9a57-a50b65d85283-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-rgls6\" (UID: \"941d07b2-62c9-43e6-9a57-a50b65d85283\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rgls6"
Jul 21 01:37:42.496657 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:42.496323    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j69ch\" (UniqueName: \"kubernetes.io/projected/941d07b2-62c9-43e6-9a57-a50b65d85283-kube-api-access-j69ch\") pod \"prometheus-operator-5945b5cfd-rgls6\" (UID: \"941d07b2-62c9-43e6-9a57-a50b65d85283\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rgls6"
Jul 21 01:37:42.597698 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:42.597608    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/941d07b2-62c9-43e6-9a57-a50b65d85283-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-rgls6\" (UID: \"941d07b2-62c9-43e6-9a57-a50b65d85283\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rgls6"
Jul 21 01:37:42.597844 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:42.597703    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j69ch\" (UniqueName: \"kubernetes.io/projected/941d07b2-62c9-43e6-9a57-a50b65d85283-kube-api-access-j69ch\") pod \"prometheus-operator-5945b5cfd-rgls6\" (UID: \"941d07b2-62c9-43e6-9a57-a50b65d85283\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rgls6"
Jul 21 01:37:42.597844 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:42.597752    2575 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 21 01:37:42.597844 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:42.597791    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/941d07b2-62c9-43e6-9a57-a50b65d85283-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-rgls6\" (UID: \"941d07b2-62c9-43e6-9a57-a50b65d85283\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rgls6"
Jul 21 01:37:42.597844 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:42.597828    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/941d07b2-62c9-43e6-9a57-a50b65d85283-prometheus-operator-tls podName:941d07b2-62c9-43e6-9a57-a50b65d85283 nodeName:}" failed. No retries permitted until 2026-07-21 01:37:43.097805968 +0000 UTC m=+82.695446317 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/941d07b2-62c9-43e6-9a57-a50b65d85283-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-rgls6" (UID: "941d07b2-62c9-43e6-9a57-a50b65d85283") : secret "prometheus-operator-tls" not found
Jul 21 01:37:42.597994 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:42.597854    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/941d07b2-62c9-43e6-9a57-a50b65d85283-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-rgls6\" (UID: \"941d07b2-62c9-43e6-9a57-a50b65d85283\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rgls6"
Jul 21 01:37:42.598495 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:42.598473    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/941d07b2-62c9-43e6-9a57-a50b65d85283-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-rgls6\" (UID: \"941d07b2-62c9-43e6-9a57-a50b65d85283\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rgls6"
Jul 21 01:37:42.600390 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:42.600351    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/941d07b2-62c9-43e6-9a57-a50b65d85283-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-rgls6\" (UID: \"941d07b2-62c9-43e6-9a57-a50b65d85283\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rgls6"
Jul 21 01:37:42.612969 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:42.612942    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j69ch\" (UniqueName: \"kubernetes.io/projected/941d07b2-62c9-43e6-9a57-a50b65d85283-kube-api-access-j69ch\") pod \"prometheus-operator-5945b5cfd-rgls6\" (UID: \"941d07b2-62c9-43e6-9a57-a50b65d85283\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rgls6"
Jul 21 01:37:43.102088 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:43.102055    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/941d07b2-62c9-43e6-9a57-a50b65d85283-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-rgls6\" (UID: \"941d07b2-62c9-43e6-9a57-a50b65d85283\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rgls6"
Jul 21 01:37:43.102263 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:43.102222    2575 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 21 01:37:43.102327 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:43.102302    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/941d07b2-62c9-43e6-9a57-a50b65d85283-prometheus-operator-tls podName:941d07b2-62c9-43e6-9a57-a50b65d85283 nodeName:}" failed. No retries permitted until 2026-07-21 01:37:44.102278094 +0000 UTC m=+83.699918426 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/941d07b2-62c9-43e6-9a57-a50b65d85283-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-rgls6" (UID: "941d07b2-62c9-43e6-9a57-a50b65d85283") : secret "prometheus-operator-tls" not found
Jul 21 01:37:43.444850 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:43.444762    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:44.109746 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:44.109711    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/941d07b2-62c9-43e6-9a57-a50b65d85283-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-rgls6\" (UID: \"941d07b2-62c9-43e6-9a57-a50b65d85283\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rgls6"
Jul 21 01:37:44.112326 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:44.112303    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/941d07b2-62c9-43e6-9a57-a50b65d85283-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-rgls6\" (UID: \"941d07b2-62c9-43e6-9a57-a50b65d85283\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-rgls6"
Jul 21 01:37:44.180319 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:44.180282    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-rgls6"
Jul 21 01:37:46.214296 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:46.214256    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:46.274406 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:46.274368    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:49.485319 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:49.485293    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-twkvj"
Jul 21 01:37:49.556601 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:49.556567    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:49.618372 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:49.618342    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:50.210241 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:50.210202    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:50.267006 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:50.266969    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:51.056572 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:51.056537    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:51.111858 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:51.111822    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:51.436065 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:51.436044    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:37:51.481546 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:51.481520    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-rgls6"]
Jul 21 01:37:51.484345 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:37:51.484317    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod941d07b2_62c9_43e6_9a57_a50b65d85283.slice/crio-8e78274b735ab58778088a667812d3276956e198a1ef4444d3a1ef425fcbff03 WatchSource:0}: Error finding container 8e78274b735ab58778088a667812d3276956e198a1ef4444d3a1ef425fcbff03: Status 404 returned error can't find the container with id 8e78274b735ab58778088a667812d3276956e198a1ef4444d3a1ef425fcbff03
Jul 21 01:37:51.534739 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:51.534691    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ccdfdb849-jjmzc" event={"ID":"3de54e9e-7461-45fb-ab72-2e66eebbf253","Type":"ContainerStarted","Data":"6d330c125817f7dc075da5080e1041dd3f55b5c2df9d9c484c9f36d5ffdfc65a"}
Jul 21 01:37:51.536285 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:51.536258    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-q6t9r" event={"ID":"0937c807-6c37-4807-8499-e71c8f8ceb02","Type":"ContainerStarted","Data":"c59dd0d7b6b001b7b27c7f62668ed9964f0c658606e2d6ed7d31e562bdf5c1e1"}
Jul 21 01:37:51.536434 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:51.536395    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-q6t9r"
Jul 21 01:37:51.537526 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:51.537500    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-rgls6" event={"ID":"941d07b2-62c9-43e6-9a57-a50b65d85283","Type":"ContainerStarted","Data":"8e78274b735ab58778088a667812d3276956e198a1ef4444d3a1ef425fcbff03"}
Jul 21 01:37:51.538242 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:51.538216    2575 patch_prober.go:28] interesting pod/downloads-5bc67dcd46-q6t9r container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.12:8080/\": dial tcp 10.133.0.12:8080: connect: connection refused" start-of-body=
Jul 21 01:37:51.538338 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:51.538271    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-5bc67dcd46-q6t9r" podUID="0937c807-6c37-4807-8499-e71c8f8ceb02" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.12:8080/\": dial tcp 10.133.0.12:8080: connect: connection refused"
Jul 21 01:37:51.553185 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:51.553130    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7ccdfdb849-jjmzc" podStartSLOduration=1.129021376 podStartE2EDuration="11.553115496s" totalImagesPullingTime="10.42409412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:37:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:37:51.551487475 +0000 UTC m=+91.149127839" watchObservedRunningTime="2026-07-21 01:37:51.553115496 +0000 UTC m=+91.150755847"
Jul 21 01:37:51.566128 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:51.566076    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-q6t9r" podStartSLOduration=2.117405531 podStartE2EDuration="22.56605861s" totalImagesPullingTime="20.448653079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:37:51.56524097 +0000 UTC m=+91.162881318" watchObservedRunningTime="2026-07-21 01:37:51.56605861 +0000 UTC m=+91.163698966"
Jul 21 01:37:52.217450 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:52.217399    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:52.280295 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:52.280255    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:52.561647 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:52.561401    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-q6t9r"
Jul 21 01:37:53.813406 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:53.813353    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:54.534025 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:54.533988    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:54.553346 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:54.553313    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-rgls6" event={"ID":"941d07b2-62c9-43e6-9a57-a50b65d85283","Type":"ContainerStarted","Data":"13a4584cf66457eec14f8efa78d9a59b9d55c496e13eba8ca29f1055eec4bb80"}
Jul 21 01:37:54.553508 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:54.553371    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-rgls6" event={"ID":"941d07b2-62c9-43e6-9a57-a50b65d85283","Type":"ContainerStarted","Data":"f056f1fe1f4f4d5576cfc34ea7b732d3ebcd93d8584896bc05407408a8c5efd5"}
Jul 21 01:37:54.569940 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:54.569896    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-rgls6" podStartSLOduration=10.589202314 podStartE2EDuration="12.569881184s" totalImagesPullingTime="1.98067887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:37:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:37:54.568432685 +0000 UTC m=+94.166073037" watchObservedRunningTime="2026-07-21 01:37:54.569881184 +0000 UTC m=+94.167521534"
Jul 21 01:37:54.635126 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:54.635095    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:56.443416 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:56.443379    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:56.459747 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:56.459708    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb" podUID="122f620b-920e-4174-8658-489db0d018ad" containerName="registry" containerID="cri-o://541a2ce3dd9e30721c6d27ffaf2de20acce99addc1a4d84750c315618c7497a9" gracePeriod=30
Jul 21 01:37:56.767604 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:56.767575    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:37:56.913485 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:56.913456    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-bound-sa-token\") pod \"122f620b-920e-4174-8658-489db0d018ad\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") "
Jul 21 01:37:56.913485 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:56.913491    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6v7cg\" (UniqueName: \"kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-kube-api-access-6v7cg\") pod \"122f620b-920e-4174-8658-489db0d018ad\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") "
Jul 21 01:37:56.913745 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:56.913524    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/122f620b-920e-4174-8658-489db0d018ad-installation-pull-secrets\") pod \"122f620b-920e-4174-8658-489db0d018ad\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") "
Jul 21 01:37:56.913745 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:56.913649    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/122f620b-920e-4174-8658-489db0d018ad-trusted-ca\") pod \"122f620b-920e-4174-8658-489db0d018ad\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") "
Jul 21 01:37:56.913857 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:56.913810    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:56.913985 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:56.913966    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-registry-tls\") pod \"122f620b-920e-4174-8658-489db0d018ad\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") "
Jul 21 01:37:56.914053 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:56.914039    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/122f620b-920e-4174-8658-489db0d018ad-ca-trust-extracted\") pod \"122f620b-920e-4174-8658-489db0d018ad\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") "
Jul 21 01:37:56.914107 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:56.914069    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/122f620b-920e-4174-8658-489db0d018ad-image-registry-private-configuration\") pod \"122f620b-920e-4174-8658-489db0d018ad\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") "
Jul 21 01:37:56.914107 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:56.914101    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/122f620b-920e-4174-8658-489db0d018ad-registry-certificates\") pod \"122f620b-920e-4174-8658-489db0d018ad\" (UID: \"122f620b-920e-4174-8658-489db0d018ad\") "
Jul 21 01:37:56.914630 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:56.914579    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/122f620b-920e-4174-8658-489db0d018ad-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "122f620b-920e-4174-8658-489db0d018ad" (UID: "122f620b-920e-4174-8658-489db0d018ad"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 01:37:56.914876 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:56.914841    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/122f620b-920e-4174-8658-489db0d018ad-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "122f620b-920e-4174-8658-489db0d018ad" (UID: "122f620b-920e-4174-8658-489db0d018ad"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 01:37:56.916918 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:56.916885    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/122f620b-920e-4174-8658-489db0d018ad-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "122f620b-920e-4174-8658-489db0d018ad" (UID: "122f620b-920e-4174-8658-489db0d018ad"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 01:37:56.916918 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:56.916889    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "122f620b-920e-4174-8658-489db0d018ad" (UID: "122f620b-920e-4174-8658-489db0d018ad"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 01:37:56.917222 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:56.917190    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "122f620b-920e-4174-8658-489db0d018ad" (UID: "122f620b-920e-4174-8658-489db0d018ad"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 01:37:56.917309 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:56.917262    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-kube-api-access-6v7cg" (OuterVolumeSpecName: "kube-api-access-6v7cg") pod "122f620b-920e-4174-8658-489db0d018ad" (UID: "122f620b-920e-4174-8658-489db0d018ad"). InnerVolumeSpecName "kube-api-access-6v7cg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 01:37:56.917309 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:56.917298    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/122f620b-920e-4174-8658-489db0d018ad-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "122f620b-920e-4174-8658-489db0d018ad" (UID: "122f620b-920e-4174-8658-489db0d018ad"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 01:37:56.923592 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:56.923561    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/122f620b-920e-4174-8658-489db0d018ad-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "122f620b-920e-4174-8658-489db0d018ad" (UID: "122f620b-920e-4174-8658-489db0d018ad"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 01:37:57.015024 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:57.014940    2575 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/122f620b-920e-4174-8658-489db0d018ad-registry-certificates\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 01:37:57.015024 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:57.014976    2575 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-bound-sa-token\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 01:37:57.015024 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:57.014989    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v7cg\" (UniqueName: \"kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-kube-api-access-6v7cg\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 01:37:57.015024 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:57.015001    2575 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/122f620b-920e-4174-8658-489db0d018ad-installation-pull-secrets\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 01:37:57.015024 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:57.015013    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/122f620b-920e-4174-8658-489db0d018ad-trusted-ca\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 01:37:57.015024 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:57.015025    2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/122f620b-920e-4174-8658-489db0d018ad-registry-tls\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 01:37:57.015426 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:57.015036    2575 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/122f620b-920e-4174-8658-489db0d018ad-ca-trust-extracted\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 01:37:57.015426 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:57.015049    2575 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/122f620b-920e-4174-8658-489db0d018ad-image-registry-private-configuration\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 01:37:57.567848 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:57.567809    2575 generic.go:356] "Generic (PLEG): container finished" podID="122f620b-920e-4174-8658-489db0d018ad" containerID="541a2ce3dd9e30721c6d27ffaf2de20acce99addc1a4d84750c315618c7497a9" exitCode=0
Jul 21 01:37:57.568337 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:57.567881    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb"
Jul 21 01:37:57.568337 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:57.567901    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb" event={"ID":"122f620b-920e-4174-8658-489db0d018ad","Type":"ContainerDied","Data":"541a2ce3dd9e30721c6d27ffaf2de20acce99addc1a4d84750c315618c7497a9"}
Jul 21 01:37:57.568337 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:57.567968    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b9b98f9dc-g4qkb" event={"ID":"122f620b-920e-4174-8658-489db0d018ad","Type":"ContainerDied","Data":"4715ed7f47ad0922994b37d8aee5cdbfe2478f0df80f7afb53d91e053821c298"}
Jul 21 01:37:57.568337 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:57.568001    2575 scope.go:117] "RemoveContainer" containerID="541a2ce3dd9e30721c6d27ffaf2de20acce99addc1a4d84750c315618c7497a9"
Jul 21 01:37:57.577468 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:57.577449    2575 scope.go:117] "RemoveContainer" containerID="541a2ce3dd9e30721c6d27ffaf2de20acce99addc1a4d84750c315618c7497a9"
Jul 21 01:37:57.577789 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:37:57.577763    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"541a2ce3dd9e30721c6d27ffaf2de20acce99addc1a4d84750c315618c7497a9\": container with ID starting with 541a2ce3dd9e30721c6d27ffaf2de20acce99addc1a4d84750c315618c7497a9 not found: ID does not exist" containerID="541a2ce3dd9e30721c6d27ffaf2de20acce99addc1a4d84750c315618c7497a9"
Jul 21 01:37:57.577873 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:57.577795    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"541a2ce3dd9e30721c6d27ffaf2de20acce99addc1a4d84750c315618c7497a9"} err="failed to get container status \"541a2ce3dd9e30721c6d27ffaf2de20acce99addc1a4d84750c315618c7497a9\": rpc error: code = NotFound desc = could not find container \"541a2ce3dd9e30721c6d27ffaf2de20acce99addc1a4d84750c315618c7497a9\": container with ID starting with 541a2ce3dd9e30721c6d27ffaf2de20acce99addc1a4d84750c315618c7497a9 not found: ID does not exist"
Jul 21 01:37:57.584932 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:57.584910    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b9b98f9dc-g4qkb"]
Jul 21 01:37:57.589522 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:57.589502    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-b9b98f9dc-g4qkb"]
Jul 21 01:37:58.138941 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:58.138901    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:58.239449 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:58.239420    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:59.025677 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:59.025648    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="122f620b-920e-4174-8658-489db0d018ad" path="/var/lib/kubelet/pods/122f620b-920e-4174-8658-489db0d018ad/volumes"
Jul 21 01:37:59.211967 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:59.211932    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:37:59.259260 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:37:59.259226    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:00.486671 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:00.486639    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-79db7cc8cd-f6q5z"
Jul 21 01:38:00.618739 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:00.618702    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:00.682751 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:00.682719    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:00.745121 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:00.745043    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:00.772290 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:00.772261    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7ccdfdb849-jjmzc"
Jul 21 01:38:00.772452 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:00.772327    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7ccdfdb849-jjmzc"
Jul 21 01:38:00.778108 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:00.778082    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7ccdfdb849-jjmzc"
Jul 21 01:38:00.804744 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:00.804682    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:01.059154 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.059126    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:01.088596 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.088564    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-gtp7f"]
Jul 21 01:38:01.088867 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.088855    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="122f620b-920e-4174-8658-489db0d018ad" containerName="registry"
Jul 21 01:38:01.088913 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.088870    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="122f620b-920e-4174-8658-489db0d018ad" containerName="registry"
Jul 21 01:38:01.088950 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.088925    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="122f620b-920e-4174-8658-489db0d018ad" containerName="registry"
Jul 21 01:38:01.132555 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.132521    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-gtp7f"]
Jul 21 01:38:01.132750 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.132672    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gtp7f"
Jul 21 01:38:01.135491 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.135458    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 21 01:38:01.135491 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.135463    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 21 01:38:01.135770 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.135754    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-7444q\""
Jul 21 01:38:01.151174 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.151136    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:01.219491 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.219447    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:01.229910 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.229880    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5g8kh"]
Jul 21 01:38:01.247398 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.247373    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5dec2ddc-a63d-479d-8fa7-f422b0ed0579-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-gtp7f\" (UID: \"5dec2ddc-a63d-479d-8fa7-f422b0ed0579\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gtp7f"
Jul 21 01:38:01.247555 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.247409    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tljm\" (UniqueName: \"kubernetes.io/projected/5dec2ddc-a63d-479d-8fa7-f422b0ed0579-kube-api-access-9tljm\") pod \"openshift-state-metrics-79f6958cdf-gtp7f\" (UID: \"5dec2ddc-a63d-479d-8fa7-f422b0ed0579\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gtp7f"
Jul 21 01:38:01.247555 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.247449    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5dec2ddc-a63d-479d-8fa7-f422b0ed0579-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-gtp7f\" (UID: \"5dec2ddc-a63d-479d-8fa7-f422b0ed0579\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gtp7f"
Jul 21 01:38:01.247555 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.247477    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5dec2ddc-a63d-479d-8fa7-f422b0ed0579-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-gtp7f\" (UID: \"5dec2ddc-a63d-479d-8fa7-f422b0ed0579\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gtp7f"
Jul 21 01:38:01.253270 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.253249    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5g8kh"
Jul 21 01:38:01.255496 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.255454    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 21 01:38:01.255621 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.255590    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 21 01:38:01.255773 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.255755    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wpz6w\""
Jul 21 01:38:01.255920 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.255760    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 21 01:38:01.334858 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.334773    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:01.348135 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.348105    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5dec2ddc-a63d-479d-8fa7-f422b0ed0579-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-gtp7f\" (UID: \"5dec2ddc-a63d-479d-8fa7-f422b0ed0579\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gtp7f"
Jul 21 01:38:01.348307 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.348147    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5dec2ddc-a63d-479d-8fa7-f422b0ed0579-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-gtp7f\" (UID: \"5dec2ddc-a63d-479d-8fa7-f422b0ed0579\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gtp7f"
Jul 21 01:38:01.348307 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.348192    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5dec2ddc-a63d-479d-8fa7-f422b0ed0579-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-gtp7f\" (UID: \"5dec2ddc-a63d-479d-8fa7-f422b0ed0579\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gtp7f"
Jul 21 01:38:01.348307 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.348230    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tljm\" (UniqueName: \"kubernetes.io/projected/5dec2ddc-a63d-479d-8fa7-f422b0ed0579-kube-api-access-9tljm\") pod \"openshift-state-metrics-79f6958cdf-gtp7f\" (UID: \"5dec2ddc-a63d-479d-8fa7-f422b0ed0579\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gtp7f"
Jul 21 01:38:01.348307 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:38:01.348305    2575 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 21 01:38:01.348519 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:38:01.348364    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dec2ddc-a63d-479d-8fa7-f422b0ed0579-openshift-state-metrics-tls podName:5dec2ddc-a63d-479d-8fa7-f422b0ed0579 nodeName:}" failed. No retries permitted until 2026-07-21 01:38:01.84834532 +0000 UTC m=+101.445985648 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/5dec2ddc-a63d-479d-8fa7-f422b0ed0579-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-gtp7f" (UID: "5dec2ddc-a63d-479d-8fa7-f422b0ed0579") : secret "openshift-state-metrics-tls" not found
Jul 21 01:38:01.349180 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.349159    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5dec2ddc-a63d-479d-8fa7-f422b0ed0579-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-gtp7f\" (UID: \"5dec2ddc-a63d-479d-8fa7-f422b0ed0579\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gtp7f"
Jul 21 01:38:01.351241 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.351212    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5dec2ddc-a63d-479d-8fa7-f422b0ed0579-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-gtp7f\" (UID: \"5dec2ddc-a63d-479d-8fa7-f422b0ed0579\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gtp7f"
Jul 21 01:38:01.359314 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.359289    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tljm\" (UniqueName: \"kubernetes.io/projected/5dec2ddc-a63d-479d-8fa7-f422b0ed0579-kube-api-access-9tljm\") pod \"openshift-state-metrics-79f6958cdf-gtp7f\" (UID: \"5dec2ddc-a63d-479d-8fa7-f422b0ed0579\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gtp7f"
Jul 21 01:38:01.448926 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.448885    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/734ab4c2-b74d-48ec-9b98-a0a42eaf797e-node-exporter-wtmp\") pod \"node-exporter-5g8kh\" (UID: \"734ab4c2-b74d-48ec-9b98-a0a42eaf797e\") " pod="openshift-monitoring/node-exporter-5g8kh"
Jul 21 01:38:01.448926 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.448934    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/734ab4c2-b74d-48ec-9b98-a0a42eaf797e-root\") pod \"node-exporter-5g8kh\" (UID: \"734ab4c2-b74d-48ec-9b98-a0a42eaf797e\") " pod="openshift-monitoring/node-exporter-5g8kh"
Jul 21 01:38:01.449171 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.448957    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/734ab4c2-b74d-48ec-9b98-a0a42eaf797e-metrics-client-ca\") pod \"node-exporter-5g8kh\" (UID: \"734ab4c2-b74d-48ec-9b98-a0a42eaf797e\") " pod="openshift-monitoring/node-exporter-5g8kh"
Jul 21 01:38:01.449171 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.449015    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/734ab4c2-b74d-48ec-9b98-a0a42eaf797e-node-exporter-accelerators-collector-config\") pod \"node-exporter-5g8kh\" (UID: \"734ab4c2-b74d-48ec-9b98-a0a42eaf797e\") " pod="openshift-monitoring/node-exporter-5g8kh"
Jul 21 01:38:01.449171 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.449080    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvhzj\" (UniqueName: \"kubernetes.io/projected/734ab4c2-b74d-48ec-9b98-a0a42eaf797e-kube-api-access-gvhzj\") pod \"node-exporter-5g8kh\" (UID: \"734ab4c2-b74d-48ec-9b98-a0a42eaf797e\") " pod="openshift-monitoring/node-exporter-5g8kh"
Jul 21 01:38:01.449171 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.449130    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/734ab4c2-b74d-48ec-9b98-a0a42eaf797e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5g8kh\" (UID: \"734ab4c2-b74d-48ec-9b98-a0a42eaf797e\") " pod="openshift-monitoring/node-exporter-5g8kh"
Jul 21 01:38:01.449171 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.449170    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/734ab4c2-b74d-48ec-9b98-a0a42eaf797e-sys\") pod \"node-exporter-5g8kh\" (UID: \"734ab4c2-b74d-48ec-9b98-a0a42eaf797e\") " pod="openshift-monitoring/node-exporter-5g8kh"
Jul 21 01:38:01.449397 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.449196    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/734ab4c2-b74d-48ec-9b98-a0a42eaf797e-node-exporter-textfile\") pod \"node-exporter-5g8kh\" (UID: \"734ab4c2-b74d-48ec-9b98-a0a42eaf797e\") " pod="openshift-monitoring/node-exporter-5g8kh"
Jul 21 01:38:01.449397 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.449220    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/734ab4c2-b74d-48ec-9b98-a0a42eaf797e-node-exporter-tls\") pod \"node-exporter-5g8kh\" (UID: \"734ab4c2-b74d-48ec-9b98-a0a42eaf797e\") " pod="openshift-monitoring/node-exporter-5g8kh"
Jul 21 01:38:01.449782 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.449761    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:01.534671 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.534634    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:01.549726 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.549698    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvhzj\" (UniqueName: \"kubernetes.io/projected/734ab4c2-b74d-48ec-9b98-a0a42eaf797e-kube-api-access-gvhzj\") pod \"node-exporter-5g8kh\" (UID: \"734ab4c2-b74d-48ec-9b98-a0a42eaf797e\") " pod="openshift-monitoring/node-exporter-5g8kh"
Jul 21 01:38:01.549893 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.549741    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/734ab4c2-b74d-48ec-9b98-a0a42eaf797e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5g8kh\" (UID: \"734ab4c2-b74d-48ec-9b98-a0a42eaf797e\") " pod="openshift-monitoring/node-exporter-5g8kh"
Jul 21 01:38:01.549893 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.549767    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/734ab4c2-b74d-48ec-9b98-a0a42eaf797e-sys\") pod \"node-exporter-5g8kh\" (UID: \"734ab4c2-b74d-48ec-9b98-a0a42eaf797e\") " pod="openshift-monitoring/node-exporter-5g8kh"
Jul 21 01:38:01.550000 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.549904    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/734ab4c2-b74d-48ec-9b98-a0a42eaf797e-node-exporter-textfile\") pod \"node-exporter-5g8kh\" (UID: \"734ab4c2-b74d-48ec-9b98-a0a42eaf797e\") " pod="openshift-monitoring/node-exporter-5g8kh"
Jul 21 01:38:01.550000 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.549979    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/734ab4c2-b74d-48ec-9b98-a0a42eaf797e-sys\") pod \"node-exporter-5g8kh\" (UID: \"734ab4c2-b74d-48ec-9b98-a0a42eaf797e\") " pod="openshift-monitoring/node-exporter-5g8kh"
Jul 21 01:38:01.550210 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:38:01.550190    2575 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 21 01:38:01.550275 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.550252    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/734ab4c2-b74d-48ec-9b98-a0a42eaf797e-node-exporter-textfile\") pod \"node-exporter-5g8kh\" (UID: \"734ab4c2-b74d-48ec-9b98-a0a42eaf797e\") " pod="openshift-monitoring/node-exporter-5g8kh"
Jul 21 01:38:01.550275 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:38:01.550272    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/734ab4c2-b74d-48ec-9b98-a0a42eaf797e-node-exporter-tls podName:734ab4c2-b74d-48ec-9b98-a0a42eaf797e nodeName:}" failed. No retries permitted until 2026-07-21 01:38:02.050251392 +0000 UTC m=+101.647891728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/734ab4c2-b74d-48ec-9b98-a0a42eaf797e-node-exporter-tls") pod "node-exporter-5g8kh" (UID: "734ab4c2-b74d-48ec-9b98-a0a42eaf797e") : secret "node-exporter-tls" not found
Jul 21 01:38:01.550385 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.550268    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/734ab4c2-b74d-48ec-9b98-a0a42eaf797e-node-exporter-tls\") pod \"node-exporter-5g8kh\" (UID: \"734ab4c2-b74d-48ec-9b98-a0a42eaf797e\") " pod="openshift-monitoring/node-exporter-5g8kh"
Jul 21 01:38:01.550385 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.550348    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/734ab4c2-b74d-48ec-9b98-a0a42eaf797e-node-exporter-wtmp\") pod \"node-exporter-5g8kh\" (UID: \"734ab4c2-b74d-48ec-9b98-a0a42eaf797e\") " pod="openshift-monitoring/node-exporter-5g8kh"
Jul 21 01:38:01.550541 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.550394    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/734ab4c2-b74d-48ec-9b98-a0a42eaf797e-root\") pod \"node-exporter-5g8kh\" (UID: \"734ab4c2-b74d-48ec-9b98-a0a42eaf797e\") " pod="openshift-monitoring/node-exporter-5g8kh"
Jul 21 01:38:01.550541 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.550417    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/734ab4c2-b74d-48ec-9b98-a0a42eaf797e-metrics-client-ca\") pod \"node-exporter-5g8kh\" (UID: \"734ab4c2-b74d-48ec-9b98-a0a42eaf797e\") " pod="openshift-monitoring/node-exporter-5g8kh"
Jul 21 01:38:01.550541 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.550446    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/734ab4c2-b74d-48ec-9b98-a0a42eaf797e-node-exporter-accelerators-collector-config\") pod \"node-exporter-5g8kh\" (UID: \"734ab4c2-b74d-48ec-9b98-a0a42eaf797e\") " pod="openshift-monitoring/node-exporter-5g8kh"
Jul 21 01:38:01.550541 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.550507    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/734ab4c2-b74d-48ec-9b98-a0a42eaf797e-root\") pod \"node-exporter-5g8kh\" (UID: \"734ab4c2-b74d-48ec-9b98-a0a42eaf797e\") " pod="openshift-monitoring/node-exporter-5g8kh"
Jul 21 01:38:01.550759 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.550607    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/734ab4c2-b74d-48ec-9b98-a0a42eaf797e-node-exporter-wtmp\") pod \"node-exporter-5g8kh\" (UID: \"734ab4c2-b74d-48ec-9b98-a0a42eaf797e\") " pod="openshift-monitoring/node-exporter-5g8kh"
Jul 21 01:38:01.551177 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.551024    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/734ab4c2-b74d-48ec-9b98-a0a42eaf797e-node-exporter-accelerators-collector-config\") pod \"node-exporter-5g8kh\" (UID: \"734ab4c2-b74d-48ec-9b98-a0a42eaf797e\") " pod="openshift-monitoring/node-exporter-5g8kh"
Jul 21 01:38:01.551177 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.551051    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/734ab4c2-b74d-48ec-9b98-a0a42eaf797e-metrics-client-ca\") pod \"node-exporter-5g8kh\" (UID: \"734ab4c2-b74d-48ec-9b98-a0a42eaf797e\") " pod="openshift-monitoring/node-exporter-5g8kh"
Jul 21 01:38:01.552520 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.552497    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/734ab4c2-b74d-48ec-9b98-a0a42eaf797e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5g8kh\" (UID: \"734ab4c2-b74d-48ec-9b98-a0a42eaf797e\") " pod="openshift-monitoring/node-exporter-5g8kh"
Jul 21 01:38:01.571691 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.571667    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvhzj\" (UniqueName: \"kubernetes.io/projected/734ab4c2-b74d-48ec-9b98-a0a42eaf797e-kube-api-access-gvhzj\") pod \"node-exporter-5g8kh\" (UID: \"734ab4c2-b74d-48ec-9b98-a0a42eaf797e\") " pod="openshift-monitoring/node-exporter-5g8kh"
Jul 21 01:38:01.585315 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.585242    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7ccdfdb849-jjmzc"]
Jul 21 01:38:01.587634 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.587601    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7ccdfdb849-jjmzc"
Jul 21 01:38:01.628166 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.628140    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:01.703072 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.703044    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:01.801835 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.801807    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:01.852847 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.852770    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5dec2ddc-a63d-479d-8fa7-f422b0ed0579-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-gtp7f\" (UID: \"5dec2ddc-a63d-479d-8fa7-f422b0ed0579\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gtp7f"
Jul 21 01:38:01.855315 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.855291    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5dec2ddc-a63d-479d-8fa7-f422b0ed0579-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-gtp7f\" (UID: \"5dec2ddc-a63d-479d-8fa7-f422b0ed0579\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gtp7f"
Jul 21 01:38:01.903627 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:01.903583    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:02.043719 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.043688    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gtp7f"
Jul 21 01:38:02.054359 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.054332    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/734ab4c2-b74d-48ec-9b98-a0a42eaf797e-node-exporter-tls\") pod \"node-exporter-5g8kh\" (UID: \"734ab4c2-b74d-48ec-9b98-a0a42eaf797e\") " pod="openshift-monitoring/node-exporter-5g8kh"
Jul 21 01:38:02.054507 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:38:02.054478    2575 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 21 01:38:02.054581 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:38:02.054559    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/734ab4c2-b74d-48ec-9b98-a0a42eaf797e-node-exporter-tls podName:734ab4c2-b74d-48ec-9b98-a0a42eaf797e nodeName:}" failed. No retries permitted until 2026-07-21 01:38:03.054540381 +0000 UTC m=+102.652180712 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/734ab4c2-b74d-48ec-9b98-a0a42eaf797e-node-exporter-tls") pod "node-exporter-5g8kh" (UID: "734ab4c2-b74d-48ec-9b98-a0a42eaf797e") : secret "node-exporter-tls" not found
Jul 21 01:38:02.214095 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.214067    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:02.219357 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.219338    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-gtp7f"]
Jul 21 01:38:02.327835 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.327791    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:02.348181 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.348137    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 21 01:38:02.368594 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.368570    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.371285 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.371264    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-x7kh9\""
Jul 21 01:38:02.371506 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.371492    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 21 01:38:02.371557 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.371532    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 21 01:38:02.371776 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.371759    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 21 01:38:02.371935 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.371918    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 21 01:38:02.372074 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.372060    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 21 01:38:02.372357 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.372343    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 21 01:38:02.372434 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.372346    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 21 01:38:02.375673 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.375210    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 21 01:38:02.375673 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.375570    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 21 01:38:02.377247 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.377229    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 21 01:38:02.430360 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.430338    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:02.456563 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.456529    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1e261108-17de-48b2-8051-544ca6a67905-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.456563 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.456565    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1e261108-17de-48b2-8051-544ca6a67905-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.456793 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.456586    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1e261108-17de-48b2-8051-544ca6a67905-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.456793 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.456608    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e261108-17de-48b2-8051-544ca6a67905-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.456793 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.456657    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1e261108-17de-48b2-8051-544ca6a67905-config-volume\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.456793 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.456739    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1e261108-17de-48b2-8051-544ca6a67905-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.456947 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.456832    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1e261108-17de-48b2-8051-544ca6a67905-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.456947 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.456855    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d547p\" (UniqueName: \"kubernetes.io/projected/1e261108-17de-48b2-8051-544ca6a67905-kube-api-access-d547p\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.456947 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.456883    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1e261108-17de-48b2-8051-544ca6a67905-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.456947 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.456903    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e261108-17de-48b2-8051-544ca6a67905-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.456947 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.456922    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1e261108-17de-48b2-8051-544ca6a67905-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.456947 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.456940    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1e261108-17de-48b2-8051-544ca6a67905-web-config\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.457130 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.457012    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1e261108-17de-48b2-8051-544ca6a67905-config-out\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.558190 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.558150    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1e261108-17de-48b2-8051-544ca6a67905-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.558662 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.558198    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1e261108-17de-48b2-8051-544ca6a67905-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.558662 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.558226    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e261108-17de-48b2-8051-544ca6a67905-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.558662 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.558251    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1e261108-17de-48b2-8051-544ca6a67905-config-volume\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.558662 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.558280    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1e261108-17de-48b2-8051-544ca6a67905-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.558662 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.558324    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1e261108-17de-48b2-8051-544ca6a67905-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.558662 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.558347    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d547p\" (UniqueName: \"kubernetes.io/projected/1e261108-17de-48b2-8051-544ca6a67905-kube-api-access-d547p\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.558662 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.558376    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1e261108-17de-48b2-8051-544ca6a67905-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.558662 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.558403    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e261108-17de-48b2-8051-544ca6a67905-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.558662 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.558432    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1e261108-17de-48b2-8051-544ca6a67905-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.558662 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.558456    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1e261108-17de-48b2-8051-544ca6a67905-web-config\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.558662 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.558509    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1e261108-17de-48b2-8051-544ca6a67905-config-out\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.558662 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.558546    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1e261108-17de-48b2-8051-544ca6a67905-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.558662 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.558586    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1e261108-17de-48b2-8051-544ca6a67905-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.559381 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:38:02.558707    2575 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 21 01:38:02.559381 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:38:02.558772    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e261108-17de-48b2-8051-544ca6a67905-secret-alertmanager-main-tls podName:1e261108-17de-48b2-8051-544ca6a67905 nodeName:}" failed. No retries permitted until 2026-07-21 01:38:03.058750079 +0000 UTC m=+102.656390412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/1e261108-17de-48b2-8051-544ca6a67905-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "1e261108-17de-48b2-8051-544ca6a67905") : secret "alertmanager-main-tls" not found
Jul 21 01:38:02.560289 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.560238    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e261108-17de-48b2-8051-544ca6a67905-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.561007 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.560988    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e261108-17de-48b2-8051-544ca6a67905-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.561358 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.561333    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1e261108-17de-48b2-8051-544ca6a67905-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.561882 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.561836    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1e261108-17de-48b2-8051-544ca6a67905-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.561882 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.561845    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1e261108-17de-48b2-8051-544ca6a67905-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.562030 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.561884    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1e261108-17de-48b2-8051-544ca6a67905-config-volume\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.562030 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.561992    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1e261108-17de-48b2-8051-544ca6a67905-config-out\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.562528 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.562502    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1e261108-17de-48b2-8051-544ca6a67905-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.564885 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.563991    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1e261108-17de-48b2-8051-544ca6a67905-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.564885 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.564404    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1e261108-17de-48b2-8051-544ca6a67905-web-config\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.572341 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.572319    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d547p\" (UniqueName: \"kubernetes.io/projected/1e261108-17de-48b2-8051-544ca6a67905-kube-api-access-d547p\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:02.588667 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.588637    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gtp7f" event={"ID":"5dec2ddc-a63d-479d-8fa7-f422b0ed0579","Type":"ContainerStarted","Data":"781a724c3b1eb6e4e381f020331b869ac5e4d11107181c713053b86e609b666e"}
Jul 21 01:38:02.588775 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.588710    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gtp7f" event={"ID":"5dec2ddc-a63d-479d-8fa7-f422b0ed0579","Type":"ContainerStarted","Data":"7e5de9e59901905fa89ad4ec93af49829b1dd24045c764ed05b6c9552206b356"}
Jul 21 01:38:02.588775 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.588740    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gtp7f" event={"ID":"5dec2ddc-a63d-479d-8fa7-f422b0ed0579","Type":"ContainerStarted","Data":"20e7276fb320bd8985c10eb8a26239dd74a6808d10d82fb846a4a2cd76153805"}
Jul 21 01:38:02.628169 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.628082    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:02.729216 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:02.729163    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:03.063132 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:03.063095    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/734ab4c2-b74d-48ec-9b98-a0a42eaf797e-node-exporter-tls\") pod \"node-exporter-5g8kh\" (UID: \"734ab4c2-b74d-48ec-9b98-a0a42eaf797e\") " pod="openshift-monitoring/node-exporter-5g8kh"
Jul 21 01:38:03.063296 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:03.063147    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1e261108-17de-48b2-8051-544ca6a67905-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:03.065394 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:03.065371    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/734ab4c2-b74d-48ec-9b98-a0a42eaf797e-node-exporter-tls\") pod \"node-exporter-5g8kh\" (UID: \"734ab4c2-b74d-48ec-9b98-a0a42eaf797e\") " pod="openshift-monitoring/node-exporter-5g8kh"
Jul 21 01:38:03.065524 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:03.065508    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1e261108-17de-48b2-8051-544ca6a67905-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1e261108-17de-48b2-8051-544ca6a67905\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:03.280803 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:03.280770    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 21 01:38:03.364114 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:03.364037    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5g8kh"
Jul 21 01:38:03.387582 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:38:03.387536    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod734ab4c2_b74d_48ec_9b98_a0a42eaf797e.slice/crio-8729097fb0973b1db52c209bbedc7684173861d3fc604e9b69ab4e1723f92fe9 WatchSource:0}: Error finding container 8729097fb0973b1db52c209bbedc7684173861d3fc604e9b69ab4e1723f92fe9: Status 404 returned error can't find the container with id 8729097fb0973b1db52c209bbedc7684173861d3fc604e9b69ab4e1723f92fe9
Jul 21 01:38:03.461399 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:03.461325    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 21 01:38:03.464363 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:38:03.464336    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e261108_17de_48b2_8051_544ca6a67905.slice/crio-96d6b178b15ebc305ecaf497fa40000590dda3ba537bf028262f67158a861a8e WatchSource:0}: Error finding container 96d6b178b15ebc305ecaf497fa40000590dda3ba537bf028262f67158a861a8e: Status 404 returned error can't find the container with id 96d6b178b15ebc305ecaf497fa40000590dda3ba537bf028262f67158a861a8e
Jul 21 01:38:03.576647 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:03.576549    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:03.594412 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:03.594383    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1e261108-17de-48b2-8051-544ca6a67905","Type":"ContainerStarted","Data":"96d6b178b15ebc305ecaf497fa40000590dda3ba537bf028262f67158a861a8e"}
Jul 21 01:38:03.595923 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:03.595898    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5g8kh" event={"ID":"734ab4c2-b74d-48ec-9b98-a0a42eaf797e","Type":"ContainerStarted","Data":"8729097fb0973b1db52c209bbedc7684173861d3fc604e9b69ab4e1723f92fe9"}
Jul 21 01:38:03.676926 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:03.676896    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:04.122966 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:04.122926    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:04.225288 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:04.224532    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:04.324041 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:04.324000    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:05.605235 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:05.605203    2575 generic.go:356] "Generic (PLEG): container finished" podID="734ab4c2-b74d-48ec-9b98-a0a42eaf797e" containerID="e217f05781b9ba3b5c40e267f25fd475f5cabd5da73a449396fdc2a838562588" exitCode=0
Jul 21 01:38:05.605734 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:05.605290    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5g8kh" event={"ID":"734ab4c2-b74d-48ec-9b98-a0a42eaf797e","Type":"ContainerDied","Data":"e217f05781b9ba3b5c40e267f25fd475f5cabd5da73a449396fdc2a838562588"}
Jul 21 01:38:05.606958 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:05.606933    2575 generic.go:356] "Generic (PLEG): container finished" podID="1e261108-17de-48b2-8051-544ca6a67905" containerID="2073d302362a59d6cee43a8224133030983981ef851ecdfe1b4642f277241b21" exitCode=0
Jul 21 01:38:05.607075 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:05.607024    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1e261108-17de-48b2-8051-544ca6a67905","Type":"ContainerDied","Data":"2073d302362a59d6cee43a8224133030983981ef851ecdfe1b4642f277241b21"}
Jul 21 01:38:05.609205 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:05.609179    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gtp7f" event={"ID":"5dec2ddc-a63d-479d-8fa7-f422b0ed0579","Type":"ContainerStarted","Data":"c4b6e7562e1cbf7ac752b1dd3e673a2c79aaeaf3cb1fd69f859b24a9f1544bcb"}
Jul 21 01:38:05.690635 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:05.690573    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gtp7f" podStartSLOduration=2.257678114 podStartE2EDuration="4.690557757s" totalImagesPullingTime="2.432879643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:38:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:38:05.658765689 +0000 UTC m=+105.256406061" watchObservedRunningTime="2026-07-21 01:38:05.690557757 +0000 UTC m=+105.288198108"
Jul 21 01:38:05.965158 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:05.965080    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7fbdf944d4-p6p4r"]
Jul 21 01:38:05.975293 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:05.975268    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:05.985871 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:05.985846    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fbdf944d4-p6p4r"]
Jul 21 01:38:05.986020 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:05.985955    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fbdf944d4-p6p4r"
Jul 21 01:38:05.994663 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:05.994509    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 21 01:38:06.064671 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:06.064642    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:06.087193 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:06.087164    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f632ced6-7ead-4416-aaf8-95554d5c74b3-trusted-ca-bundle\") pod \"console-7fbdf944d4-p6p4r\" (UID: \"f632ced6-7ead-4416-aaf8-95554d5c74b3\") " pod="openshift-console/console-7fbdf944d4-p6p4r"
Jul 21 01:38:06.087339 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:06.087200    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f632ced6-7ead-4416-aaf8-95554d5c74b3-service-ca\") pod \"console-7fbdf944d4-p6p4r\" (UID: \"f632ced6-7ead-4416-aaf8-95554d5c74b3\") " pod="openshift-console/console-7fbdf944d4-p6p4r"
Jul 21 01:38:06.087339 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:06.087225    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f632ced6-7ead-4416-aaf8-95554d5c74b3-oauth-serving-cert\") pod \"console-7fbdf944d4-p6p4r\" (UID: \"f632ced6-7ead-4416-aaf8-95554d5c74b3\") " pod="openshift-console/console-7fbdf944d4-p6p4r"
Jul 21 01:38:06.087339 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:06.087327    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f632ced6-7ead-4416-aaf8-95554d5c74b3-console-config\") pod \"console-7fbdf944d4-p6p4r\" (UID: \"f632ced6-7ead-4416-aaf8-95554d5c74b3\") " pod="openshift-console/console-7fbdf944d4-p6p4r"
Jul 21 01:38:06.087488 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:06.087374    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f632ced6-7ead-4416-aaf8-95554d5c74b3-console-oauth-config\") pod \"console-7fbdf944d4-p6p4r\" (UID: \"f632ced6-7ead-4416-aaf8-95554d5c74b3\") " pod="openshift-console/console-7fbdf944d4-p6p4r"
Jul 21 01:38:06.087488 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:06.087471    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-762pd\" (UniqueName: \"kubernetes.io/projected/f632ced6-7ead-4416-aaf8-95554d5c74b3-kube-api-access-762pd\") pod \"console-7fbdf944d4-p6p4r\" (UID: \"f632ced6-7ead-4416-aaf8-95554d5c74b3\") " pod="openshift-console/console-7fbdf944d4-p6p4r"
Jul 21 01:38:06.087567 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:06.087501    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f632ced6-7ead-4416-aaf8-95554d5c74b3-console-serving-cert\") pod \"console-7fbdf944d4-p6p4r\" (UID: \"f632ced6-7ead-4416-aaf8-95554d5c74b3\") " pod="openshift-console/console-7fbdf944d4-p6p4r"
Jul 21 01:38:06.164377 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:06.164340    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:06.188514 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:06.188483    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f632ced6-7ead-4416-aaf8-95554d5c74b3-console-config\") pod \"console-7fbdf944d4-p6p4r\" (UID: \"f632ced6-7ead-4416-aaf8-95554d5c74b3\") " pod="openshift-console/console-7fbdf944d4-p6p4r"
Jul 21 01:38:06.188737 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:06.188533    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f632ced6-7ead-4416-aaf8-95554d5c74b3-console-oauth-config\") pod \"console-7fbdf944d4-p6p4r\" (UID: \"f632ced6-7ead-4416-aaf8-95554d5c74b3\") " pod="openshift-console/console-7fbdf944d4-p6p4r"
Jul 21 01:38:06.189403 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:06.188935    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-762pd\" (UniqueName: \"kubernetes.io/projected/f632ced6-7ead-4416-aaf8-95554d5c74b3-kube-api-access-762pd\") pod \"console-7fbdf944d4-p6p4r\" (UID: \"f632ced6-7ead-4416-aaf8-95554d5c74b3\") " pod="openshift-console/console-7fbdf944d4-p6p4r"
Jul 21 01:38:06.189403 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:06.188992    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f632ced6-7ead-4416-aaf8-95554d5c74b3-console-serving-cert\") pod \"console-7fbdf944d4-p6p4r\" (UID: \"f632ced6-7ead-4416-aaf8-95554d5c74b3\") " pod="openshift-console/console-7fbdf944d4-p6p4r"
Jul 21 01:38:06.189403 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:06.189027    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f632ced6-7ead-4416-aaf8-95554d5c74b3-trusted-ca-bundle\") pod \"console-7fbdf944d4-p6p4r\" (UID: \"f632ced6-7ead-4416-aaf8-95554d5c74b3\") " pod="openshift-console/console-7fbdf944d4-p6p4r"
Jul 21 01:38:06.189403 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:06.189056    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f632ced6-7ead-4416-aaf8-95554d5c74b3-service-ca\") pod \"console-7fbdf944d4-p6p4r\" (UID: \"f632ced6-7ead-4416-aaf8-95554d5c74b3\") " pod="openshift-console/console-7fbdf944d4-p6p4r"
Jul 21 01:38:06.189403 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:06.189081    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f632ced6-7ead-4416-aaf8-95554d5c74b3-oauth-serving-cert\") pod \"console-7fbdf944d4-p6p4r\" (UID: \"f632ced6-7ead-4416-aaf8-95554d5c74b3\") " pod="openshift-console/console-7fbdf944d4-p6p4r"
Jul 21 01:38:06.189403 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:06.189332    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f632ced6-7ead-4416-aaf8-95554d5c74b3-console-config\") pod \"console-7fbdf944d4-p6p4r\" (UID: \"f632ced6-7ead-4416-aaf8-95554d5c74b3\") " pod="openshift-console/console-7fbdf944d4-p6p4r"
Jul 21 01:38:06.189808 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:06.189741    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f632ced6-7ead-4416-aaf8-95554d5c74b3-oauth-serving-cert\") pod \"console-7fbdf944d4-p6p4r\" (UID: \"f632ced6-7ead-4416-aaf8-95554d5c74b3\") " pod="openshift-console/console-7fbdf944d4-p6p4r"
Jul 21 01:38:06.189942 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:06.189911    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f632ced6-7ead-4416-aaf8-95554d5c74b3-service-ca\") pod \"console-7fbdf944d4-p6p4r\" (UID: \"f632ced6-7ead-4416-aaf8-95554d5c74b3\") " pod="openshift-console/console-7fbdf944d4-p6p4r"
Jul 21 01:38:06.190068 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:06.190050    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f632ced6-7ead-4416-aaf8-95554d5c74b3-trusted-ca-bundle\") pod \"console-7fbdf944d4-p6p4r\" (UID: \"f632ced6-7ead-4416-aaf8-95554d5c74b3\") " pod="openshift-console/console-7fbdf944d4-p6p4r"
Jul 21 01:38:06.191799 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:06.191763    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f632ced6-7ead-4416-aaf8-95554d5c74b3-console-oauth-config\") pod \"console-7fbdf944d4-p6p4r\" (UID: \"f632ced6-7ead-4416-aaf8-95554d5c74b3\") " pod="openshift-console/console-7fbdf944d4-p6p4r"
Jul 21 01:38:06.191799 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:06.191763    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f632ced6-7ead-4416-aaf8-95554d5c74b3-console-serving-cert\") pod \"console-7fbdf944d4-p6p4r\" (UID: \"f632ced6-7ead-4416-aaf8-95554d5c74b3\") " pod="openshift-console/console-7fbdf944d4-p6p4r"
Jul 21 01:38:06.196780 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:06.196761    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-762pd\" (UniqueName: \"kubernetes.io/projected/f632ced6-7ead-4416-aaf8-95554d5c74b3-kube-api-access-762pd\") pod \"console-7fbdf944d4-p6p4r\" (UID: \"f632ced6-7ead-4416-aaf8-95554d5c74b3\") " pod="openshift-console/console-7fbdf944d4-p6p4r"
Jul 21 01:38:06.263978 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:06.263896    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:06.297727 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:06.297702    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fbdf944d4-p6p4r"
Jul 21 01:38:06.454469 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:06.454440    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fbdf944d4-p6p4r"]
Jul 21 01:38:06.457246 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:38:06.457215    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf632ced6_7ead_4416_aaf8_95554d5c74b3.slice/crio-26374f57f64451ab9744e151ef188c9a4e3f5f50ccc6d220209d245689961e44 WatchSource:0}: Error finding container 26374f57f64451ab9744e151ef188c9a4e3f5f50ccc6d220209d245689961e44: Status 404 returned error can't find the container with id 26374f57f64451ab9744e151ef188c9a4e3f5f50ccc6d220209d245689961e44
Jul 21 01:38:06.616530 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:06.616494    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fbdf944d4-p6p4r" event={"ID":"f632ced6-7ead-4416-aaf8-95554d5c74b3","Type":"ContainerStarted","Data":"1d6a3ed9fdf2ca94a104f7e7b40ea52ad3c715d887c246d69cb64b566e82edb7"}
Jul 21 01:38:06.616983 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:06.616554    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fbdf944d4-p6p4r" event={"ID":"f632ced6-7ead-4416-aaf8-95554d5c74b3","Type":"ContainerStarted","Data":"26374f57f64451ab9744e151ef188c9a4e3f5f50ccc6d220209d245689961e44"}
Jul 21 01:38:06.618832 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:06.618802    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5g8kh" event={"ID":"734ab4c2-b74d-48ec-9b98-a0a42eaf797e","Type":"ContainerStarted","Data":"e2d19fca13f05223ab5bab1cdc2ab79a723b5a5cea1f560808cd953cb266dd9b"}
Jul 21 01:38:06.618982 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:06.618872    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5g8kh" event={"ID":"734ab4c2-b74d-48ec-9b98-a0a42eaf797e","Type":"ContainerStarted","Data":"638a2ebba0c96ee043dd4ba274e4fef0de7bbae46d88d39b4f2dc98c175a8b74"}
Jul 21 01:38:06.636688 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:06.636629    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7fbdf944d4-p6p4r" podStartSLOduration=1.636589767 podStartE2EDuration="1.636589767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:38:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:38:06.634139415 +0000 UTC m=+106.231779767" watchObservedRunningTime="2026-07-21 01:38:06.636589767 +0000 UTC m=+106.234230120"
Jul 21 01:38:06.659817 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:06.659772    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5g8kh" podStartSLOduration=4.057895685 podStartE2EDuration="5.659756727s" totalImagesPullingTime="1.525530188s" totalInitContainerRuntime="76.330854ms" isStatefulPod=true podCreationTimestamp="2026-07-21 01:38:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:38:06.658299668 +0000 UTC m=+106.255940019" watchObservedRunningTime="2026-07-21 01:38:06.659756727 +0000 UTC m=+106.257397077"
Jul 21 01:38:06.732352 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:06.732310    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:06.781593 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:06.781512    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:07.273100 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.273056    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:07.368185 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.368150    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:07.372666 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.372643    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 21 01:38:07.389591 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.389566    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.393888 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.393847    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5d990860p99ce\""
Jul 21 01:38:07.393888 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.393877    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 21 01:38:07.394110 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.393899    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 21 01:38:07.394110 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.394026    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 21 01:38:07.394110 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.394054    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 21 01:38:07.394110 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.394094    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 21 01:38:07.394306 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.394148    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 21 01:38:07.394306 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.394196    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 21 01:38:07.394306 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.394223    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 21 01:38:07.394306 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.394245    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 21 01:38:07.394508 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.394414    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 21 01:38:07.394508 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.394423    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-j7ghp\""
Jul 21 01:38:07.394508 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.394439    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 21 01:38:07.394680 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.394659    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 21 01:38:07.394737 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.394680    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 21 01:38:07.394737 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.394695    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 21 01:38:07.395800 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.395778    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 21 01:38:07.401180 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.401155    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 21 01:38:07.402461 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.402438    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c426235f-0cee-44f4-b4f7-5ebc837e7f22-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.402648 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.402482    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c426235f-0cee-44f4-b4f7-5ebc837e7f22-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.402648 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.402523    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c426235f-0cee-44f4-b4f7-5ebc837e7f22-web-config\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.402648 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.402576    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c426235f-0cee-44f4-b4f7-5ebc837e7f22-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.402648 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.402631    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c426235f-0cee-44f4-b4f7-5ebc837e7f22-config-out\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.402872 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.402669    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c426235f-0cee-44f4-b4f7-5ebc837e7f22-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.402872 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.402711    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c426235f-0cee-44f4-b4f7-5ebc837e7f22-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.402872 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.402745    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9h7z\" (UniqueName: \"kubernetes.io/projected/c426235f-0cee-44f4-b4f7-5ebc837e7f22-kube-api-access-w9h7z\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.402872 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.402774    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c426235f-0cee-44f4-b4f7-5ebc837e7f22-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.402872 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.402804    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c426235f-0cee-44f4-b4f7-5ebc837e7f22-config\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.402872 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.402829    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c426235f-0cee-44f4-b4f7-5ebc837e7f22-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.402872 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.402857    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c426235f-0cee-44f4-b4f7-5ebc837e7f22-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.403204 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.402879    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c426235f-0cee-44f4-b4f7-5ebc837e7f22-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.403204 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.402935    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c426235f-0cee-44f4-b4f7-5ebc837e7f22-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.403204 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.402958    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c426235f-0cee-44f4-b4f7-5ebc837e7f22-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.403204 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.402987    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c426235f-0cee-44f4-b4f7-5ebc837e7f22-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.403204 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.403044    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c426235f-0cee-44f4-b4f7-5ebc837e7f22-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.403204 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.403071    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c426235f-0cee-44f4-b4f7-5ebc837e7f22-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.403204 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.403097    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c426235f-0cee-44f4-b4f7-5ebc837e7f22-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.403204 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.403125    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c426235f-0cee-44f4-b4f7-5ebc837e7f22-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.485319 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.485259    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:07.504405 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.504376    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c426235f-0cee-44f4-b4f7-5ebc837e7f22-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.504405 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.504407    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c426235f-0cee-44f4-b4f7-5ebc837e7f22-config-out\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.504577 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.504425    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c426235f-0cee-44f4-b4f7-5ebc837e7f22-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.504577 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.504449    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c426235f-0cee-44f4-b4f7-5ebc837e7f22-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.504577 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:38:07.504552    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c426235f-0cee-44f4-b4f7-5ebc837e7f22-prometheus-trusted-ca-bundle podName:c426235f-0cee-44f4-b4f7-5ebc837e7f22 nodeName:}" failed. No retries permitted until 2026-07-21 01:38:08.00453316 +0000 UTC m=+107.602173489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/c426235f-0cee-44f4-b4f7-5ebc837e7f22-prometheus-trusted-ca-bundle") pod "prometheus-k8s-0" (UID: "c426235f-0cee-44f4-b4f7-5ebc837e7f22") : configmap references non-existent config key: ca-bundle.crt
Jul 21 01:38:07.504577 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.504568    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9h7z\" (UniqueName: \"kubernetes.io/projected/c426235f-0cee-44f4-b4f7-5ebc837e7f22-kube-api-access-w9h7z\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.504821 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.504594    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c426235f-0cee-44f4-b4f7-5ebc837e7f22-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.504821 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.504734    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c426235f-0cee-44f4-b4f7-5ebc837e7f22-config\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.504821 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.504774    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c426235f-0cee-44f4-b4f7-5ebc837e7f22-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.504821 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.504806    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c426235f-0cee-44f4-b4f7-5ebc837e7f22-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.505012 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.504829    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c426235f-0cee-44f4-b4f7-5ebc837e7f22-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.505012 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.504873    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c426235f-0cee-44f4-b4f7-5ebc837e7f22-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.505012 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.504898    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c426235f-0cee-44f4-b4f7-5ebc837e7f22-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.505012 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.504927    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c426235f-0cee-44f4-b4f7-5ebc837e7f22-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.505012 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.504970    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c426235f-0cee-44f4-b4f7-5ebc837e7f22-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.505012 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.504993    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c426235f-0cee-44f4-b4f7-5ebc837e7f22-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.505299 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.505017    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c426235f-0cee-44f4-b4f7-5ebc837e7f22-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.505299 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.505043    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c426235f-0cee-44f4-b4f7-5ebc837e7f22-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.505299 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.505111    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c426235f-0cee-44f4-b4f7-5ebc837e7f22-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.505299 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.505153    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c426235f-0cee-44f4-b4f7-5ebc837e7f22-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.505299 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.505159    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c426235f-0cee-44f4-b4f7-5ebc837e7f22-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.505299 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.505189    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c426235f-0cee-44f4-b4f7-5ebc837e7f22-web-config\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.506245 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.505999    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c426235f-0cee-44f4-b4f7-5ebc837e7f22-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.506245 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.506227    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c426235f-0cee-44f4-b4f7-5ebc837e7f22-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.509470 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.508924    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c426235f-0cee-44f4-b4f7-5ebc837e7f22-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.509470 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.509054    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c426235f-0cee-44f4-b4f7-5ebc837e7f22-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.509470 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.509240    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c426235f-0cee-44f4-b4f7-5ebc837e7f22-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.509470 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.509342    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c426235f-0cee-44f4-b4f7-5ebc837e7f22-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.509470 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.509365    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c426235f-0cee-44f4-b4f7-5ebc837e7f22-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.509992 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.509696    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c426235f-0cee-44f4-b4f7-5ebc837e7f22-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.509992 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.509773    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c426235f-0cee-44f4-b4f7-5ebc837e7f22-config-out\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.509992 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.509974    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c426235f-0cee-44f4-b4f7-5ebc837e7f22-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.511725 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.511651    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c426235f-0cee-44f4-b4f7-5ebc837e7f22-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.513971 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.512602    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c426235f-0cee-44f4-b4f7-5ebc837e7f22-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.513971 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.512886    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c426235f-0cee-44f4-b4f7-5ebc837e7f22-web-config\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.513971 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.513262    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c426235f-0cee-44f4-b4f7-5ebc837e7f22-config\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.514181 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.514129    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c426235f-0cee-44f4-b4f7-5ebc837e7f22-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.514181 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.514141    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c426235f-0cee-44f4-b4f7-5ebc837e7f22-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.518638 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.516007    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c426235f-0cee-44f4-b4f7-5ebc837e7f22-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.518638 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.517542    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9h7z\" (UniqueName: \"kubernetes.io/projected/c426235f-0cee-44f4-b4f7-5ebc837e7f22-kube-api-access-w9h7z\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:07.626377 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:07.626308    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1e261108-17de-48b2-8051-544ca6a67905","Type":"ContainerStarted","Data":"a756aa035fb3350fe41c4866814505bd7b540af9a7dce87442c93b0856d80a71"}
Jul 21 01:38:08.008803 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:08.008773    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c426235f-0cee-44f4-b4f7-5ebc837e7f22-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:08.009681 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:08.009659    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c426235f-0cee-44f4-b4f7-5ebc837e7f22-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c426235f-0cee-44f4-b4f7-5ebc837e7f22\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:08.301861 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:08.301782    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:08.448295 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:08.448272    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 21 01:38:08.451021 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:38:08.450996    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc426235f_0cee_44f4_b4f7_5ebc837e7f22.slice/crio-3c6e18c9790ab73ddb830b9f07a22ccd84f5e8bdc6f55dc9aeafd1528992b34f WatchSource:0}: Error finding container 3c6e18c9790ab73ddb830b9f07a22ccd84f5e8bdc6f55dc9aeafd1528992b34f: Status 404 returned error can't find the container with id 3c6e18c9790ab73ddb830b9f07a22ccd84f5e8bdc6f55dc9aeafd1528992b34f
Jul 21 01:38:08.632829 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:08.632799    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1e261108-17de-48b2-8051-544ca6a67905","Type":"ContainerStarted","Data":"fd37fcfc50537c486b2d930c89c4cbb8166d356a09a2bd232b7e0bca189af65e"}
Jul 21 01:38:08.633204 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:08.632879    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1e261108-17de-48b2-8051-544ca6a67905","Type":"ContainerStarted","Data":"caa060ced32458cfc37127d21574e1bbf2da57bc567438f53afc219d65f49351"}
Jul 21 01:38:08.633204 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:08.632924    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1e261108-17de-48b2-8051-544ca6a67905","Type":"ContainerStarted","Data":"22e7a7e6738bcd9eb54cb516718a9eb12930f6f231429fa49a0c515b00800e1b"}
Jul 21 01:38:08.633204 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:08.632961    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1e261108-17de-48b2-8051-544ca6a67905","Type":"ContainerStarted","Data":"be66828a326467fe4e3b17677f2c7d22f734f728c2e2680064434867ee1afc3b"}
Jul 21 01:38:08.634275 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:08.634255    2575 generic.go:356] "Generic (PLEG): container finished" podID="c426235f-0cee-44f4-b4f7-5ebc837e7f22" containerID="c56078108b68ee6e69e32e98081657335ecac6817212e36d0943b699d89192c4" exitCode=0
Jul 21 01:38:08.634332 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:08.634284    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c426235f-0cee-44f4-b4f7-5ebc837e7f22","Type":"ContainerDied","Data":"c56078108b68ee6e69e32e98081657335ecac6817212e36d0943b699d89192c4"}
Jul 21 01:38:08.634332 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:08.634328    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c426235f-0cee-44f4-b4f7-5ebc837e7f22","Type":"ContainerStarted","Data":"3c6e18c9790ab73ddb830b9f07a22ccd84f5e8bdc6f55dc9aeafd1528992b34f"}
Jul 21 01:38:09.644185 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:09.644148    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1e261108-17de-48b2-8051-544ca6a67905","Type":"ContainerStarted","Data":"1e9947617c97f238ddcbbf62798444de6a097f5c841a8108884c3c6027157721"}
Jul 21 01:38:09.680699 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:09.680577    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.927441424 podStartE2EDuration="7.680555817s" totalImagesPullingTime="4.730118123s" totalInitContainerRuntime="22.99627ms" isStatefulPod=false podCreationTimestamp="2026-07-21 01:38:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:38:09.678736298 +0000 UTC m=+109.276376666" watchObservedRunningTime="2026-07-21 01:38:09.680555817 +0000 UTC m=+109.278196170"
Jul 21 01:38:10.484675 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:10.484639    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xbk6c"
Jul 21 01:38:12.548420 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:12.548396    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fbdf944d4-p6p4r"]
Jul 21 01:38:13.665126 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:13.665089    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c426235f-0cee-44f4-b4f7-5ebc837e7f22","Type":"ContainerStarted","Data":"3b7d92103ae231eb6d43be7b66da2578aae0d254b61f817e5a163c89eda9837f"}
Jul 21 01:38:13.665518 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:13.665167    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c426235f-0cee-44f4-b4f7-5ebc837e7f22","Type":"ContainerStarted","Data":"2893605c79c21e4e82143c577619f1e0be030227f6b8edfe5a55175a4ae83b5d"}
Jul 21 01:38:15.676470 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:15.676440    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c426235f-0cee-44f4-b4f7-5ebc837e7f22","Type":"ContainerStarted","Data":"ae886c10b77bf5eb7a7754c3f1cb05b1b8e4d64bfe691458f9fef2d1278a2881"}
Jul 21 01:38:15.676952 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:15.676505    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c426235f-0cee-44f4-b4f7-5ebc837e7f22","Type":"ContainerStarted","Data":"4f70e5086b9a97541f743c937408f62f3ee304530583bc42810c62eefa5096ae"}
Jul 21 01:38:15.676952 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:15.676532    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c426235f-0cee-44f4-b4f7-5ebc837e7f22","Type":"ContainerStarted","Data":"2501d41ad2e1eec013e73a13acaf1b429186bf7675ab331ef9e5a76e6a6a540a"}
Jul 21 01:38:15.676952 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:15.676556    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c426235f-0cee-44f4-b4f7-5ebc837e7f22","Type":"ContainerStarted","Data":"bc2a31c2f026491cd4e88ec9e153be3a62de824843bbf1d9b0a7f72f914b4a16"}
Jul 21 01:38:15.705498 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:15.705443    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.767476441 podStartE2EDuration="8.705423445s" totalImagesPullingTime="5.9187258s" totalInitContainerRuntime="19.221204ms" isStatefulPod=false podCreationTimestamp="2026-07-21 01:38:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:38:15.702919475 +0000 UTC m=+115.300559826" watchObservedRunningTime="2026-07-21 01:38:15.705423445 +0000 UTC m=+115.303063796"
Jul 21 01:38:16.298038 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:16.298003    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7fbdf944d4-p6p4r"
Jul 21 01:38:18.302172 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:18.302142    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:38:21.056055 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:21.056027    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:21.116249 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:21.116186    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:23.818542 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:23.818504    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:23.920742 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:23.920707    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:24.400385 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:24.400357    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-drc4p_86db8e5d-3869-4ad0-8d3d-989e132f6038/serve-healthcheck-canary/0.log"
Jul 21 01:38:27.607530 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:27.607467    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7ccdfdb849-jjmzc" podUID="3de54e9e-7461-45fb-ab72-2e66eebbf253" containerName="console" containerID="cri-o://6d330c125817f7dc075da5080e1041dd3f55b5c2df9d9c484c9f36d5ffdfc65a" gracePeriod=15
Jul 21 01:38:27.724831 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:27.724798    2575 generic.go:356] "Generic (PLEG): container finished" podID="3de54e9e-7461-45fb-ab72-2e66eebbf253" containerID="6d330c125817f7dc075da5080e1041dd3f55b5c2df9d9c484c9f36d5ffdfc65a" exitCode=0
Jul 21 01:38:27.724964 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:27.724867    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ccdfdb849-jjmzc" event={"ID":"3de54e9e-7461-45fb-ab72-2e66eebbf253","Type":"ContainerDied","Data":"6d330c125817f7dc075da5080e1041dd3f55b5c2df9d9c484c9f36d5ffdfc65a"}
Jul 21 01:38:27.861886 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:27.861830    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ccdfdb849-jjmzc"
Jul 21 01:38:27.993870 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:27.993840    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3de54e9e-7461-45fb-ab72-2e66eebbf253-console-serving-cert\") pod \"3de54e9e-7461-45fb-ab72-2e66eebbf253\" (UID: \"3de54e9e-7461-45fb-ab72-2e66eebbf253\") "
Jul 21 01:38:27.994057 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:27.993891    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3de54e9e-7461-45fb-ab72-2e66eebbf253-console-oauth-config\") pod \"3de54e9e-7461-45fb-ab72-2e66eebbf253\" (UID: \"3de54e9e-7461-45fb-ab72-2e66eebbf253\") "
Jul 21 01:38:27.994057 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:27.993919    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3de54e9e-7461-45fb-ab72-2e66eebbf253-oauth-serving-cert\") pod \"3de54e9e-7461-45fb-ab72-2e66eebbf253\" (UID: \"3de54e9e-7461-45fb-ab72-2e66eebbf253\") "
Jul 21 01:38:27.994057 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:27.993981    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3de54e9e-7461-45fb-ab72-2e66eebbf253-console-config\") pod \"3de54e9e-7461-45fb-ab72-2e66eebbf253\" (UID: \"3de54e9e-7461-45fb-ab72-2e66eebbf253\") "
Jul 21 01:38:27.994057 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:27.994041    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6qpn\" (UniqueName: \"kubernetes.io/projected/3de54e9e-7461-45fb-ab72-2e66eebbf253-kube-api-access-d6qpn\") pod \"3de54e9e-7461-45fb-ab72-2e66eebbf253\" (UID: \"3de54e9e-7461-45fb-ab72-2e66eebbf253\") "
Jul 21 01:38:27.994264 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:27.994068    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3de54e9e-7461-45fb-ab72-2e66eebbf253-service-ca\") pod \"3de54e9e-7461-45fb-ab72-2e66eebbf253\" (UID: \"3de54e9e-7461-45fb-ab72-2e66eebbf253\") "
Jul 21 01:38:27.994414 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:27.994382    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3de54e9e-7461-45fb-ab72-2e66eebbf253-console-config" (OuterVolumeSpecName: "console-config") pod "3de54e9e-7461-45fb-ab72-2e66eebbf253" (UID: "3de54e9e-7461-45fb-ab72-2e66eebbf253"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 01:38:27.994542 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:27.994462    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3de54e9e-7461-45fb-ab72-2e66eebbf253-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3de54e9e-7461-45fb-ab72-2e66eebbf253" (UID: "3de54e9e-7461-45fb-ab72-2e66eebbf253"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 01:38:27.994542 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:27.994515    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3de54e9e-7461-45fb-ab72-2e66eebbf253-service-ca" (OuterVolumeSpecName: "service-ca") pod "3de54e9e-7461-45fb-ab72-2e66eebbf253" (UID: "3de54e9e-7461-45fb-ab72-2e66eebbf253"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 01:38:27.996178 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:27.996155    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3de54e9e-7461-45fb-ab72-2e66eebbf253-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3de54e9e-7461-45fb-ab72-2e66eebbf253" (UID: "3de54e9e-7461-45fb-ab72-2e66eebbf253"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 01:38:27.996178 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:27.996165    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de54e9e-7461-45fb-ab72-2e66eebbf253-kube-api-access-d6qpn" (OuterVolumeSpecName: "kube-api-access-d6qpn") pod "3de54e9e-7461-45fb-ab72-2e66eebbf253" (UID: "3de54e9e-7461-45fb-ab72-2e66eebbf253"). InnerVolumeSpecName "kube-api-access-d6qpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 01:38:27.996306 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:27.996173    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3de54e9e-7461-45fb-ab72-2e66eebbf253-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3de54e9e-7461-45fb-ab72-2e66eebbf253" (UID: "3de54e9e-7461-45fb-ab72-2e66eebbf253"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 01:38:28.094904 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:28.094852    2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3de54e9e-7461-45fb-ab72-2e66eebbf253-console-config\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 01:38:28.094904 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:28.094899    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6qpn\" (UniqueName: \"kubernetes.io/projected/3de54e9e-7461-45fb-ab72-2e66eebbf253-kube-api-access-d6qpn\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 01:38:28.094904 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:28.094910    2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3de54e9e-7461-45fb-ab72-2e66eebbf253-service-ca\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 01:38:28.094904 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:28.094919    2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3de54e9e-7461-45fb-ab72-2e66eebbf253-console-serving-cert\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 01:38:28.095156 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:28.094928    2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3de54e9e-7461-45fb-ab72-2e66eebbf253-console-oauth-config\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 01:38:28.095156 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:28.094936    2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3de54e9e-7461-45fb-ab72-2e66eebbf253-oauth-serving-cert\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 01:38:28.730449 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:28.730420    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ccdfdb849-jjmzc"
Jul 21 01:38:28.730936 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:28.730421    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ccdfdb849-jjmzc" event={"ID":"3de54e9e-7461-45fb-ab72-2e66eebbf253","Type":"ContainerDied","Data":"96864016f1f349a26816e042241f9c5f228b25161e440e1e9c37568b148e279e"}
Jul 21 01:38:28.730936 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:28.730547    2575 scope.go:117] "RemoveContainer" containerID="6d330c125817f7dc075da5080e1041dd3f55b5c2df9d9c484c9f36d5ffdfc65a"
Jul 21 01:38:28.749776 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:28.749751    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7ccdfdb849-jjmzc"]
Jul 21 01:38:28.755248 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:28.755228    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7ccdfdb849-jjmzc"]
Jul 21 01:38:29.025225 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:29.025194    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3de54e9e-7461-45fb-ab72-2e66eebbf253" path="/var/lib/kubelet/pods/3de54e9e-7461-45fb-ab72-2e66eebbf253/volumes"
Jul 21 01:38:30.180867 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:30.180834    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:30.282824 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:30.282792    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:30.347488 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:30.347452    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:37.125734 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:37.125636    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:38:37.571121 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:37.571058    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7fbdf944d4-p6p4r" podUID="f632ced6-7ead-4416-aaf8-95554d5c74b3" containerName="console" containerID="cri-o://1d6a3ed9fdf2ca94a104f7e7b40ea52ad3c715d887c246d69cb64b566e82edb7" gracePeriod=15
Jul 21 01:38:37.778875 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:37.778731    2575 generic.go:356] "Generic (PLEG): container finished" podID="f632ced6-7ead-4416-aaf8-95554d5c74b3" containerID="1d6a3ed9fdf2ca94a104f7e7b40ea52ad3c715d887c246d69cb64b566e82edb7" exitCode=0
Jul 21 01:38:37.778875 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:37.778803    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fbdf944d4-p6p4r" event={"ID":"f632ced6-7ead-4416-aaf8-95554d5c74b3","Type":"ContainerDied","Data":"1d6a3ed9fdf2ca94a104f7e7b40ea52ad3c715d887c246d69cb64b566e82edb7"}
Jul 21 01:38:37.900819 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:37.900786    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fbdf944d4-p6p4r"
Jul 21 01:38:37.977275 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:37.977247    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f632ced6-7ead-4416-aaf8-95554d5c74b3-oauth-serving-cert\") pod \"f632ced6-7ead-4416-aaf8-95554d5c74b3\" (UID: \"f632ced6-7ead-4416-aaf8-95554d5c74b3\") "
Jul 21 01:38:37.977275 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:37.977280    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f632ced6-7ead-4416-aaf8-95554d5c74b3-console-serving-cert\") pod \"f632ced6-7ead-4416-aaf8-95554d5c74b3\" (UID: \"f632ced6-7ead-4416-aaf8-95554d5c74b3\") "
Jul 21 01:38:37.977519 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:37.977314    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f632ced6-7ead-4416-aaf8-95554d5c74b3-console-oauth-config\") pod \"f632ced6-7ead-4416-aaf8-95554d5c74b3\" (UID: \"f632ced6-7ead-4416-aaf8-95554d5c74b3\") "
Jul 21 01:38:37.977519 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:37.977348    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f632ced6-7ead-4416-aaf8-95554d5c74b3-trusted-ca-bundle\") pod \"f632ced6-7ead-4416-aaf8-95554d5c74b3\" (UID: \"f632ced6-7ead-4416-aaf8-95554d5c74b3\") "
Jul 21 01:38:37.977519 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:37.977369    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-762pd\" (UniqueName: \"kubernetes.io/projected/f632ced6-7ead-4416-aaf8-95554d5c74b3-kube-api-access-762pd\") pod \"f632ced6-7ead-4416-aaf8-95554d5c74b3\" (UID: \"f632ced6-7ead-4416-aaf8-95554d5c74b3\") "
Jul 21 01:38:37.977519 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:37.977406    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f632ced6-7ead-4416-aaf8-95554d5c74b3-service-ca\") pod \"f632ced6-7ead-4416-aaf8-95554d5c74b3\" (UID: \"f632ced6-7ead-4416-aaf8-95554d5c74b3\") "
Jul 21 01:38:37.977519 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:37.977467    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f632ced6-7ead-4416-aaf8-95554d5c74b3-console-config\") pod \"f632ced6-7ead-4416-aaf8-95554d5c74b3\" (UID: \"f632ced6-7ead-4416-aaf8-95554d5c74b3\") "
Jul 21 01:38:37.977836 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:37.977805    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f632ced6-7ead-4416-aaf8-95554d5c74b3-service-ca" (OuterVolumeSpecName: "service-ca") pod "f632ced6-7ead-4416-aaf8-95554d5c74b3" (UID: "f632ced6-7ead-4416-aaf8-95554d5c74b3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 01:38:37.977896 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:37.977815    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f632ced6-7ead-4416-aaf8-95554d5c74b3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f632ced6-7ead-4416-aaf8-95554d5c74b3" (UID: "f632ced6-7ead-4416-aaf8-95554d5c74b3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 01:38:37.978021 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:37.977997    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f632ced6-7ead-4416-aaf8-95554d5c74b3-console-config" (OuterVolumeSpecName: "console-config") pod "f632ced6-7ead-4416-aaf8-95554d5c74b3" (UID: "f632ced6-7ead-4416-aaf8-95554d5c74b3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 01:38:37.978096 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:37.978034    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f632ced6-7ead-4416-aaf8-95554d5c74b3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f632ced6-7ead-4416-aaf8-95554d5c74b3" (UID: "f632ced6-7ead-4416-aaf8-95554d5c74b3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 01:38:37.979517 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:37.979495    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f632ced6-7ead-4416-aaf8-95554d5c74b3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f632ced6-7ead-4416-aaf8-95554d5c74b3" (UID: "f632ced6-7ead-4416-aaf8-95554d5c74b3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 01:38:37.979652 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:37.979538    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f632ced6-7ead-4416-aaf8-95554d5c74b3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f632ced6-7ead-4416-aaf8-95554d5c74b3" (UID: "f632ced6-7ead-4416-aaf8-95554d5c74b3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 01:38:37.980209 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:37.980191    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f632ced6-7ead-4416-aaf8-95554d5c74b3-kube-api-access-762pd" (OuterVolumeSpecName: "kube-api-access-762pd") pod "f632ced6-7ead-4416-aaf8-95554d5c74b3" (UID: "f632ced6-7ead-4416-aaf8-95554d5c74b3"). InnerVolumeSpecName "kube-api-access-762pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 01:38:38.078939 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:38.078898    2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f632ced6-7ead-4416-aaf8-95554d5c74b3-console-config\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 01:38:38.079181 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:38.079164    2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f632ced6-7ead-4416-aaf8-95554d5c74b3-oauth-serving-cert\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 01:38:38.079290 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:38.079277    2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f632ced6-7ead-4416-aaf8-95554d5c74b3-console-serving-cert\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 01:38:38.079384 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:38.079371    2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f632ced6-7ead-4416-aaf8-95554d5c74b3-console-oauth-config\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 01:38:38.079470 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:38.079458    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f632ced6-7ead-4416-aaf8-95554d5c74b3-trusted-ca-bundle\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 01:38:38.079555 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:38.079543    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-762pd\" (UniqueName: \"kubernetes.io/projected/f632ced6-7ead-4416-aaf8-95554d5c74b3-kube-api-access-762pd\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 01:38:38.079661 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:38.079647    2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f632ced6-7ead-4416-aaf8-95554d5c74b3-service-ca\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 01:38:38.783398 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:38.783361    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fbdf944d4-p6p4r" event={"ID":"f632ced6-7ead-4416-aaf8-95554d5c74b3","Type":"ContainerDied","Data":"26374f57f64451ab9744e151ef188c9a4e3f5f50ccc6d220209d245689961e44"}
Jul 21 01:38:38.783398 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:38.783381    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fbdf944d4-p6p4r"
Jul 21 01:38:38.783910 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:38.783421    2575 scope.go:117] "RemoveContainer" containerID="1d6a3ed9fdf2ca94a104f7e7b40ea52ad3c715d887c246d69cb64b566e82edb7"
Jul 21 01:38:38.802988 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:38.802965    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fbdf944d4-p6p4r"]
Jul 21 01:38:38.806681 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:38.806657    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7fbdf944d4-p6p4r"]
Jul 21 01:38:39.024853 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:38:39.024820    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f632ced6-7ead-4416-aaf8-95554d5c74b3" path="/var/lib/kubelet/pods/f632ced6-7ead-4416-aaf8-95554d5c74b3/volumes"
Jul 21 01:39:08.302756 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:39:08.302713    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:39:08.322148 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:39:08.322119    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:39:08.913658 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:39:08.913597    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 01:39:17.275151 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:39:17.275115    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:39:17.325735 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:39:17.325679    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:39:20.860118 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:39:20.860078    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:39:20.911990 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:39:20.911951    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:39:28.002870 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:39:28.002829    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:39:28.052374 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:39:28.052339    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:39:28.111802 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:39:28.111776    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:39:28.162578 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:39:28.162551    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:39:28.918800 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:39:28.918770    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:39:29.020011 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:39:29.019985    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:39:29.211546 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:39:29.211477    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:39:29.265692 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:39:29.265664    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:39:36.056439 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:39:36.056400    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:39:36.154673 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:39:36.154645    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:39:39.115730 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:39:39.115681    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:39:39.169907 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:39:39.169877    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:39:54.584799 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:39:54.584755    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:39:54.647586 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:39:54.647555    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:39:56.305949 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:39:56.305919    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:39:57.779779 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:39:57.779745    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:39:57.881577 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:39:57.881544    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:40:24.391124 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:24.391094    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:40:25.172482 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:25.172452    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:40:27.991942 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:27.991911    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:40:28.040161 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:28.040134    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:40:29.007521 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:29.007490    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:40:29.174733 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:29.174701    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:40:29.498895 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:29.498862    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:40:29.601544 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:29.601512    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:40:31.868569 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:31.868532    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:40:32.080744 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:32.080714    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:40:34.790205 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:34.790174    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:40:34.852751 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:34.852722    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:40:38.209246 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:38.209210    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:40:38.306629 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:38.306579    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:40:39.516001 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:39.515965    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:40:39.788404 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:39.788377    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:40:39.837797 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:39.837769    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:40:40.238872 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:40.238796    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hlvs5"]
Jul 21 01:40:40.239085 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:40.239073    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3de54e9e-7461-45fb-ab72-2e66eebbf253" containerName="console"
Jul 21 01:40:40.239125 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:40.239086    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3de54e9e-7461-45fb-ab72-2e66eebbf253" containerName="console"
Jul 21 01:40:40.239125 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:40.239101    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f632ced6-7ead-4416-aaf8-95554d5c74b3" containerName="console"
Jul 21 01:40:40.239125 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:40.239107    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f632ced6-7ead-4416-aaf8-95554d5c74b3" containerName="console"
Jul 21 01:40:40.239216 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:40.239161    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3de54e9e-7461-45fb-ab72-2e66eebbf253" containerName="console"
Jul 21 01:40:40.239216 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:40.239169    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f632ced6-7ead-4416-aaf8-95554d5c74b3" containerName="console"
Jul 21 01:40:40.241049 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:40.241028    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hlvs5"
Jul 21 01:40:40.243312 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:40.243296    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 21 01:40:40.243399 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:40.243335    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 21 01:40:40.244208 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:40.244171    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-d8gmz\""
Jul 21 01:40:40.251950 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:40.251930    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hlvs5"]
Jul 21 01:40:40.360839 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:40.360810    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjc6n\" (UniqueName: \"kubernetes.io/projected/e2880148-447d-4138-87cb-b4b8840bb62c-kube-api-access-vjc6n\") pod \"openshift-lws-operator-fd8ccff4c-hlvs5\" (UID: \"e2880148-447d-4138-87cb-b4b8840bb62c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hlvs5"
Jul 21 01:40:40.360994 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:40.360850    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2880148-447d-4138-87cb-b4b8840bb62c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hlvs5\" (UID: \"e2880148-447d-4138-87cb-b4b8840bb62c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hlvs5"
Jul 21 01:40:40.461843 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:40.461809    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjc6n\" (UniqueName: \"kubernetes.io/projected/e2880148-447d-4138-87cb-b4b8840bb62c-kube-api-access-vjc6n\") pod \"openshift-lws-operator-fd8ccff4c-hlvs5\" (UID: \"e2880148-447d-4138-87cb-b4b8840bb62c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hlvs5"
Jul 21 01:40:40.461843 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:40.461849    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2880148-447d-4138-87cb-b4b8840bb62c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hlvs5\" (UID: \"e2880148-447d-4138-87cb-b4b8840bb62c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hlvs5"
Jul 21 01:40:40.462234 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:40.462217    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2880148-447d-4138-87cb-b4b8840bb62c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hlvs5\" (UID: \"e2880148-447d-4138-87cb-b4b8840bb62c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hlvs5"
Jul 21 01:40:40.469844 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:40.469815    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjc6n\" (UniqueName: \"kubernetes.io/projected/e2880148-447d-4138-87cb-b4b8840bb62c-kube-api-access-vjc6n\") pod \"openshift-lws-operator-fd8ccff4c-hlvs5\" (UID: \"e2880148-447d-4138-87cb-b4b8840bb62c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hlvs5"
Jul 21 01:40:40.561527 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:40.561498    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hlvs5"
Jul 21 01:40:40.683862 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:40.683841    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hlvs5"]
Jul 21 01:40:40.686419 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:40:40.686383    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2880148_447d_4138_87cb_b4b8840bb62c.slice/crio-4705362d1b5b3bb1846b9ab6c1cb9a340c0b1aaf54a6972115678a2bc9a023c2 WatchSource:0}: Error finding container 4705362d1b5b3bb1846b9ab6c1cb9a340c0b1aaf54a6972115678a2bc9a023c2: Status 404 returned error can't find the container with id 4705362d1b5b3bb1846b9ab6c1cb9a340c0b1aaf54a6972115678a2bc9a023c2
Jul 21 01:40:41.223404 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:41.223368    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hlvs5" event={"ID":"e2880148-447d-4138-87cb-b4b8840bb62c","Type":"ContainerStarted","Data":"4705362d1b5b3bb1846b9ab6c1cb9a340c0b1aaf54a6972115678a2bc9a023c2"}
Jul 21 01:40:43.233185 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:43.233142    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hlvs5" event={"ID":"e2880148-447d-4138-87cb-b4b8840bb62c","Type":"ContainerStarted","Data":"74ba8d1ef7931d34f569b8b1479f8b32c03ce8f6305335e6bb16862977c87a28"}
Jul 21 01:40:43.252105 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:43.252058    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hlvs5" podStartSLOduration=0.855495076 podStartE2EDuration="3.252042961s" totalImagesPullingTime="2.396547885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:40:43.250643605 +0000 UTC m=+262.848283961" watchObservedRunningTime="2026-07-21 01:40:43.252042961 +0000 UTC m=+262.849683312"
Jul 21 01:40:43.969709 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:43.969678    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:40:44.070045 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:44.070010    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:40:45.942153 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:45.942127    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:40:58.827883 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:58.827848    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:40:58.925608 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:58.925578    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:40:59.415883 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:59.415844    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:40:59.705579 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:59.705499    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:40:59.809473 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:59.809440    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:40:59.850707 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:59.850670    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj"]
Jul 21 01:40:59.858600 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:59.858577    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj"
Jul 21 01:40:59.861781 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:59.861756    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 21 01:40:59.861938 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:59.861768    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wdvgf\""
Jul 21 01:40:59.861938 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:59.861811    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 21 01:40:59.861938 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:59.861909    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 21 01:40:59.861938 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:59.861924    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 21 01:40:59.870522 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:59.870499    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj"]
Jul 21 01:40:59.909274 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:59.909238    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/770d775f-9702-42e6-872b-fb85843c999f-manifests\") pod \"opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj\" (UID: \"770d775f-9702-42e6-872b-fb85843c999f\") " pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj"
Jul 21 01:40:59.909444 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:59.909280    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/770d775f-9702-42e6-872b-fb85843c999f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj\" (UID: \"770d775f-9702-42e6-872b-fb85843c999f\") " pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj"
Jul 21 01:40:59.909444 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:59.909310    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4dz6\" (UniqueName: \"kubernetes.io/projected/770d775f-9702-42e6-872b-fb85843c999f-kube-api-access-s4dz6\") pod \"opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj\" (UID: \"770d775f-9702-42e6-872b-fb85843c999f\") " pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj"
Jul 21 01:40:59.909444 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:59.909385    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/770d775f-9702-42e6-872b-fb85843c999f-tmp\") pod \"opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj\" (UID: \"770d775f-9702-42e6-872b-fb85843c999f\") " pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj"
Jul 21 01:40:59.909444 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:40:59.909418    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/770d775f-9702-42e6-872b-fb85843c999f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj\" (UID: \"770d775f-9702-42e6-872b-fb85843c999f\") " pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj"
Jul 21 01:41:00.010600 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:00.010566    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/770d775f-9702-42e6-872b-fb85843c999f-manifests\") pod \"opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj\" (UID: \"770d775f-9702-42e6-872b-fb85843c999f\") " pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj"
Jul 21 01:41:00.010773 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:00.010603    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/770d775f-9702-42e6-872b-fb85843c999f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj\" (UID: \"770d775f-9702-42e6-872b-fb85843c999f\") " pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj"
Jul 21 01:41:00.010773 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:00.010647    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4dz6\" (UniqueName: \"kubernetes.io/projected/770d775f-9702-42e6-872b-fb85843c999f-kube-api-access-s4dz6\") pod \"opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj\" (UID: \"770d775f-9702-42e6-872b-fb85843c999f\") " pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj"
Jul 21 01:41:00.010773 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:00.010690    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/770d775f-9702-42e6-872b-fb85843c999f-tmp\") pod \"opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj\" (UID: \"770d775f-9702-42e6-872b-fb85843c999f\") " pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj"
Jul 21 01:41:00.010773 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:00.010711    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/770d775f-9702-42e6-872b-fb85843c999f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj\" (UID: \"770d775f-9702-42e6-872b-fb85843c999f\") " pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj"
Jul 21 01:41:00.011048 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:00.011025    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/770d775f-9702-42e6-872b-fb85843c999f-manifests\") pod \"opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj\" (UID: \"770d775f-9702-42e6-872b-fb85843c999f\") " pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj"
Jul 21 01:41:00.012960 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:00.012933    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/770d775f-9702-42e6-872b-fb85843c999f-tmp\") pod \"opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj\" (UID: \"770d775f-9702-42e6-872b-fb85843c999f\") " pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj"
Jul 21 01:41:00.013164 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:00.013146    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/770d775f-9702-42e6-872b-fb85843c999f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj\" (UID: \"770d775f-9702-42e6-872b-fb85843c999f\") " pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj"
Jul 21 01:41:00.013257 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:00.013239    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/770d775f-9702-42e6-872b-fb85843c999f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj\" (UID: \"770d775f-9702-42e6-872b-fb85843c999f\") " pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj"
Jul 21 01:41:00.015584 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:00.015564    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:41:00.031948 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:00.031928    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4dz6\" (UniqueName: \"kubernetes.io/projected/770d775f-9702-42e6-872b-fb85843c999f-kube-api-access-s4dz6\") pod \"opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj\" (UID: \"770d775f-9702-42e6-872b-fb85843c999f\") " pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj"
Jul 21 01:41:00.106862 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:00.106830    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:41:00.168695 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:00.168660    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj"
Jul 21 01:41:00.206558 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:00.206527    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:41:00.281141 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:00.281105    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:41:00.297717 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:00.297697    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj"]
Jul 21 01:41:00.299651 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:41:00.299606    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod770d775f_9702_42e6_872b_fb85843c999f.slice/crio-f9a6165e80cb06d28efb161ae5f930fd53b1e3dbe3c0c1974fca4b33b1107d45 WatchSource:0}: Error finding container f9a6165e80cb06d28efb161ae5f930fd53b1e3dbe3c0c1974fca4b33b1107d45: Status 404 returned error can't find the container with id f9a6165e80cb06d28efb161ae5f930fd53b1e3dbe3c0c1974fca4b33b1107d45
Jul 21 01:41:00.355629 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:00.355581    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:41:01.305376 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:01.305339    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj" event={"ID":"770d775f-9702-42e6-872b-fb85843c999f","Type":"ContainerStarted","Data":"f9a6165e80cb06d28efb161ae5f930fd53b1e3dbe3c0c1974fca4b33b1107d45"}
Jul 21 01:41:02.887283 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:02.887236    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:41:03.316584 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:03.316547    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj" event={"ID":"770d775f-9702-42e6-872b-fb85843c999f","Type":"ContainerStarted","Data":"9f9fc221a97ce95dd64dd7b254179b50597c148356f0e5208bbf302855e442d1"}
Jul 21 01:41:04.196467 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:04.196439    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:41:04.297196 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:04.297171    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:41:05.329668 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:05.329637    2575 generic.go:356] "Generic (PLEG): container finished" podID="770d775f-9702-42e6-872b-fb85843c999f" containerID="9f9fc221a97ce95dd64dd7b254179b50597c148356f0e5208bbf302855e442d1" exitCode=0
Jul 21 01:41:05.330034 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:05.329706    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj" event={"ID":"770d775f-9702-42e6-872b-fb85843c999f","Type":"ContainerDied","Data":"9f9fc221a97ce95dd64dd7b254179b50597c148356f0e5208bbf302855e442d1"}
Jul 21 01:41:06.336200 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:06.336165    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj" event={"ID":"770d775f-9702-42e6-872b-fb85843c999f","Type":"ContainerStarted","Data":"71bee50c608473363370da0dc03d635a5abb49589d923d3daaa0afd93ebb34a5"}
Jul 21 01:41:06.336568 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:06.336377    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj"
Jul 21 01:41:06.359119 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:06.359076    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj" podStartSLOduration=3.011016895 podStartE2EDuration="7.35906153s" totalImagesPullingTime="3.309725666s" totalInitContainerRuntime="1.038318969s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:40:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:41:06.356898176 +0000 UTC m=+285.954538526" watchObservedRunningTime="2026-07-21 01:41:06.35906153 +0000 UTC m=+285.956701880"
Jul 21 01:41:17.343250 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:17.343214    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj"
Jul 21 01:41:18.768029 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:18.767991    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-95964d979-smzzg"]
Jul 21 01:41:18.770804 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:18.770786    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-95964d979-smzzg"
Jul 21 01:41:18.776349 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:18.774475    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 21 01:41:18.776349 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:18.774774    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 21 01:41:18.776349 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:18.775105    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-n59fd\""
Jul 21 01:41:18.776349 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:18.775320    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 21 01:41:18.786412 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:18.786385    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-95964d979-smzzg"]
Jul 21 01:41:18.866129 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:18.866092    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/65d1d87b-eb13-439e-801a-d663fa1c8559-manager-config\") pod \"lws-controller-manager-95964d979-smzzg\" (UID: \"65d1d87b-eb13-439e-801a-d663fa1c8559\") " pod="openshift-lws-operator/lws-controller-manager-95964d979-smzzg"
Jul 21 01:41:18.866129 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:18.866127    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/65d1d87b-eb13-439e-801a-d663fa1c8559-metrics-cert\") pod \"lws-controller-manager-95964d979-smzzg\" (UID: \"65d1d87b-eb13-439e-801a-d663fa1c8559\") " pod="openshift-lws-operator/lws-controller-manager-95964d979-smzzg"
Jul 21 01:41:18.866327 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:18.866229    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v5mz\" (UniqueName: \"kubernetes.io/projected/65d1d87b-eb13-439e-801a-d663fa1c8559-kube-api-access-6v5mz\") pod \"lws-controller-manager-95964d979-smzzg\" (UID: \"65d1d87b-eb13-439e-801a-d663fa1c8559\") " pod="openshift-lws-operator/lws-controller-manager-95964d979-smzzg"
Jul 21 01:41:18.866327 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:18.866260    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65d1d87b-eb13-439e-801a-d663fa1c8559-cert\") pod \"lws-controller-manager-95964d979-smzzg\" (UID: \"65d1d87b-eb13-439e-801a-d663fa1c8559\") " pod="openshift-lws-operator/lws-controller-manager-95964d979-smzzg"
Jul 21 01:41:18.967231 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:18.967200    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/65d1d87b-eb13-439e-801a-d663fa1c8559-manager-config\") pod \"lws-controller-manager-95964d979-smzzg\" (UID: \"65d1d87b-eb13-439e-801a-d663fa1c8559\") " pod="openshift-lws-operator/lws-controller-manager-95964d979-smzzg"
Jul 21 01:41:18.967231 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:18.967229    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/65d1d87b-eb13-439e-801a-d663fa1c8559-metrics-cert\") pod \"lws-controller-manager-95964d979-smzzg\" (UID: \"65d1d87b-eb13-439e-801a-d663fa1c8559\") " pod="openshift-lws-operator/lws-controller-manager-95964d979-smzzg"
Jul 21 01:41:18.967461 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:18.967288    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v5mz\" (UniqueName: \"kubernetes.io/projected/65d1d87b-eb13-439e-801a-d663fa1c8559-kube-api-access-6v5mz\") pod \"lws-controller-manager-95964d979-smzzg\" (UID: \"65d1d87b-eb13-439e-801a-d663fa1c8559\") " pod="openshift-lws-operator/lws-controller-manager-95964d979-smzzg"
Jul 21 01:41:18.967517 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:18.967464    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65d1d87b-eb13-439e-801a-d663fa1c8559-cert\") pod \"lws-controller-manager-95964d979-smzzg\" (UID: \"65d1d87b-eb13-439e-801a-d663fa1c8559\") " pod="openshift-lws-operator/lws-controller-manager-95964d979-smzzg"
Jul 21 01:41:18.967944 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:18.967922    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/65d1d87b-eb13-439e-801a-d663fa1c8559-manager-config\") pod \"lws-controller-manager-95964d979-smzzg\" (UID: \"65d1d87b-eb13-439e-801a-d663fa1c8559\") " pod="openshift-lws-operator/lws-controller-manager-95964d979-smzzg"
Jul 21 01:41:18.969588 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:18.969559    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/65d1d87b-eb13-439e-801a-d663fa1c8559-metrics-cert\") pod \"lws-controller-manager-95964d979-smzzg\" (UID: \"65d1d87b-eb13-439e-801a-d663fa1c8559\") " pod="openshift-lws-operator/lws-controller-manager-95964d979-smzzg"
Jul 21 01:41:18.969729 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:18.969712    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65d1d87b-eb13-439e-801a-d663fa1c8559-cert\") pod \"lws-controller-manager-95964d979-smzzg\" (UID: \"65d1d87b-eb13-439e-801a-d663fa1c8559\") " pod="openshift-lws-operator/lws-controller-manager-95964d979-smzzg"
Jul 21 01:41:18.981396 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:18.981375    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v5mz\" (UniqueName: \"kubernetes.io/projected/65d1d87b-eb13-439e-801a-d663fa1c8559-kube-api-access-6v5mz\") pod \"lws-controller-manager-95964d979-smzzg\" (UID: \"65d1d87b-eb13-439e-801a-d663fa1c8559\") " pod="openshift-lws-operator/lws-controller-manager-95964d979-smzzg"
Jul 21 01:41:19.085084 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:19.085008    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-95964d979-smzzg"
Jul 21 01:41:19.222673 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:19.222647    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-95964d979-smzzg"]
Jul 21 01:41:19.225387 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:41:19.225358    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d1d87b_eb13_439e_801a_d663fa1c8559.slice/crio-df4d3123407c07974a2e7a5a1d85fb37e2d4dc7e7d600bd49ea7a04df681982a WatchSource:0}: Error finding container df4d3123407c07974a2e7a5a1d85fb37e2d4dc7e7d600bd49ea7a04df681982a: Status 404 returned error can't find the container with id df4d3123407c07974a2e7a5a1d85fb37e2d4dc7e7d600bd49ea7a04df681982a
Jul 21 01:41:19.390537 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:19.390454    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-95964d979-smzzg" event={"ID":"65d1d87b-eb13-439e-801a-d663fa1c8559","Type":"ContainerStarted","Data":"df4d3123407c07974a2e7a5a1d85fb37e2d4dc7e7d600bd49ea7a04df681982a"}
Jul 21 01:41:19.401206 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:19.401171    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:41:19.739900 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:19.739829    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5cc474c686-5sshb"]
Jul 21 01:41:19.744413 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:19.744382    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cc474c686-5sshb"
Jul 21 01:41:19.747782 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:19.747753    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 21 01:41:19.749070 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:19.749038    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 21 01:41:19.749205 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:19.749117    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 21 01:41:19.749205 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:19.749117    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-87t7z\""
Jul 21 01:41:19.754218 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:19.754199    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 21 01:41:19.762360 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:19.762338    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cc474c686-5sshb"]
Jul 21 01:41:19.876058 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:19.876019    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17a7e457-ecc9-483f-8534-35108f24e501-tmp\") pod \"kube-auth-proxy-5cc474c686-5sshb\" (UID: \"17a7e457-ecc9-483f-8534-35108f24e501\") " pod="openshift-ingress/kube-auth-proxy-5cc474c686-5sshb"
Jul 21 01:41:19.876528 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:19.876165    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnqwq\" (UniqueName: \"kubernetes.io/projected/17a7e457-ecc9-483f-8534-35108f24e501-kube-api-access-wnqwq\") pod \"kube-auth-proxy-5cc474c686-5sshb\" (UID: \"17a7e457-ecc9-483f-8534-35108f24e501\") " pod="openshift-ingress/kube-auth-proxy-5cc474c686-5sshb"
Jul 21 01:41:19.876528 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:19.876215    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17a7e457-ecc9-483f-8534-35108f24e501-tls-certs\") pod \"kube-auth-proxy-5cc474c686-5sshb\" (UID: \"17a7e457-ecc9-483f-8534-35108f24e501\") " pod="openshift-ingress/kube-auth-proxy-5cc474c686-5sshb"
Jul 21 01:41:19.977479 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:19.977441    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17a7e457-ecc9-483f-8534-35108f24e501-tmp\") pod \"kube-auth-proxy-5cc474c686-5sshb\" (UID: \"17a7e457-ecc9-483f-8534-35108f24e501\") " pod="openshift-ingress/kube-auth-proxy-5cc474c686-5sshb"
Jul 21 01:41:19.977647 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:19.977543    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnqwq\" (UniqueName: \"kubernetes.io/projected/17a7e457-ecc9-483f-8534-35108f24e501-kube-api-access-wnqwq\") pod \"kube-auth-proxy-5cc474c686-5sshb\" (UID: \"17a7e457-ecc9-483f-8534-35108f24e501\") " pod="openshift-ingress/kube-auth-proxy-5cc474c686-5sshb"
Jul 21 01:41:19.977647 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:19.977571    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17a7e457-ecc9-483f-8534-35108f24e501-tls-certs\") pod \"kube-auth-proxy-5cc474c686-5sshb\" (UID: \"17a7e457-ecc9-483f-8534-35108f24e501\") " pod="openshift-ingress/kube-auth-proxy-5cc474c686-5sshb"
Jul 21 01:41:19.979953 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:19.979917    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17a7e457-ecc9-483f-8534-35108f24e501-tmp\") pod \"kube-auth-proxy-5cc474c686-5sshb\" (UID: \"17a7e457-ecc9-483f-8534-35108f24e501\") " pod="openshift-ingress/kube-auth-proxy-5cc474c686-5sshb"
Jul 21 01:41:19.980139 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:19.980120    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17a7e457-ecc9-483f-8534-35108f24e501-tls-certs\") pod \"kube-auth-proxy-5cc474c686-5sshb\" (UID: \"17a7e457-ecc9-483f-8534-35108f24e501\") " pod="openshift-ingress/kube-auth-proxy-5cc474c686-5sshb"
Jul 21 01:41:19.989196 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:19.989167    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnqwq\" (UniqueName: \"kubernetes.io/projected/17a7e457-ecc9-483f-8534-35108f24e501-kube-api-access-wnqwq\") pod \"kube-auth-proxy-5cc474c686-5sshb\" (UID: \"17a7e457-ecc9-483f-8534-35108f24e501\") " pod="openshift-ingress/kube-auth-proxy-5cc474c686-5sshb"
Jul 21 01:41:20.055833 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:20.055796    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cc474c686-5sshb"
Jul 21 01:41:20.212262 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:20.212208    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cc474c686-5sshb"]
Jul 21 01:41:20.214375 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:41:20.214350    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a7e457_ecc9_483f_8534_35108f24e501.slice/crio-6f6165a7c15fcdbfdbef8703ec36ba2283cafd6635ac12f3e7654581a790e22a WatchSource:0}: Error finding container 6f6165a7c15fcdbfdbef8703ec36ba2283cafd6635ac12f3e7654581a790e22a: Status 404 returned error can't find the container with id 6f6165a7c15fcdbfdbef8703ec36ba2283cafd6635ac12f3e7654581a790e22a
Jul 21 01:41:20.397372 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:20.397289    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cc474c686-5sshb" event={"ID":"17a7e457-ecc9-483f-8534-35108f24e501","Type":"ContainerStarted","Data":"6f6165a7c15fcdbfdbef8703ec36ba2283cafd6635ac12f3e7654581a790e22a"}
Jul 21 01:41:20.617958 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:20.617912    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:41:20.801436 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:20.801395    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:41:20.872757 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:20.872733    2575 kubelet.go:1648] "Image garbage collection succeeded"
Jul 21 01:41:20.991938 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:20.991858    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:41:21.227011 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:21.226980    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:41:21.297083 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:21.297034    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:41:21.370020 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:21.369993    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:41:21.404556 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:21.404520    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-95964d979-smzzg" event={"ID":"65d1d87b-eb13-439e-801a-d663fa1c8559","Type":"ContainerStarted","Data":"bc5887c66abe64fab60beaf3d3e35dd75b2a163c361420913117322da562c6fc"}
Jul 21 01:41:21.404740 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:21.404667    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-95964d979-smzzg"
Jul 21 01:41:21.421786 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:21.421736    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-95964d979-smzzg" podStartSLOduration=1.934192562 podStartE2EDuration="3.421721994s" totalImagesPullingTime="1.487529432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:41:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:41:21.419427343 +0000 UTC m=+301.017067693" watchObservedRunningTime="2026-07-21 01:41:21.421721994 +0000 UTC m=+301.019362345"
Jul 21 01:41:22.199661 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:22.199606    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:41:22.747917 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:22.747895    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 21 01:41:23.414914 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:23.414875    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cc474c686-5sshb" event={"ID":"17a7e457-ecc9-483f-8534-35108f24e501","Type":"ContainerStarted","Data":"8d3b5eaf8669bcd84a51f760bf08815f171550319d82fd457deb528aaf4821c4"}
Jul 21 01:41:23.430693 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:23.430648    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5cc474c686-5sshb" podStartSLOduration=1.901065906 podStartE2EDuration="4.430633315s" totalImagesPullingTime="2.529567409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:41:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:41:23.429062212 +0000 UTC m=+303.026702564" watchObservedRunningTime="2026-07-21 01:41:23.430633315 +0000 UTC m=+303.028273683"
Jul 21 01:41:26.739593 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:26.739546    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:41:26.796501 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:26.796467    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:41:27.322875 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:27.322843    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:41:27.595726 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:27.595647    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:41:27.797232 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:27.797198    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:41:31.311835 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:31.311779    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:41:31.417067 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:31.416994    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:41:31.515799 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:31.515736    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:41:31.635516 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:31.635439    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:41:31.870643 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:31.870598    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:41:32.410964 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:32.410920    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-95964d979-smzzg"
Jul 21 01:41:45.449107 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:45.449075    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:41:45.506860 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:41:45.506832    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:42:06.554205 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:42:06.554171    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:42:06.652254 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:42:06.652228    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:42:20.931560 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:42:20.931526    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:42:21.034134 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:42:21.034103    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:42:21.120549 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:42:21.120508    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:42:21.875682 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:42:21.875645    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:42:22.083875 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:42:22.083846    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:42:49.898252 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:42:49.898221    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:42:54.090443 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:42:54.090408    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:42:56.494086 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:42:56.494054    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:42:56.729003 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:42:56.728962    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:42:57.097471 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:42:57.097442    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:00.483939 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:00.483882    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:00.827047 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:00.827014    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:00.925594 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:00.925558    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:01.525836 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:01.525802    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:03.541287 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:03.541256    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:05.323818 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:05.323786    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:06.733024 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:06.732991    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:06.842735 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:06.842693    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:10.521395 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:10.521359    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:10.623766 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:10.623736    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:13.177321 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:13.177286    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:13.277176 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:13.277146    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:14.130903 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:14.130869    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:14.231429 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:14.231393    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:14.699384 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:14.699351    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:14.800192 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:14.800164    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:15.094179 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:15.094148    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:15.195317 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:15.195284    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:18.869362 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:18.869323    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:18.970416 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:18.970387    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:24.066113 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:24.066081    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x6grd"]
Jul 21 01:43:24.069317 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:24.069301    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x6grd"
Jul 21 01:43:24.071697 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:24.071672    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-7nz7g\""
Jul 21 01:43:24.072196 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:24.072173    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 21 01:43:24.072312 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:24.072294    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 21 01:43:24.079830 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:24.079808    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x6grd"]
Jul 21 01:43:24.223705 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:24.223673    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gmk2\" (UniqueName: \"kubernetes.io/projected/d1f95ad8-8bd7-4956-b56c-5404a04152b6-kube-api-access-6gmk2\") pod \"limitador-operator-controller-manager-859d7b78bd-x6grd\" (UID: \"d1f95ad8-8bd7-4956-b56c-5404a04152b6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x6grd"
Jul 21 01:43:24.325135 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:24.325061    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gmk2\" (UniqueName: \"kubernetes.io/projected/d1f95ad8-8bd7-4956-b56c-5404a04152b6-kube-api-access-6gmk2\") pod \"limitador-operator-controller-manager-859d7b78bd-x6grd\" (UID: \"d1f95ad8-8bd7-4956-b56c-5404a04152b6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x6grd"
Jul 21 01:43:24.332434 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:24.332415    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gmk2\" (UniqueName: \"kubernetes.io/projected/d1f95ad8-8bd7-4956-b56c-5404a04152b6-kube-api-access-6gmk2\") pod \"limitador-operator-controller-manager-859d7b78bd-x6grd\" (UID: \"d1f95ad8-8bd7-4956-b56c-5404a04152b6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x6grd"
Jul 21 01:43:24.380261 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:24.380236    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x6grd"
Jul 21 01:43:24.509194 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:24.509168    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x6grd"]
Jul 21 01:43:24.512014 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:43:24.511988    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1f95ad8_8bd7_4956_b56c_5404a04152b6.slice/crio-854402ca2a379111483c7641cf1a2d0f493f181de53aae02d36bba311d512b9a WatchSource:0}: Error finding container 854402ca2a379111483c7641cf1a2d0f493f181de53aae02d36bba311d512b9a: Status 404 returned error can't find the container with id 854402ca2a379111483c7641cf1a2d0f493f181de53aae02d36bba311d512b9a
Jul 21 01:43:24.514035 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:24.514019    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 21 01:43:24.927858 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:24.927825    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x6grd" event={"ID":"d1f95ad8-8bd7-4956-b56c-5404a04152b6","Type":"ContainerStarted","Data":"854402ca2a379111483c7641cf1a2d0f493f181de53aae02d36bba311d512b9a"}
Jul 21 01:43:25.335393 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:25.335360    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:25.541376 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:25.541334    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:25.647200 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:25.647128    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:25.744741 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:25.744702    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:26.938871 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:26.938834    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x6grd" event={"ID":"d1f95ad8-8bd7-4956-b56c-5404a04152b6","Type":"ContainerStarted","Data":"010c7799a8dd30739d16eb06739897227d8662ae179951e748ce009eda80efec"}
Jul 21 01:43:26.939241 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:26.938939    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x6grd"
Jul 21 01:43:26.954350 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:26.954301    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x6grd" podStartSLOduration=1.314723313 podStartE2EDuration="2.954288158s" totalImagesPullingTime="1.639564845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:43:26.952687292 +0000 UTC m=+426.550327642" watchObservedRunningTime="2026-07-21 01:43:26.954288158 +0000 UTC m=+426.551928508"
Jul 21 01:43:28.437339 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:28.437312    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:28.545569 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:28.545538    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:33.225992 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:33.225961    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:33.322514 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:33.322476    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:36.760958 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:36.760924    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:36.818731 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:36.818704    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:37.945562 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:37.945526    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x6grd"
Jul 21 01:43:41.084839 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:41.084803    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:43.389203 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:43.389134    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:43.488394 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:43.488364    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:46.593132 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:46.593094    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:46.693382 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:46.693350    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:51.182047 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:51.182015    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:51.282595 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:51.282565    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:51.382845 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:51.382810    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:51.435671 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:51.435591    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:56.585598 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:56.585561    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:43:56.640975 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:43:56.640943    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:44:04.746103 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:04.746064    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:44:04.807718 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:04.807690    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:44:20.939541 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:20.939509    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:44:21.038330 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:21.038293    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:44:21.138651 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:21.138606    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:44:21.687949 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:21.687911    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-gfszs"]
Jul 21 01:44:21.691169 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:21.691147    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-gfszs"
Jul 21 01:44:21.693532 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:21.693512    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-8glcl\""
Jul 21 01:44:21.697210 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:21.697180    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-gfszs"]
Jul 21 01:44:21.725797 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:21.725771    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:44:21.793075 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:21.793052    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnxq8\" (UniqueName: \"kubernetes.io/projected/b89533bc-0700-40a7-b7ef-9e104c07a5f6-kube-api-access-cnxq8\") pod \"authorino-58cd955675-gfszs\" (UID: \"b89533bc-0700-40a7-b7ef-9e104c07a5f6\") " pod="kuadrant-system/authorino-58cd955675-gfszs"
Jul 21 01:44:21.807801 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:21.807772    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-gnql4"]
Jul 21 01:44:21.810943 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:21.810929    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-gnql4"
Jul 21 01:44:21.817911 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:21.817893    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-gnql4"]
Jul 21 01:44:21.850515 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:21.850466    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:44:21.894181 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:21.894151    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnxq8\" (UniqueName: \"kubernetes.io/projected/b89533bc-0700-40a7-b7ef-9e104c07a5f6-kube-api-access-cnxq8\") pod \"authorino-58cd955675-gfszs\" (UID: \"b89533bc-0700-40a7-b7ef-9e104c07a5f6\") " pod="kuadrant-system/authorino-58cd955675-gfszs"
Jul 21 01:44:21.894280 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:21.894214    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glkzn\" (UniqueName: \"kubernetes.io/projected/4679ce98-ef83-471f-831d-80c42e278bd4-kube-api-access-glkzn\") pod \"authorino-75d556bd77-gnql4\" (UID: \"4679ce98-ef83-471f-831d-80c42e278bd4\") " pod="kuadrant-system/authorino-75d556bd77-gnql4"
Jul 21 01:44:21.902167 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:21.902142    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnxq8\" (UniqueName: \"kubernetes.io/projected/b89533bc-0700-40a7-b7ef-9e104c07a5f6-kube-api-access-cnxq8\") pod \"authorino-58cd955675-gfszs\" (UID: \"b89533bc-0700-40a7-b7ef-9e104c07a5f6\") " pod="kuadrant-system/authorino-58cd955675-gfszs"
Jul 21 01:44:21.995342 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:21.995287    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glkzn\" (UniqueName: \"kubernetes.io/projected/4679ce98-ef83-471f-831d-80c42e278bd4-kube-api-access-glkzn\") pod \"authorino-75d556bd77-gnql4\" (UID: \"4679ce98-ef83-471f-831d-80c42e278bd4\") " pod="kuadrant-system/authorino-75d556bd77-gnql4"
Jul 21 01:44:22.000348 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:22.000331    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-gfszs"
Jul 21 01:44:22.002303 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:22.002284    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glkzn\" (UniqueName: \"kubernetes.io/projected/4679ce98-ef83-471f-831d-80c42e278bd4-kube-api-access-glkzn\") pod \"authorino-75d556bd77-gnql4\" (UID: \"4679ce98-ef83-471f-831d-80c42e278bd4\") " pod="kuadrant-system/authorino-75d556bd77-gnql4"
Jul 21 01:44:22.120125 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:22.120100    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-gnql4"
Jul 21 01:44:22.127363 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:22.127339    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-gfszs"]
Jul 21 01:44:22.131355 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:44:22.131326    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb89533bc_0700_40a7_b7ef_9e104c07a5f6.slice/crio-269aee115bf29eb8f300a402327c999d2ab562b767732028f6b18da2375d78d9 WatchSource:0}: Error finding container 269aee115bf29eb8f300a402327c999d2ab562b767732028f6b18da2375d78d9: Status 404 returned error can't find the container with id 269aee115bf29eb8f300a402327c999d2ab562b767732028f6b18da2375d78d9
Jul 21 01:44:22.166170 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:22.166138    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-gfszs" event={"ID":"b89533bc-0700-40a7-b7ef-9e104c07a5f6","Type":"ContainerStarted","Data":"269aee115bf29eb8f300a402327c999d2ab562b767732028f6b18da2375d78d9"}
Jul 21 01:44:22.246231 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:22.246206    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-gnql4"]
Jul 21 01:44:22.248476 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:44:22.248449    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4679ce98_ef83_471f_831d_80c42e278bd4.slice/crio-7056d1e0ee440c2729ad8e1110556410054d4bac3e59873d7d68b19cf87387bd WatchSource:0}: Error finding container 7056d1e0ee440c2729ad8e1110556410054d4bac3e59873d7d68b19cf87387bd: Status 404 returned error can't find the container with id 7056d1e0ee440c2729ad8e1110556410054d4bac3e59873d7d68b19cf87387bd
Jul 21 01:44:23.185373 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:23.185338    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-gnql4" event={"ID":"4679ce98-ef83-471f-831d-80c42e278bd4","Type":"ContainerStarted","Data":"7056d1e0ee440c2729ad8e1110556410054d4bac3e59873d7d68b19cf87387bd"}
Jul 21 01:44:26.201606 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:26.201569    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-gnql4" event={"ID":"4679ce98-ef83-471f-831d-80c42e278bd4","Type":"ContainerStarted","Data":"325238ecce72aeb1b02f2bf851fb7b57ebdded17e6a1919598b6486d78b48400"}
Jul 21 01:44:26.202996 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:26.202970    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-gfszs" event={"ID":"b89533bc-0700-40a7-b7ef-9e104c07a5f6","Type":"ContainerStarted","Data":"736db3a1b3b1591f5488d6c90dfb10efb651334c5ba71f2b20cf7e6eb4a4c2c9"}
Jul 21 01:44:26.214899 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:26.214857    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-gnql4" podStartSLOduration=1.787895639 podStartE2EDuration="5.214843093s" totalImagesPullingTime="3.426947454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:44:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:44:26.213790806 +0000 UTC m=+485.811431157" watchObservedRunningTime="2026-07-21 01:44:26.214843093 +0000 UTC m=+485.812483443"
Jul 21 01:44:26.227383 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:26.227333    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-gfszs" podStartSLOduration=1.67324607 podStartE2EDuration="5.227318197s" totalImagesPullingTime="3.554072127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:44:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:44:26.225950827 +0000 UTC m=+485.823591178" watchObservedRunningTime="2026-07-21 01:44:26.227318197 +0000 UTC m=+485.824958549"
Jul 21 01:44:26.249133 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:26.249109    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-gfszs"]
Jul 21 01:44:28.211915 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:28.211875    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-gfszs" podUID="b89533bc-0700-40a7-b7ef-9e104c07a5f6" containerName="authorino" containerID="cri-o://736db3a1b3b1591f5488d6c90dfb10efb651334c5ba71f2b20cf7e6eb4a4c2c9" gracePeriod=30
Jul 21 01:44:28.452008 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:28.451985    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-gfszs"
Jul 21 01:44:28.552625 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:28.552591    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnxq8\" (UniqueName: \"kubernetes.io/projected/b89533bc-0700-40a7-b7ef-9e104c07a5f6-kube-api-access-cnxq8\") pod \"b89533bc-0700-40a7-b7ef-9e104c07a5f6\" (UID: \"b89533bc-0700-40a7-b7ef-9e104c07a5f6\") "
Jul 21 01:44:28.554688 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:28.554667    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b89533bc-0700-40a7-b7ef-9e104c07a5f6-kube-api-access-cnxq8" (OuterVolumeSpecName: "kube-api-access-cnxq8") pod "b89533bc-0700-40a7-b7ef-9e104c07a5f6" (UID: "b89533bc-0700-40a7-b7ef-9e104c07a5f6"). InnerVolumeSpecName "kube-api-access-cnxq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 01:44:28.653794 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:28.653769    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnxq8\" (UniqueName: \"kubernetes.io/projected/b89533bc-0700-40a7-b7ef-9e104c07a5f6-kube-api-access-cnxq8\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 01:44:29.217100 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:29.217069    2575 generic.go:356] "Generic (PLEG): container finished" podID="b89533bc-0700-40a7-b7ef-9e104c07a5f6" containerID="736db3a1b3b1591f5488d6c90dfb10efb651334c5ba71f2b20cf7e6eb4a4c2c9" exitCode=0
Jul 21 01:44:29.217453 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:29.217110    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-gfszs"
Jul 21 01:44:29.217453 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:29.217153    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-gfszs" event={"ID":"b89533bc-0700-40a7-b7ef-9e104c07a5f6","Type":"ContainerDied","Data":"736db3a1b3b1591f5488d6c90dfb10efb651334c5ba71f2b20cf7e6eb4a4c2c9"}
Jul 21 01:44:29.217453 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:29.217193    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-gfszs" event={"ID":"b89533bc-0700-40a7-b7ef-9e104c07a5f6","Type":"ContainerDied","Data":"269aee115bf29eb8f300a402327c999d2ab562b767732028f6b18da2375d78d9"}
Jul 21 01:44:29.217453 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:29.217211    2575 scope.go:117] "RemoveContainer" containerID="736db3a1b3b1591f5488d6c90dfb10efb651334c5ba71f2b20cf7e6eb4a4c2c9"
Jul 21 01:44:29.225336 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:29.225318    2575 scope.go:117] "RemoveContainer" containerID="736db3a1b3b1591f5488d6c90dfb10efb651334c5ba71f2b20cf7e6eb4a4c2c9"
Jul 21 01:44:29.225566 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:44:29.225551    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"736db3a1b3b1591f5488d6c90dfb10efb651334c5ba71f2b20cf7e6eb4a4c2c9\": container with ID starting with 736db3a1b3b1591f5488d6c90dfb10efb651334c5ba71f2b20cf7e6eb4a4c2c9 not found: ID does not exist" containerID="736db3a1b3b1591f5488d6c90dfb10efb651334c5ba71f2b20cf7e6eb4a4c2c9"
Jul 21 01:44:29.225633 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:29.225574    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"736db3a1b3b1591f5488d6c90dfb10efb651334c5ba71f2b20cf7e6eb4a4c2c9"} err="failed to get container status \"736db3a1b3b1591f5488d6c90dfb10efb651334c5ba71f2b20cf7e6eb4a4c2c9\": rpc error: code = NotFound desc = could not find container \"736db3a1b3b1591f5488d6c90dfb10efb651334c5ba71f2b20cf7e6eb4a4c2c9\": container with ID starting with 736db3a1b3b1591f5488d6c90dfb10efb651334c5ba71f2b20cf7e6eb4a4c2c9 not found: ID does not exist"
Jul 21 01:44:29.231563 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:29.231544    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-gfszs"]
Jul 21 01:44:29.237585 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:29.237564    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-gfszs"]
Jul 21 01:44:31.026810 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:31.026778    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b89533bc-0700-40a7-b7ef-9e104c07a5f6" path="/var/lib/kubelet/pods/b89533bc-0700-40a7-b7ef-9e104c07a5f6/volumes"
Jul 21 01:44:41.998024 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:41.997991    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:44:42.096059 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:42.096028    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:44:42.492729 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:42.492694    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:44:55.014331 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:55.014297    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:44:55.251524 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:55.251492    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-mlssg"]
Jul 21 01:44:55.251854 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:55.251841    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b89533bc-0700-40a7-b7ef-9e104c07a5f6" containerName="authorino"
Jul 21 01:44:55.251898 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:55.251856    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b89533bc-0700-40a7-b7ef-9e104c07a5f6" containerName="authorino"
Jul 21 01:44:55.251936 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:55.251915    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b89533bc-0700-40a7-b7ef-9e104c07a5f6" containerName="authorino"
Jul 21 01:44:55.256918 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:55.256903    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-mlssg"
Jul 21 01:44:55.263760 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:55.263730    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-mlssg"]
Jul 21 01:44:55.378535 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:55.378504    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxrg7\" (UniqueName: \"kubernetes.io/projected/3d1f187e-ae97-445c-9aab-353c0463990c-kube-api-access-pxrg7\") pod \"authorino-5d4b67c446-mlssg\" (UID: \"3d1f187e-ae97-445c-9aab-353c0463990c\") " pod="kuadrant-system/authorino-5d4b67c446-mlssg"
Jul 21 01:44:55.438449 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:55.438364    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:44:55.479579 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:55.479550    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxrg7\" (UniqueName: \"kubernetes.io/projected/3d1f187e-ae97-445c-9aab-353c0463990c-kube-api-access-pxrg7\") pod \"authorino-5d4b67c446-mlssg\" (UID: \"3d1f187e-ae97-445c-9aab-353c0463990c\") " pod="kuadrant-system/authorino-5d4b67c446-mlssg"
Jul 21 01:44:55.489935 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:55.489909    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxrg7\" (UniqueName: \"kubernetes.io/projected/3d1f187e-ae97-445c-9aab-353c0463990c-kube-api-access-pxrg7\") pod \"authorino-5d4b67c446-mlssg\" (UID: \"3d1f187e-ae97-445c-9aab-353c0463990c\") " pod="kuadrant-system/authorino-5d4b67c446-mlssg"
Jul 21 01:44:55.501724 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:55.501702    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-mlssg"]
Jul 21 01:44:55.501930 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:55.501919    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-mlssg"
Jul 21 01:44:55.527906 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:55.527884    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-78bfd94d5d-xzzf6"]
Jul 21 01:44:55.531460 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:55.531439    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78bfd94d5d-xzzf6"
Jul 21 01:44:55.539623 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:55.539586    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78bfd94d5d-xzzf6"]
Jul 21 01:44:55.629568 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:55.629546    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-mlssg"]
Jul 21 01:44:55.631870 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:44:55.631839    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d1f187e_ae97_445c_9aab_353c0463990c.slice/crio-1eb2a2cc0b78bd314923513650a317a9a30abf17ff6697b0e1779bbe824b66dc WatchSource:0}: Error finding container 1eb2a2cc0b78bd314923513650a317a9a30abf17ff6697b0e1779bbe824b66dc: Status 404 returned error can't find the container with id 1eb2a2cc0b78bd314923513650a317a9a30abf17ff6697b0e1779bbe824b66dc
Jul 21 01:44:55.681045 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:55.681018    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhn6h\" (UniqueName: \"kubernetes.io/projected/c12eb749-5085-4c35-8053-6c2176580935-kube-api-access-qhn6h\") pod \"authorino-78bfd94d5d-xzzf6\" (UID: \"c12eb749-5085-4c35-8053-6c2176580935\") " pod="kuadrant-system/authorino-78bfd94d5d-xzzf6"
Jul 21 01:44:55.766847 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:55.766822    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-78bfd94d5d-xzzf6"]
Jul 21 01:44:55.767065 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:44:55.767048    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-qhn6h], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-78bfd94d5d-xzzf6" podUID="c12eb749-5085-4c35-8053-6c2176580935"
Jul 21 01:44:55.781406 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:55.781361    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhn6h\" (UniqueName: \"kubernetes.io/projected/c12eb749-5085-4c35-8053-6c2176580935-kube-api-access-qhn6h\") pod \"authorino-78bfd94d5d-xzzf6\" (UID: \"c12eb749-5085-4c35-8053-6c2176580935\") " pod="kuadrant-system/authorino-78bfd94d5d-xzzf6"
Jul 21 01:44:55.789285 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:55.789264    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhn6h\" (UniqueName: \"kubernetes.io/projected/c12eb749-5085-4c35-8053-6c2176580935-kube-api-access-qhn6h\") pod \"authorino-78bfd94d5d-xzzf6\" (UID: \"c12eb749-5085-4c35-8053-6c2176580935\") " pod="kuadrant-system/authorino-78bfd94d5d-xzzf6"
Jul 21 01:44:55.793699 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:55.793680    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76d4889cd6-dcv8k"]
Jul 21 01:44:55.797922 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:55.797905    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76d4889cd6-dcv8k"
Jul 21 01:44:55.800014 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:55.799998    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 21 01:44:55.804749 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:55.804686    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76d4889cd6-dcv8k"]
Jul 21 01:44:55.882099 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:55.882076    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9qmq\" (UniqueName: \"kubernetes.io/projected/5d80a479-d9b9-468d-9bcf-1a3257ff92e1-kube-api-access-g9qmq\") pod \"authorino-76d4889cd6-dcv8k\" (UID: \"5d80a479-d9b9-468d-9bcf-1a3257ff92e1\") " pod="kuadrant-system/authorino-76d4889cd6-dcv8k"
Jul 21 01:44:55.882208 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:55.882130    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5d80a479-d9b9-468d-9bcf-1a3257ff92e1-tls-cert\") pod \"authorino-76d4889cd6-dcv8k\" (UID: \"5d80a479-d9b9-468d-9bcf-1a3257ff92e1\") " pod="kuadrant-system/authorino-76d4889cd6-dcv8k"
Jul 21 01:44:55.982954 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:55.982918    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9qmq\" (UniqueName: \"kubernetes.io/projected/5d80a479-d9b9-468d-9bcf-1a3257ff92e1-kube-api-access-g9qmq\") pod \"authorino-76d4889cd6-dcv8k\" (UID: \"5d80a479-d9b9-468d-9bcf-1a3257ff92e1\") " pod="kuadrant-system/authorino-76d4889cd6-dcv8k"
Jul 21 01:44:55.983130 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:55.983082    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5d80a479-d9b9-468d-9bcf-1a3257ff92e1-tls-cert\") pod \"authorino-76d4889cd6-dcv8k\" (UID: \"5d80a479-d9b9-468d-9bcf-1a3257ff92e1\") " pod="kuadrant-system/authorino-76d4889cd6-dcv8k"
Jul 21 01:44:55.985393 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:55.985375    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5d80a479-d9b9-468d-9bcf-1a3257ff92e1-tls-cert\") pod \"authorino-76d4889cd6-dcv8k\" (UID: \"5d80a479-d9b9-468d-9bcf-1a3257ff92e1\") " pod="kuadrant-system/authorino-76d4889cd6-dcv8k"
Jul 21 01:44:55.989931 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:55.989911    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9qmq\" (UniqueName: \"kubernetes.io/projected/5d80a479-d9b9-468d-9bcf-1a3257ff92e1-kube-api-access-g9qmq\") pod \"authorino-76d4889cd6-dcv8k\" (UID: \"5d80a479-d9b9-468d-9bcf-1a3257ff92e1\") " pod="kuadrant-system/authorino-76d4889cd6-dcv8k"
Jul 21 01:44:56.107497 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:56.107476    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76d4889cd6-dcv8k"
Jul 21 01:44:56.244039 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:56.244008    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76d4889cd6-dcv8k"]
Jul 21 01:44:56.247014 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:44:56.246987    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d80a479_d9b9_468d_9bcf_1a3257ff92e1.slice/crio-13b8f2e9af7b3dec3f3ecf8257f4558e459ec8e43e3a1df1247860b8f0bcfec9 WatchSource:0}: Error finding container 13b8f2e9af7b3dec3f3ecf8257f4558e459ec8e43e3a1df1247860b8f0bcfec9: Status 404 returned error can't find the container with id 13b8f2e9af7b3dec3f3ecf8257f4558e459ec8e43e3a1df1247860b8f0bcfec9
Jul 21 01:44:56.338061 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:56.337990    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-mlssg" event={"ID":"3d1f187e-ae97-445c-9aab-353c0463990c","Type":"ContainerStarted","Data":"192ba98023af45ac33f17a205c059ff35d1cf0cb381b5d43ac7c477d1e83fa4c"}
Jul 21 01:44:56.338061 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:56.338008    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-mlssg" podUID="3d1f187e-ae97-445c-9aab-353c0463990c" containerName="authorino" containerID="cri-o://192ba98023af45ac33f17a205c059ff35d1cf0cb381b5d43ac7c477d1e83fa4c" gracePeriod=30
Jul 21 01:44:56.338061 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:56.338034    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-mlssg" event={"ID":"3d1f187e-ae97-445c-9aab-353c0463990c","Type":"ContainerStarted","Data":"1eb2a2cc0b78bd314923513650a317a9a30abf17ff6697b0e1779bbe824b66dc"}
Jul 21 01:44:56.339185 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:56.339166    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78bfd94d5d-xzzf6"
Jul 21 01:44:56.339284 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:56.339189    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76d4889cd6-dcv8k" event={"ID":"5d80a479-d9b9-468d-9bcf-1a3257ff92e1","Type":"ContainerStarted","Data":"13b8f2e9af7b3dec3f3ecf8257f4558e459ec8e43e3a1df1247860b8f0bcfec9"}
Jul 21 01:44:56.347142 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:56.347126    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78bfd94d5d-xzzf6"
Jul 21 01:44:56.353168 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:56.353129    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-mlssg" podStartSLOduration=0.962380306 podStartE2EDuration="1.353116018s" totalImagesPullingTime="390.735712ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:44:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:44:56.351087095 +0000 UTC m=+515.948727439" watchObservedRunningTime="2026-07-21 01:44:56.353116018 +0000 UTC m=+515.950756368"
Jul 21 01:44:56.486656 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:56.486625    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhn6h\" (UniqueName: \"kubernetes.io/projected/c12eb749-5085-4c35-8053-6c2176580935-kube-api-access-qhn6h\") pod \"c12eb749-5085-4c35-8053-6c2176580935\" (UID: \"c12eb749-5085-4c35-8053-6c2176580935\") "
Jul 21 01:44:56.488451 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:56.488420    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c12eb749-5085-4c35-8053-6c2176580935-kube-api-access-qhn6h" (OuterVolumeSpecName: "kube-api-access-qhn6h") pod "c12eb749-5085-4c35-8053-6c2176580935" (UID: "c12eb749-5085-4c35-8053-6c2176580935"). InnerVolumeSpecName "kube-api-access-qhn6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 01:44:56.586496 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:56.586474    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-mlssg"
Jul 21 01:44:56.587642 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:56.587625    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhn6h\" (UniqueName: \"kubernetes.io/projected/c12eb749-5085-4c35-8053-6c2176580935-kube-api-access-qhn6h\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 01:44:56.688238 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:56.688210    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxrg7\" (UniqueName: \"kubernetes.io/projected/3d1f187e-ae97-445c-9aab-353c0463990c-kube-api-access-pxrg7\") pod \"3d1f187e-ae97-445c-9aab-353c0463990c\" (UID: \"3d1f187e-ae97-445c-9aab-353c0463990c\") "
Jul 21 01:44:56.690142 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:56.690109    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d1f187e-ae97-445c-9aab-353c0463990c-kube-api-access-pxrg7" (OuterVolumeSpecName: "kube-api-access-pxrg7") pod "3d1f187e-ae97-445c-9aab-353c0463990c" (UID: "3d1f187e-ae97-445c-9aab-353c0463990c"). InnerVolumeSpecName "kube-api-access-pxrg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 01:44:56.789372 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:56.789342    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxrg7\" (UniqueName: \"kubernetes.io/projected/3d1f187e-ae97-445c-9aab-353c0463990c-kube-api-access-pxrg7\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 01:44:57.345088 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:57.345001    2575 generic.go:356] "Generic (PLEG): container finished" podID="3d1f187e-ae97-445c-9aab-353c0463990c" containerID="192ba98023af45ac33f17a205c059ff35d1cf0cb381b5d43ac7c477d1e83fa4c" exitCode=0
Jul 21 01:44:57.345088 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:57.345034    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-mlssg"
Jul 21 01:44:57.345562 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:57.345081    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-mlssg" event={"ID":"3d1f187e-ae97-445c-9aab-353c0463990c","Type":"ContainerDied","Data":"192ba98023af45ac33f17a205c059ff35d1cf0cb381b5d43ac7c477d1e83fa4c"}
Jul 21 01:44:57.345562 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:57.345122    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-mlssg" event={"ID":"3d1f187e-ae97-445c-9aab-353c0463990c","Type":"ContainerDied","Data":"1eb2a2cc0b78bd314923513650a317a9a30abf17ff6697b0e1779bbe824b66dc"}
Jul 21 01:44:57.345562 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:57.345141    2575 scope.go:117] "RemoveContainer" containerID="192ba98023af45ac33f17a205c059ff35d1cf0cb381b5d43ac7c477d1e83fa4c"
Jul 21 01:44:57.346705 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:57.346682    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78bfd94d5d-xzzf6"
Jul 21 01:44:57.346705 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:57.346691    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76d4889cd6-dcv8k" event={"ID":"5d80a479-d9b9-468d-9bcf-1a3257ff92e1","Type":"ContainerStarted","Data":"523c19303c6742ee34bb6a233aed006de49853970883f89f96b52d19a32eb774"}
Jul 21 01:44:57.353480 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:57.353463    2575 scope.go:117] "RemoveContainer" containerID="192ba98023af45ac33f17a205c059ff35d1cf0cb381b5d43ac7c477d1e83fa4c"
Jul 21 01:44:57.353739 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:44:57.353722    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"192ba98023af45ac33f17a205c059ff35d1cf0cb381b5d43ac7c477d1e83fa4c\": container with ID starting with 192ba98023af45ac33f17a205c059ff35d1cf0cb381b5d43ac7c477d1e83fa4c not found: ID does not exist" containerID="192ba98023af45ac33f17a205c059ff35d1cf0cb381b5d43ac7c477d1e83fa4c"
Jul 21 01:44:57.353796 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:57.353745    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"192ba98023af45ac33f17a205c059ff35d1cf0cb381b5d43ac7c477d1e83fa4c"} err="failed to get container status \"192ba98023af45ac33f17a205c059ff35d1cf0cb381b5d43ac7c477d1e83fa4c\": rpc error: code = NotFound desc = could not find container \"192ba98023af45ac33f17a205c059ff35d1cf0cb381b5d43ac7c477d1e83fa4c\": container with ID starting with 192ba98023af45ac33f17a205c059ff35d1cf0cb381b5d43ac7c477d1e83fa4c not found: ID does not exist"
Jul 21 01:44:57.363848 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:57.363809    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-76d4889cd6-dcv8k" podStartSLOduration=1.9383437749999999 podStartE2EDuration="2.363797595s" totalImagesPullingTime="425.45382ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:44:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:44:57.363143866 +0000 UTC m=+516.960784278" watchObservedRunningTime="2026-07-21 01:44:57.363797595 +0000 UTC m=+516.961437945"
Jul 21 01:44:57.379291 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:57.379263    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-mlssg"]
Jul 21 01:44:57.382638 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:57.382594    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-mlssg"]
Jul 21 01:44:57.385542 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:57.385517    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-gnql4"]
Jul 21 01:44:57.385791 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:57.385770    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-gnql4" podUID="4679ce98-ef83-471f-831d-80c42e278bd4" containerName="authorino" containerID="cri-o://325238ecce72aeb1b02f2bf851fb7b57ebdded17e6a1919598b6486d78b48400" gracePeriod=30
Jul 21 01:44:57.404717 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:57.404685    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-78bfd94d5d-xzzf6"]
Jul 21 01:44:57.409684 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:57.409657    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-78bfd94d5d-xzzf6"]
Jul 21 01:44:57.634873 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:57.634846    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-gnql4"
Jul 21 01:44:57.797946 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:57.797912    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glkzn\" (UniqueName: \"kubernetes.io/projected/4679ce98-ef83-471f-831d-80c42e278bd4-kube-api-access-glkzn\") pod \"4679ce98-ef83-471f-831d-80c42e278bd4\" (UID: \"4679ce98-ef83-471f-831d-80c42e278bd4\") "
Jul 21 01:44:57.799889 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:57.799864    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4679ce98-ef83-471f-831d-80c42e278bd4-kube-api-access-glkzn" (OuterVolumeSpecName: "kube-api-access-glkzn") pod "4679ce98-ef83-471f-831d-80c42e278bd4" (UID: "4679ce98-ef83-471f-831d-80c42e278bd4"). InnerVolumeSpecName "kube-api-access-glkzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 01:44:57.899453 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:57.899395    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glkzn\" (UniqueName: \"kubernetes.io/projected/4679ce98-ef83-471f-831d-80c42e278bd4-kube-api-access-glkzn\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 01:44:58.353066 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:58.353027    2575 generic.go:356] "Generic (PLEG): container finished" podID="4679ce98-ef83-471f-831d-80c42e278bd4" containerID="325238ecce72aeb1b02f2bf851fb7b57ebdded17e6a1919598b6486d78b48400" exitCode=0
Jul 21 01:44:58.353451 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:58.353072    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-gnql4"
Jul 21 01:44:58.353451 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:58.353091    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-gnql4" event={"ID":"4679ce98-ef83-471f-831d-80c42e278bd4","Type":"ContainerDied","Data":"325238ecce72aeb1b02f2bf851fb7b57ebdded17e6a1919598b6486d78b48400"}
Jul 21 01:44:58.353451 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:58.353129    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-gnql4" event={"ID":"4679ce98-ef83-471f-831d-80c42e278bd4","Type":"ContainerDied","Data":"7056d1e0ee440c2729ad8e1110556410054d4bac3e59873d7d68b19cf87387bd"}
Jul 21 01:44:58.353451 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:58.353147    2575 scope.go:117] "RemoveContainer" containerID="325238ecce72aeb1b02f2bf851fb7b57ebdded17e6a1919598b6486d78b48400"
Jul 21 01:44:58.361564 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:58.361546    2575 scope.go:117] "RemoveContainer" containerID="325238ecce72aeb1b02f2bf851fb7b57ebdded17e6a1919598b6486d78b48400"
Jul 21 01:44:58.361823 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:44:58.361803    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"325238ecce72aeb1b02f2bf851fb7b57ebdded17e6a1919598b6486d78b48400\": container with ID starting with 325238ecce72aeb1b02f2bf851fb7b57ebdded17e6a1919598b6486d78b48400 not found: ID does not exist" containerID="325238ecce72aeb1b02f2bf851fb7b57ebdded17e6a1919598b6486d78b48400"
Jul 21 01:44:58.361879 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:58.361831    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"325238ecce72aeb1b02f2bf851fb7b57ebdded17e6a1919598b6486d78b48400"} err="failed to get container status \"325238ecce72aeb1b02f2bf851fb7b57ebdded17e6a1919598b6486d78b48400\": rpc error: code = NotFound desc = could not find container \"325238ecce72aeb1b02f2bf851fb7b57ebdded17e6a1919598b6486d78b48400\": container with ID starting with 325238ecce72aeb1b02f2bf851fb7b57ebdded17e6a1919598b6486d78b48400 not found: ID does not exist"
Jul 21 01:44:58.372141 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:58.372123    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-gnql4"]
Jul 21 01:44:58.374166 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:58.374148    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-gnql4"]
Jul 21 01:44:59.025671 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:59.025639    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d1f187e-ae97-445c-9aab-353c0463990c" path="/var/lib/kubelet/pods/3d1f187e-ae97-445c-9aab-353c0463990c/volumes"
Jul 21 01:44:59.025946 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:59.025934    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4679ce98-ef83-471f-831d-80c42e278bd4" path="/var/lib/kubelet/pods/4679ce98-ef83-471f-831d-80c42e278bd4/volumes"
Jul 21 01:44:59.026203 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:44:59.026193    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c12eb749-5085-4c35-8053-6c2176580935" path="/var/lib/kubelet/pods/c12eb749-5085-4c35-8053-6c2176580935/volumes"
Jul 21 01:45:02.208937 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:45:02.208909    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:45:02.386590 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:45:02.386554    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:45:02.448342 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:45:02.448312    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:45:05.643074 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:45:05.643043    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:45:05.743004 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:45:05.742973    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:45:37.228920 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:45:37.228877    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:45:37.327503 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:45:37.327462    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:45:37.781271 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:45:37.781238    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:45:37.990264 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:45:37.990221    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:45:38.369707 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:45:38.369663    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:45:38.462357 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:45:38.462325    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:45:48.644058 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:45:48.644020    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:45:48.742869 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:45:48.742833    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:45:55.702136 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:45:55.702106    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:46:07.037860 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:07.037804    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:46:14.505508 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:14.505477    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-mhlj6"]
Jul 21 01:46:14.505991 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:14.505822    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d1f187e-ae97-445c-9aab-353c0463990c" containerName="authorino"
Jul 21 01:46:14.505991 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:14.505834    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d1f187e-ae97-445c-9aab-353c0463990c" containerName="authorino"
Jul 21 01:46:14.505991 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:14.505850    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4679ce98-ef83-471f-831d-80c42e278bd4" containerName="authorino"
Jul 21 01:46:14.505991 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:14.505856    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4679ce98-ef83-471f-831d-80c42e278bd4" containerName="authorino"
Jul 21 01:46:14.505991 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:14.505914    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4679ce98-ef83-471f-831d-80c42e278bd4" containerName="authorino"
Jul 21 01:46:14.505991 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:14.505922    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d1f187e-ae97-445c-9aab-353c0463990c" containerName="authorino"
Jul 21 01:46:14.508825 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:14.508809    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mhlj6"
Jul 21 01:46:14.511230 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:14.511210    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 21 01:46:14.512020 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:14.511994    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 21 01:46:14.512134 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:14.511997    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 21 01:46:14.512134 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:14.512005    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-fqk5w\""
Jul 21 01:46:14.516982 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:14.516959    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-mhlj6"]
Jul 21 01:46:14.535285 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:14.535260    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mhlj6\" (UID: \"b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mhlj6"
Jul 21 01:46:14.535380 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:14.535298    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvczf\" (UniqueName: \"kubernetes.io/projected/b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d-kube-api-access-qvczf\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mhlj6\" (UID: \"b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mhlj6"
Jul 21 01:46:14.535380 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:14.535370    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mhlj6\" (UID: \"b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mhlj6"
Jul 21 01:46:14.535487 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:14.535423    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mhlj6\" (UID: \"b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mhlj6"
Jul 21 01:46:14.535487 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:14.535455    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mhlj6\" (UID: \"b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mhlj6"
Jul 21 01:46:14.535549 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:14.535499    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mhlj6\" (UID: \"b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mhlj6"
Jul 21 01:46:14.536278 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:14.536258    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:46:14.636417 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:14.636385    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mhlj6\" (UID: \"b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mhlj6"
Jul 21 01:46:14.636548 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:14.636434    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mhlj6\" (UID: \"b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mhlj6"
Jul 21 01:46:14.636548 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:14.636451    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvczf\" (UniqueName: \"kubernetes.io/projected/b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d-kube-api-access-qvczf\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mhlj6\" (UID: \"b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mhlj6"
Jul 21 01:46:14.636548 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:14.636489    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mhlj6\" (UID: \"b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mhlj6"
Jul 21 01:46:14.636548 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:14.636509    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mhlj6\" (UID: \"b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mhlj6"
Jul 21 01:46:14.636548 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:14.636528    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mhlj6\" (UID: \"b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mhlj6"
Jul 21 01:46:14.636886 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:14.636864    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mhlj6\" (UID: \"b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mhlj6"
Jul 21 01:46:14.636928 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:14.636893    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mhlj6\" (UID: \"b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mhlj6"
Jul 21 01:46:14.636969 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:14.636931    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mhlj6\" (UID: \"b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mhlj6"
Jul 21 01:46:14.638751 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:14.638726    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mhlj6\" (UID: \"b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mhlj6"
Jul 21 01:46:14.639023 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:14.639008    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mhlj6\" (UID: \"b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mhlj6"
Jul 21 01:46:14.643800 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:14.643783    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvczf\" (UniqueName: \"kubernetes.io/projected/b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d-kube-api-access-qvczf\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mhlj6\" (UID: \"b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mhlj6"
Jul 21 01:46:14.821103 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:14.821069    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mhlj6"
Jul 21 01:46:14.944636 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:14.944600    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-mhlj6"]
Jul 21 01:46:14.947213 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:46:14.947176    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5cbb5c8_4f1b_4bf1_b323_d9b3f783df1d.slice/crio-3943d754110b3c08fc6c04ed9bc0bcf5f61dcfe0f70459a94f438dc2a6d3ea99 WatchSource:0}: Error finding container 3943d754110b3c08fc6c04ed9bc0bcf5f61dcfe0f70459a94f438dc2a6d3ea99: Status 404 returned error can't find the container with id 3943d754110b3c08fc6c04ed9bc0bcf5f61dcfe0f70459a94f438dc2a6d3ea99
Jul 21 01:46:15.688316 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:15.688279    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mhlj6" event={"ID":"b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d","Type":"ContainerStarted","Data":"3943d754110b3c08fc6c04ed9bc0bcf5f61dcfe0f70459a94f438dc2a6d3ea99"}
Jul 21 01:46:16.694341 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:16.694311    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mhlj6" event={"ID":"b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d","Type":"ContainerStarted","Data":"8a5c774057740d044e0f29cb007b35216f6d764f0a72365218c24668f7ae3692"}
Jul 21 01:46:16.711021 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:16.710352    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mhlj6" podStartSLOduration=1.599992287 podStartE2EDuration="2.710332708s" totalImagesPullingTime="1.110340421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:46:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:46:16.70989438 +0000 UTC m=+596.307534732" watchObservedRunningTime="2026-07-21 01:46:16.710332708 +0000 UTC m=+596.307973060"
Jul 21 01:46:23.862913 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:23.862880    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:46:24.821686 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:24.821655    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mhlj6"
Jul 21 01:46:24.821893 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:24.821738    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mhlj6"
Jul 21 01:46:24.834111 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:24.834088    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mhlj6"
Jul 21 01:46:25.054179 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:25.054144    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:46:25.750517 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:25.750492    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mhlj6"
Jul 21 01:46:30.117396 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:30.117363    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:46:37.594201 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:37.594163    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv"]
Jul 21 01:46:37.598176 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:37.598155    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv"
Jul 21 01:46:37.600473 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:37.600452    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 21 01:46:37.605786 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:37.605760    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv"]
Jul 21 01:46:37.628354 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:37.628330    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:46:37.718676 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:37.718645    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8db08254-ea0e-4d01-9951-60cbefb18d27-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv\" (UID: \"8db08254-ea0e-4d01-9951-60cbefb18d27\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv"
Jul 21 01:46:37.718676 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:37.718678    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8db08254-ea0e-4d01-9951-60cbefb18d27-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv\" (UID: \"8db08254-ea0e-4d01-9951-60cbefb18d27\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv"
Jul 21 01:46:37.718872 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:37.718706    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pnfg\" (UniqueName: \"kubernetes.io/projected/8db08254-ea0e-4d01-9951-60cbefb18d27-kube-api-access-4pnfg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv\" (UID: \"8db08254-ea0e-4d01-9951-60cbefb18d27\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv"
Jul 21 01:46:37.718872 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:37.718788    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8db08254-ea0e-4d01-9951-60cbefb18d27-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv\" (UID: \"8db08254-ea0e-4d01-9951-60cbefb18d27\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv"
Jul 21 01:46:37.718872 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:37.718855    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8db08254-ea0e-4d01-9951-60cbefb18d27-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv\" (UID: \"8db08254-ea0e-4d01-9951-60cbefb18d27\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv"
Jul 21 01:46:37.718977 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:37.718916    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8db08254-ea0e-4d01-9951-60cbefb18d27-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv\" (UID: \"8db08254-ea0e-4d01-9951-60cbefb18d27\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv"
Jul 21 01:46:37.820154 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:37.820124    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8db08254-ea0e-4d01-9951-60cbefb18d27-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv\" (UID: \"8db08254-ea0e-4d01-9951-60cbefb18d27\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv"
Jul 21 01:46:37.820341 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:37.820174    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8db08254-ea0e-4d01-9951-60cbefb18d27-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv\" (UID: \"8db08254-ea0e-4d01-9951-60cbefb18d27\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv"
Jul 21 01:46:37.820341 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:37.820208    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8db08254-ea0e-4d01-9951-60cbefb18d27-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv\" (UID: \"8db08254-ea0e-4d01-9951-60cbefb18d27\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv"
Jul 21 01:46:37.820341 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:37.820234    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8db08254-ea0e-4d01-9951-60cbefb18d27-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv\" (UID: \"8db08254-ea0e-4d01-9951-60cbefb18d27\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv"
Jul 21 01:46:37.820341 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:37.820255    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pnfg\" (UniqueName: \"kubernetes.io/projected/8db08254-ea0e-4d01-9951-60cbefb18d27-kube-api-access-4pnfg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv\" (UID: \"8db08254-ea0e-4d01-9951-60cbefb18d27\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv"
Jul 21 01:46:37.820551 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:37.820352    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8db08254-ea0e-4d01-9951-60cbefb18d27-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv\" (UID: \"8db08254-ea0e-4d01-9951-60cbefb18d27\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv"
Jul 21 01:46:37.820776 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:37.820750    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8db08254-ea0e-4d01-9951-60cbefb18d27-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv\" (UID: \"8db08254-ea0e-4d01-9951-60cbefb18d27\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv"
Jul 21 01:46:37.820898 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:37.820779    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8db08254-ea0e-4d01-9951-60cbefb18d27-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv\" (UID: \"8db08254-ea0e-4d01-9951-60cbefb18d27\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv"
Jul 21 01:46:37.820898 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:37.820803    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8db08254-ea0e-4d01-9951-60cbefb18d27-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv\" (UID: \"8db08254-ea0e-4d01-9951-60cbefb18d27\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv"
Jul 21 01:46:37.822457 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:37.822433    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8db08254-ea0e-4d01-9951-60cbefb18d27-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv\" (UID: \"8db08254-ea0e-4d01-9951-60cbefb18d27\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv"
Jul 21 01:46:37.822765 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:37.822746    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8db08254-ea0e-4d01-9951-60cbefb18d27-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv\" (UID: \"8db08254-ea0e-4d01-9951-60cbefb18d27\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv"
Jul 21 01:46:37.829134 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:37.829109    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pnfg\" (UniqueName: \"kubernetes.io/projected/8db08254-ea0e-4d01-9951-60cbefb18d27-kube-api-access-4pnfg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv\" (UID: \"8db08254-ea0e-4d01-9951-60cbefb18d27\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv"
Jul 21 01:46:37.910024 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:37.909954    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv"
Jul 21 01:46:38.045316 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:38.045290    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv"]
Jul 21 01:46:38.047414 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:46:38.047378    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8db08254_ea0e_4d01_9951_60cbefb18d27.slice/crio-f68866c84bbe4135ede08d34262ac6b59c26f5ab0018b75c30b0ce7a66bbef4f WatchSource:0}: Error finding container f68866c84bbe4135ede08d34262ac6b59c26f5ab0018b75c30b0ce7a66bbef4f: Status 404 returned error can't find the container with id f68866c84bbe4135ede08d34262ac6b59c26f5ab0018b75c30b0ce7a66bbef4f
Jul 21 01:46:38.799067 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:38.799033    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv" event={"ID":"8db08254-ea0e-4d01-9951-60cbefb18d27","Type":"ContainerStarted","Data":"59642f49522d29893b189cc002e87c5bc1850b55ab9b7f199cd408231aecfc6c"}
Jul 21 01:46:38.799436 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:38.799085    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv" event={"ID":"8db08254-ea0e-4d01-9951-60cbefb18d27","Type":"ContainerStarted","Data":"f68866c84bbe4135ede08d34262ac6b59c26f5ab0018b75c30b0ce7a66bbef4f"}
Jul 21 01:46:38.815730 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:38.815687    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv" podStartSLOduration=1.636809218 podStartE2EDuration="1.815673446s" totalImagesPullingTime="178.864228ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:46:38.814418213 +0000 UTC m=+618.412058577" watchObservedRunningTime="2026-07-21 01:46:38.815673446 +0000 UTC m=+618.413313798"
Jul 21 01:46:40.889941 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:40.889903    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:46:45.395154 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:45.395078    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:46:47.910225 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:47.910192    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv"
Jul 21 01:46:47.910686 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:47.910251    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv"
Jul 21 01:46:47.922602 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:47.922578    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv"
Jul 21 01:46:48.854530 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:46:48.854500    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv"
Jul 21 01:47:08.597530 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:08.597492    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:47:50.751912 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:50.751882    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-795ddbf4cf-d5bzl"]
Jul 21 01:47:50.755086 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:50.755070    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-795ddbf4cf-d5bzl"
Jul 21 01:47:50.763890 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:50.763865    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-795ddbf4cf-d5bzl"]
Jul 21 01:47:50.773067 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:50.773038    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/baf68c78-4944-4caa-8584-9cba4c511ea9-tls-cert\") pod \"authorino-795ddbf4cf-d5bzl\" (UID: \"baf68c78-4944-4caa-8584-9cba4c511ea9\") " pod="kuadrant-system/authorino-795ddbf4cf-d5bzl"
Jul 21 01:47:50.773166 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:50.773110    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjtqm\" (UniqueName: \"kubernetes.io/projected/baf68c78-4944-4caa-8584-9cba4c511ea9-kube-api-access-tjtqm\") pod \"authorino-795ddbf4cf-d5bzl\" (UID: \"baf68c78-4944-4caa-8584-9cba4c511ea9\") " pod="kuadrant-system/authorino-795ddbf4cf-d5bzl"
Jul 21 01:47:50.873522 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:50.873494    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/baf68c78-4944-4caa-8584-9cba4c511ea9-tls-cert\") pod \"authorino-795ddbf4cf-d5bzl\" (UID: \"baf68c78-4944-4caa-8584-9cba4c511ea9\") " pod="kuadrant-system/authorino-795ddbf4cf-d5bzl"
Jul 21 01:47:50.873673 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:50.873538    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjtqm\" (UniqueName: \"kubernetes.io/projected/baf68c78-4944-4caa-8584-9cba4c511ea9-kube-api-access-tjtqm\") pod \"authorino-795ddbf4cf-d5bzl\" (UID: \"baf68c78-4944-4caa-8584-9cba4c511ea9\") " pod="kuadrant-system/authorino-795ddbf4cf-d5bzl"
Jul 21 01:47:50.875961 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:50.875940    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/baf68c78-4944-4caa-8584-9cba4c511ea9-tls-cert\") pod \"authorino-795ddbf4cf-d5bzl\" (UID: \"baf68c78-4944-4caa-8584-9cba4c511ea9\") " pod="kuadrant-system/authorino-795ddbf4cf-d5bzl"
Jul 21 01:47:50.880900 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:50.880878    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjtqm\" (UniqueName: \"kubernetes.io/projected/baf68c78-4944-4caa-8584-9cba4c511ea9-kube-api-access-tjtqm\") pod \"authorino-795ddbf4cf-d5bzl\" (UID: \"baf68c78-4944-4caa-8584-9cba4c511ea9\") " pod="kuadrant-system/authorino-795ddbf4cf-d5bzl"
Jul 21 01:47:51.064383 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:51.064355    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-795ddbf4cf-d5bzl"
Jul 21 01:47:51.145100 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:51.145060    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:47:51.198522 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:51.198475    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-795ddbf4cf-d5bzl"]
Jul 21 01:47:51.200923 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:47:51.200893    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbaf68c78_4944_4caa_8584_9cba4c511ea9.slice/crio-8ab10c0e5d70f8e783005b46391d51f2fc7df18406d23e48ceec3514feefab6a WatchSource:0}: Error finding container 8ab10c0e5d70f8e783005b46391d51f2fc7df18406d23e48ceec3514feefab6a: Status 404 returned error can't find the container with id 8ab10c0e5d70f8e783005b46391d51f2fc7df18406d23e48ceec3514feefab6a
Jul 21 01:47:51.209084 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:51.209062    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:47:51.356501 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:51.356409    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:47:52.141472 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:52.141432    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-795ddbf4cf-d5bzl" event={"ID":"baf68c78-4944-4caa-8584-9cba4c511ea9","Type":"ContainerStarted","Data":"9e6a0f001e589c8a7d0a087269999e10d4f7092fe283929e5bdfa08628784b4b"}
Jul 21 01:47:52.141992 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:52.141482    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-795ddbf4cf-d5bzl" event={"ID":"baf68c78-4944-4caa-8584-9cba4c511ea9","Type":"ContainerStarted","Data":"8ab10c0e5d70f8e783005b46391d51f2fc7df18406d23e48ceec3514feefab6a"}
Jul 21 01:47:52.156931 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:52.156878    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-795ddbf4cf-d5bzl" podStartSLOduration=1.648027363 podStartE2EDuration="2.156860637s" totalImagesPullingTime="508.833274ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:47:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:47:52.15548358 +0000 UTC m=+691.753123932" watchObservedRunningTime="2026-07-21 01:47:52.156860637 +0000 UTC m=+691.754500988"
Jul 21 01:47:52.189558 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:52.189522    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76d4889cd6-dcv8k"]
Jul 21 01:47:52.189770 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:52.189742    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-76d4889cd6-dcv8k" podUID="5d80a479-d9b9-468d-9bcf-1a3257ff92e1" containerName="authorino" containerID="cri-o://523c19303c6742ee34bb6a233aed006de49853970883f89f96b52d19a32eb774" gracePeriod=30
Jul 21 01:47:52.359950 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:52.359911    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:47:52.441486 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:52.441458    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:47:52.452741 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:52.452711    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76d4889cd6-dcv8k"
Jul 21 01:47:52.485706 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:52.485680    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9qmq\" (UniqueName: \"kubernetes.io/projected/5d80a479-d9b9-468d-9bcf-1a3257ff92e1-kube-api-access-g9qmq\") pod \"5d80a479-d9b9-468d-9bcf-1a3257ff92e1\" (UID: \"5d80a479-d9b9-468d-9bcf-1a3257ff92e1\") "
Jul 21 01:47:52.485869 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:52.485722    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5d80a479-d9b9-468d-9bcf-1a3257ff92e1-tls-cert\") pod \"5d80a479-d9b9-468d-9bcf-1a3257ff92e1\" (UID: \"5d80a479-d9b9-468d-9bcf-1a3257ff92e1\") "
Jul 21 01:47:52.487825 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:52.487794    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d80a479-d9b9-468d-9bcf-1a3257ff92e1-kube-api-access-g9qmq" (OuterVolumeSpecName: "kube-api-access-g9qmq") pod "5d80a479-d9b9-468d-9bcf-1a3257ff92e1" (UID: "5d80a479-d9b9-468d-9bcf-1a3257ff92e1"). InnerVolumeSpecName "kube-api-access-g9qmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 01:47:52.496133 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:52.496102    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d80a479-d9b9-468d-9bcf-1a3257ff92e1-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "5d80a479-d9b9-468d-9bcf-1a3257ff92e1" (UID: "5d80a479-d9b9-468d-9bcf-1a3257ff92e1"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 01:47:52.586574 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:52.586542    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9qmq\" (UniqueName: \"kubernetes.io/projected/5d80a479-d9b9-468d-9bcf-1a3257ff92e1-kube-api-access-g9qmq\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 01:47:52.586574 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:52.586570    2575 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5d80a479-d9b9-468d-9bcf-1a3257ff92e1-tls-cert\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 01:47:53.147722 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:53.147606    2575 generic.go:356] "Generic (PLEG): container finished" podID="5d80a479-d9b9-468d-9bcf-1a3257ff92e1" containerID="523c19303c6742ee34bb6a233aed006de49853970883f89f96b52d19a32eb774" exitCode=0
Jul 21 01:47:53.147722 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:53.147659    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76d4889cd6-dcv8k"
Jul 21 01:47:53.147722 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:53.147690    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76d4889cd6-dcv8k" event={"ID":"5d80a479-d9b9-468d-9bcf-1a3257ff92e1","Type":"ContainerDied","Data":"523c19303c6742ee34bb6a233aed006de49853970883f89f96b52d19a32eb774"}
Jul 21 01:47:53.147722 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:53.147731    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76d4889cd6-dcv8k" event={"ID":"5d80a479-d9b9-468d-9bcf-1a3257ff92e1","Type":"ContainerDied","Data":"13b8f2e9af7b3dec3f3ecf8257f4558e459ec8e43e3a1df1247860b8f0bcfec9"}
Jul 21 01:47:53.148259 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:53.147750    2575 scope.go:117] "RemoveContainer" containerID="523c19303c6742ee34bb6a233aed006de49853970883f89f96b52d19a32eb774"
Jul 21 01:47:53.157047 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:53.157030    2575 scope.go:117] "RemoveContainer" containerID="523c19303c6742ee34bb6a233aed006de49853970883f89f96b52d19a32eb774"
Jul 21 01:47:53.157323 ip-10-0-129-219 kubenswrapper[2575]: E0721 01:47:53.157303    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"523c19303c6742ee34bb6a233aed006de49853970883f89f96b52d19a32eb774\": container with ID starting with 523c19303c6742ee34bb6a233aed006de49853970883f89f96b52d19a32eb774 not found: ID does not exist" containerID="523c19303c6742ee34bb6a233aed006de49853970883f89f96b52d19a32eb774"
Jul 21 01:47:53.157368 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:53.157333    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"523c19303c6742ee34bb6a233aed006de49853970883f89f96b52d19a32eb774"} err="failed to get container status \"523c19303c6742ee34bb6a233aed006de49853970883f89f96b52d19a32eb774\": rpc error: code = NotFound desc = could not find container \"523c19303c6742ee34bb6a233aed006de49853970883f89f96b52d19a32eb774\": container with ID starting with 523c19303c6742ee34bb6a233aed006de49853970883f89f96b52d19a32eb774 not found: ID does not exist"
Jul 21 01:47:53.163269 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:53.163247    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76d4889cd6-dcv8k"]
Jul 21 01:47:53.167282 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:53.167260    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76d4889cd6-dcv8k"]
Jul 21 01:47:55.025858 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:47:55.025827    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d80a479-d9b9-468d-9bcf-1a3257ff92e1" path="/var/lib/kubelet/pods/5d80a479-d9b9-468d-9bcf-1a3257ff92e1/volumes"
Jul 21 01:48:16.811740 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:48:16.811700    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:48:16.862985 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:48:16.862953    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:49:57.914944 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:49:57.914863    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:49:59.150219 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:49:59.150185    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:50:27.824481 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:50:27.824439    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:50:27.922186 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:50:27.922152    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:50:28.048452 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:50:28.048420    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:50:33.710470 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:50:33.710436    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:50:33.831574 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:50:33.831537    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:50:33.983457 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:50:33.983368    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:50:34.103588 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:50:34.103549    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:50:39.700083 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:50:39.700046    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:50:39.800233 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:50:39.800197    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:50:54.354942 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:50:54.354903    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:50:54.455711 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:50:54.455676    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:53:27.643461 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:53:27.643425    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:55:02.011813 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:55:02.011770    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:55:02.098154 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:55:02.098121    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:56:00.449113 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:56:00.449061    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:58:30.796177 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:30.796140    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj"]
Jul 21 01:58:30.796590 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:30.796513    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d80a479-d9b9-468d-9bcf-1a3257ff92e1" containerName="authorino"
Jul 21 01:58:30.796590 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:30.796523    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d80a479-d9b9-468d-9bcf-1a3257ff92e1" containerName="authorino"
Jul 21 01:58:30.796590 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:30.796587    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d80a479-d9b9-468d-9bcf-1a3257ff92e1" containerName="authorino"
Jul 21 01:58:30.799511 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:30.799495    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj"
Jul 21 01:58:30.801855 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:30.801829    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-c8ffc4b3-kserve-self-signed-certs\""
Jul 21 01:58:30.823280 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:30.823255    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj"]
Jul 21 01:58:30.859264 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:30.859240    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:58:30.903050 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:30.903026    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-home\") pod \"e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj\" (UID: \"de8e0f18-e011-4f0d-86b6-10c3f3e3a2df\") " pod="llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj"
Jul 21 01:58:30.903212 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:30.903065    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-dshm\") pod \"e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj\" (UID: \"de8e0f18-e011-4f0d-86b6-10c3f3e3a2df\") " pod="llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj"
Jul 21 01:58:30.903212 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:30.903083    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tdzx\" (UniqueName: \"kubernetes.io/projected/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-kube-api-access-6tdzx\") pod \"e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj\" (UID: \"de8e0f18-e011-4f0d-86b6-10c3f3e3a2df\") " pod="llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj"
Jul 21 01:58:30.903212 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:30.903154    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-tmp-dir\") pod \"e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj\" (UID: \"de8e0f18-e011-4f0d-86b6-10c3f3e3a2df\") " pod="llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj"
Jul 21 01:58:30.903212 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:30.903176    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-tls-certs\") pod \"e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj\" (UID: \"de8e0f18-e011-4f0d-86b6-10c3f3e3a2df\") " pod="llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj"
Jul 21 01:58:30.903381 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:30.903238    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-model-cache\") pod \"e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj\" (UID: \"de8e0f18-e011-4f0d-86b6-10c3f3e3a2df\") " pod="llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj"
Jul 21 01:58:31.004481 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:31.004450    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-model-cache\") pod \"e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj\" (UID: \"de8e0f18-e011-4f0d-86b6-10c3f3e3a2df\") " pod="llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj"
Jul 21 01:58:31.004649 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:31.004518    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-home\") pod \"e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj\" (UID: \"de8e0f18-e011-4f0d-86b6-10c3f3e3a2df\") " pod="llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj"
Jul 21 01:58:31.004649 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:31.004564    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-dshm\") pod \"e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj\" (UID: \"de8e0f18-e011-4f0d-86b6-10c3f3e3a2df\") " pod="llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj"
Jul 21 01:58:31.004649 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:31.004589    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tdzx\" (UniqueName: \"kubernetes.io/projected/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-kube-api-access-6tdzx\") pod \"e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj\" (UID: \"de8e0f18-e011-4f0d-86b6-10c3f3e3a2df\") " pod="llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj"
Jul 21 01:58:31.004799 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:31.004659    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-tmp-dir\") pod \"e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj\" (UID: \"de8e0f18-e011-4f0d-86b6-10c3f3e3a2df\") " pod="llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj"
Jul 21 01:58:31.004799 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:31.004682    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-tls-certs\") pod \"e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj\" (UID: \"de8e0f18-e011-4f0d-86b6-10c3f3e3a2df\") " pod="llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj"
Jul 21 01:58:31.004892 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:31.004856    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-model-cache\") pod \"e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj\" (UID: \"de8e0f18-e011-4f0d-86b6-10c3f3e3a2df\") " pod="llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj"
Jul 21 01:58:31.004944 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:31.004923    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-home\") pod \"e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj\" (UID: \"de8e0f18-e011-4f0d-86b6-10c3f3e3a2df\") " pod="llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj"
Jul 21 01:58:31.004991 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:31.004956    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-tmp-dir\") pod \"e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj\" (UID: \"de8e0f18-e011-4f0d-86b6-10c3f3e3a2df\") " pod="llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj"
Jul 21 01:58:31.006848 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:31.006819    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-dshm\") pod \"e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj\" (UID: \"de8e0f18-e011-4f0d-86b6-10c3f3e3a2df\") " pod="llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj"
Jul 21 01:58:31.007096 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:31.007078    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-tls-certs\") pod \"e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj\" (UID: \"de8e0f18-e011-4f0d-86b6-10c3f3e3a2df\") " pod="llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj"
Jul 21 01:58:31.011961 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:31.011943    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tdzx\" (UniqueName: \"kubernetes.io/projected/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-kube-api-access-6tdzx\") pod \"e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj\" (UID: \"de8e0f18-e011-4f0d-86b6-10c3f3e3a2df\") " pod="llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj"
Jul 21 01:58:31.108035 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:31.107975    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj"
Jul 21 01:58:31.233828 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:31.233796    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj"]
Jul 21 01:58:31.237112 ip-10-0-129-219 kubenswrapper[2575]: W0721 01:58:31.237084    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde8e0f18_e011_4f0d_86b6_10c3f3e3a2df.slice/crio-398053c610e27f983b4be4f6021db381caf7c5945dd17c79c0e1d15b52e4bd74 WatchSource:0}: Error finding container 398053c610e27f983b4be4f6021db381caf7c5945dd17c79c0e1d15b52e4bd74: Status 404 returned error can't find the container with id 398053c610e27f983b4be4f6021db381caf7c5945dd17c79c0e1d15b52e4bd74
Jul 21 01:58:31.960572 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:31.960537    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj" event={"ID":"de8e0f18-e011-4f0d-86b6-10c3f3e3a2df","Type":"ContainerStarted","Data":"efa098ad5955638a42674a000f381ea734b3374db12c4f55cb31a507ef05f4f9"}
Jul 21 01:58:31.960975 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:31.960580    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj" event={"ID":"de8e0f18-e011-4f0d-86b6-10c3f3e3a2df","Type":"ContainerStarted","Data":"398053c610e27f983b4be4f6021db381caf7c5945dd17c79c0e1d15b52e4bd74"}
Jul 21 01:58:31.975987 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:31.975946    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj" podStartSLOduration=1.975931147 podStartE2EDuration="1.975931147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:58:31.975302215 +0000 UTC m=+1331.572942568" watchObservedRunningTime="2026-07-21 01:58:31.975931147 +0000 UTC m=+1331.573571499"
Jul 21 01:58:41.108833 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:41.108742    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj"
Jul 21 01:58:41.108833 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:41.108790    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj"
Jul 21 01:58:41.121270 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:41.121240    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj"
Jul 21 01:58:42.015873 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:42.015796    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj"
Jul 21 01:58:48.113440 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:58:48.113407    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:59:02.456450 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:59:02.456415    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:59:14.022289 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:59:14.022260    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj"]
Jul 21 01:59:14.045328 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:59:14.045298    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:59:15.954353 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:59:15.954320    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 01:59:29.042548 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:59:29.042489    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj" podUID="de8e0f18-e011-4f0d-86b6-10c3f3e3a2df" containerName="main" containerID="cri-o://efa098ad5955638a42674a000f381ea734b3374db12c4f55cb31a507ef05f4f9" gracePeriod=45
Jul 21 01:59:29.213454 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:59:29.213345    2575 generic.go:356] "Generic (PLEG): container finished" podID="de8e0f18-e011-4f0d-86b6-10c3f3e3a2df" containerID="efa098ad5955638a42674a000f381ea734b3374db12c4f55cb31a507ef05f4f9" exitCode=0
Jul 21 01:59:29.213454 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:59:29.213384    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj" event={"ID":"de8e0f18-e011-4f0d-86b6-10c3f3e3a2df","Type":"ContainerDied","Data":"efa098ad5955638a42674a000f381ea734b3374db12c4f55cb31a507ef05f4f9"}
Jul 21 01:59:29.277455 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:59:29.277432    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj"
Jul 21 01:59:29.319058 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:59:29.318996    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-tls-certs\") pod \"de8e0f18-e011-4f0d-86b6-10c3f3e3a2df\" (UID: \"de8e0f18-e011-4f0d-86b6-10c3f3e3a2df\") "
Jul 21 01:59:29.319058 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:59:29.319052    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tdzx\" (UniqueName: \"kubernetes.io/projected/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-kube-api-access-6tdzx\") pod \"de8e0f18-e011-4f0d-86b6-10c3f3e3a2df\" (UID: \"de8e0f18-e011-4f0d-86b6-10c3f3e3a2df\") "
Jul 21 01:59:29.319243 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:59:29.319087    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-dshm\") pod \"de8e0f18-e011-4f0d-86b6-10c3f3e3a2df\" (UID: \"de8e0f18-e011-4f0d-86b6-10c3f3e3a2df\") "
Jul 21 01:59:29.319243 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:59:29.319109    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-home\") pod \"de8e0f18-e011-4f0d-86b6-10c3f3e3a2df\" (UID: \"de8e0f18-e011-4f0d-86b6-10c3f3e3a2df\") "
Jul 21 01:59:29.319243 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:59:29.319135    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-model-cache\") pod \"de8e0f18-e011-4f0d-86b6-10c3f3e3a2df\" (UID: \"de8e0f18-e011-4f0d-86b6-10c3f3e3a2df\") "
Jul 21 01:59:29.319243 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:59:29.319194    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-tmp-dir\") pod \"de8e0f18-e011-4f0d-86b6-10c3f3e3a2df\" (UID: \"de8e0f18-e011-4f0d-86b6-10c3f3e3a2df\") "
Jul 21 01:59:29.319436 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:59:29.319406    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-home" (OuterVolumeSpecName: "home") pod "de8e0f18-e011-4f0d-86b6-10c3f3e3a2df" (UID: "de8e0f18-e011-4f0d-86b6-10c3f3e3a2df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 01:59:29.319537 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:59:29.319511    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-model-cache" (OuterVolumeSpecName: "model-cache") pod "de8e0f18-e011-4f0d-86b6-10c3f3e3a2df" (UID: "de8e0f18-e011-4f0d-86b6-10c3f3e3a2df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 01:59:29.319537 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:59:29.319527    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "de8e0f18-e011-4f0d-86b6-10c3f3e3a2df" (UID: "de8e0f18-e011-4f0d-86b6-10c3f3e3a2df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 01:59:29.321178 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:59:29.321154    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "de8e0f18-e011-4f0d-86b6-10c3f3e3a2df" (UID: "de8e0f18-e011-4f0d-86b6-10c3f3e3a2df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 01:59:29.321178 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:59:29.321172    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-dshm" (OuterVolumeSpecName: "dshm") pod "de8e0f18-e011-4f0d-86b6-10c3f3e3a2df" (UID: "de8e0f18-e011-4f0d-86b6-10c3f3e3a2df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 01:59:29.321311 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:59:29.321225    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-kube-api-access-6tdzx" (OuterVolumeSpecName: "kube-api-access-6tdzx") pod "de8e0f18-e011-4f0d-86b6-10c3f3e3a2df" (UID: "de8e0f18-e011-4f0d-86b6-10c3f3e3a2df"). InnerVolumeSpecName "kube-api-access-6tdzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 01:59:29.420023 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:59:29.419997    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-tmp-dir\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 01:59:29.420023 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:59:29.420019    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-tls-certs\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 01:59:29.420023 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:59:29.420028    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tdzx\" (UniqueName: \"kubernetes.io/projected/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-kube-api-access-6tdzx\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 01:59:29.420189 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:59:29.420037    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-dshm\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 01:59:29.420189 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:59:29.420045    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-home\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 01:59:29.420189 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:59:29.420052    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df-model-cache\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 01:59:30.219455 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:59:30.219407    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj" event={"ID":"de8e0f18-e011-4f0d-86b6-10c3f3e3a2df","Type":"ContainerDied","Data":"398053c610e27f983b4be4f6021db381caf7c5945dd17c79c0e1d15b52e4bd74"}
Jul 21 01:59:30.219455 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:59:30.219433    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj"
Jul 21 01:59:30.219994 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:59:30.219472    2575 scope.go:117] "RemoveContainer" containerID="efa098ad5955638a42674a000f381ea734b3374db12c4f55cb31a507ef05f4f9"
Jul 21 01:59:30.241454 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:59:30.241424    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj"]
Jul 21 01:59:30.244484 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:59:30.244464    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-c8ffc4b3-kserve-5887bd656c-nxbvj"]
Jul 21 01:59:31.025588 ip-10-0-129-219 kubenswrapper[2575]: I0721 01:59:31.025553    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de8e0f18-e011-4f0d-86b6-10c3f3e3a2df" path="/var/lib/kubelet/pods/de8e0f18-e011-4f0d-86b6-10c3f3e3a2df/volumes"
Jul 21 02:00:00.417222 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:00:00.417192    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:00:00.517174 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:00:00.517119    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:02:30.613801 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:02:30.613762    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:02:37.196744 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:02:37.196704    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:02:45.468937 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:02:45.468900    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:02:50.780206 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:02:50.780166    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:02:53.883014 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:02:53.882957    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:02:53.982894 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:02:53.982860    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:02:54.342146 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:02:54.342112    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:03:00.742463 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:03:00.742430    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:03:00.842429 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:03:00.842398    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:03:01.167983 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:03:01.167952    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:03:01.330968 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:03:01.330939    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:03:01.447987 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:03:01.447899    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:03:03.953183 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:03:03.953151    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:03:04.045593 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:03:04.045557    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:03:04.114967 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:03:04.114937    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:03:04.250642 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:03:04.250546    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:03:04.548937 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:03:04.548902    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:03:04.649839 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:03:04.649808    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:03:10.184529 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:03:10.184490    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:03:10.287557 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:03:10.287525    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:03:40.157268 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:03:40.157232    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:03:44.691479 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:03:44.691445    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:03:44.857104 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:03:44.857070    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:03:45.100695 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:03:45.100657    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:03:45.384271 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:03:45.384189    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:03:45.471919 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:03:45.471888    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:03:45.592059 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:03:45.592028    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:03:46.090070 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:03:46.090041    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:03:46.223962 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:03:46.223925    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:03:48.352583 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:03:48.352552    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:03:48.493125 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:03:48.493084    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:03:48.946399 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:03:48.946364    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:03:49.047786 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:03:49.047756    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:04:13.391774 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:13.391736    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-63c3423d-6jktp"]
Jul 21 02:04:13.392184 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:13.392092    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de8e0f18-e011-4f0d-86b6-10c3f3e3a2df" containerName="main"
Jul 21 02:04:13.392184 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:13.392104    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="de8e0f18-e011-4f0d-86b6-10c3f3e3a2df" containerName="main"
Jul 21 02:04:13.392184 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:13.392177    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="de8e0f18-e011-4f0d-86b6-10c3f3e3a2df" containerName="main"
Jul 21 02:04:13.395116 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:13.395097    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-63c3423d-6jktp"
Jul 21 02:04:13.397664 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:13.397640    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 21 02:04:13.397788 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:13.397675    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x582z\""
Jul 21 02:04:13.402413 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:13.402386    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-63c3423d-6jktp"]
Jul 21 02:04:13.585499 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:13.585459    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/810b4c6d-0d66-4641-aa79-c76bfb5fbe85-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-63c3423d-6jktp\" (UID: \"810b4c6d-0d66-4641-aa79-c76bfb5fbe85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-63c3423d-6jktp"
Jul 21 02:04:13.686857 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:13.686755    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/810b4c6d-0d66-4641-aa79-c76bfb5fbe85-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-63c3423d-6jktp\" (UID: \"810b4c6d-0d66-4641-aa79-c76bfb5fbe85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-63c3423d-6jktp"
Jul 21 02:04:13.687381 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:13.687362    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/810b4c6d-0d66-4641-aa79-c76bfb5fbe85-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-63c3423d-6jktp\" (UID: \"810b4c6d-0d66-4641-aa79-c76bfb5fbe85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-63c3423d-6jktp"
Jul 21 02:04:13.706746 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:13.706721    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-63c3423d-6jktp"
Jul 21 02:04:13.829335 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:13.829313    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-63c3423d-6jktp"]
Jul 21 02:04:13.831711 ip-10-0-129-219 kubenswrapper[2575]: W0721 02:04:13.831678    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod810b4c6d_0d66_4641_aa79_c76bfb5fbe85.slice/crio-8c176509e3fdb4ea0ec043d4cd4897f4ca1aeb5524bc9830e138ae1328f98010 WatchSource:0}: Error finding container 8c176509e3fdb4ea0ec043d4cd4897f4ca1aeb5524bc9830e138ae1328f98010: Status 404 returned error can't find the container with id 8c176509e3fdb4ea0ec043d4cd4897f4ca1aeb5524bc9830e138ae1328f98010
Jul 21 02:04:13.833511 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:13.833491    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 21 02:04:14.485349 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:14.485309    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-63c3423d-6jktp" event={"ID":"810b4c6d-0d66-4641-aa79-c76bfb5fbe85","Type":"ContainerStarted","Data":"8c176509e3fdb4ea0ec043d4cd4897f4ca1aeb5524bc9830e138ae1328f98010"}
Jul 21 02:04:16.496920 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:16.496837    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-63c3423d-6jktp" event={"ID":"810b4c6d-0d66-4641-aa79-c76bfb5fbe85","Type":"ContainerStarted","Data":"8cb5fadf6ff64992f11158ea5c5de4622da316b70042db83ab5103afdc64e534"}
Jul 21 02:04:16.519768 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:16.519719    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-63c3423d-6jktp" podStartSLOduration=1.257213298 podStartE2EDuration="3.519704048s" totalImagesPullingTime="2.26249075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:04:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:04:16.517457748 +0000 UTC m=+1676.115098100" watchObservedRunningTime="2026-07-21 02:04:16.519704048 +0000 UTC m=+1676.117344400"
Jul 21 02:04:17.502934 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:17.502900    2575 generic.go:356] "Generic (PLEG): container finished" podID="810b4c6d-0d66-4641-aa79-c76bfb5fbe85" containerID="8cb5fadf6ff64992f11158ea5c5de4622da316b70042db83ab5103afdc64e534" exitCode=0
Jul 21 02:04:17.503318 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:17.502972    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-63c3423d-6jktp" event={"ID":"810b4c6d-0d66-4641-aa79-c76bfb5fbe85","Type":"ContainerDied","Data":"8cb5fadf6ff64992f11158ea5c5de4622da316b70042db83ab5103afdc64e534"}
Jul 21 02:04:18.635928 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:18.635899    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-63c3423d-6jktp"
Jul 21 02:04:18.728269 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:18.728239    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/810b4c6d-0d66-4641-aa79-c76bfb5fbe85-maas-api-service-ca\") pod \"810b4c6d-0d66-4641-aa79-c76bfb5fbe85\" (UID: \"810b4c6d-0d66-4641-aa79-c76bfb5fbe85\") "
Jul 21 02:04:18.728571 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:18.728548    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/810b4c6d-0d66-4641-aa79-c76bfb5fbe85-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "810b4c6d-0d66-4641-aa79-c76bfb5fbe85" (UID: "810b4c6d-0d66-4641-aa79-c76bfb5fbe85"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 02:04:18.828908 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:18.828878    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/810b4c6d-0d66-4641-aa79-c76bfb5fbe85-maas-api-service-ca\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:04:19.513573 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:19.513537    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-63c3423d-6jktp" event={"ID":"810b4c6d-0d66-4641-aa79-c76bfb5fbe85","Type":"ContainerDied","Data":"8c176509e3fdb4ea0ec043d4cd4897f4ca1aeb5524bc9830e138ae1328f98010"}
Jul 21 02:04:19.513573 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:19.513563    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-63c3423d-6jktp"
Jul 21 02:04:19.513811 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:19.513587    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c176509e3fdb4ea0ec043d4cd4897f4ca1aeb5524bc9830e138ae1328f98010"
Jul 21 02:04:23.832548 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:23.832507    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:04:23.966876 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:23.966834    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:04:28.462252 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:28.462214    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-63c3423d-6jktp"]
Jul 21 02:04:28.464163 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:28.464137    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-63c3423d-6jktp"]
Jul 21 02:04:28.465418 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:28.465393    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:04:28.630223 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:28.630184    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:04:28.885039 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:28.884995    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:04:29.025375 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:29.025339    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="810b4c6d-0d66-4641-aa79-c76bfb5fbe85" path="/var/lib/kubelet/pods/810b4c6d-0d66-4641-aa79-c76bfb5fbe85/volumes"
Jul 21 02:04:29.146742 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:29.146659    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:04:29.477323 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:29.477255    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:04:29.770482 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:29.770443    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:04:31.399625 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:31.399585    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:04:31.499819 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:31.499785    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:04:31.638855 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:31.638824    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:04:31.897908 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:31.897875    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:04:32.076698 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:32.076668    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:04:32.175831 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:32.175744    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:04:57.297669 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:57.297565    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-352dc19c-zxvds"]
Jul 21 02:04:57.298143 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:57.297983    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="810b4c6d-0d66-4641-aa79-c76bfb5fbe85" containerName="revoke-keys"
Jul 21 02:04:57.298143 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:57.297995    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="810b4c6d-0d66-4641-aa79-c76bfb5fbe85" containerName="revoke-keys"
Jul 21 02:04:57.298143 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:57.298057    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="810b4c6d-0d66-4641-aa79-c76bfb5fbe85" containerName="revoke-keys"
Jul 21 02:04:57.301029 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:57.301012    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-352dc19c-zxvds"
Jul 21 02:04:57.303659 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:57.303635    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 21 02:04:57.303659 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:57.303652    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x582z\""
Jul 21 02:04:57.307684 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:57.307663    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-352dc19c-zxvds"]
Jul 21 02:04:57.446553 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:57.446499    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d6cc957-61d3-4294-b534-b16da18a25d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-352dc19c-zxvds\" (UID: \"3d6cc957-61d3-4294-b534-b16da18a25d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-352dc19c-zxvds"
Jul 21 02:04:57.547537 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:57.547499    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d6cc957-61d3-4294-b534-b16da18a25d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-352dc19c-zxvds\" (UID: \"3d6cc957-61d3-4294-b534-b16da18a25d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-352dc19c-zxvds"
Jul 21 02:04:57.548154 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:57.548109    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d6cc957-61d3-4294-b534-b16da18a25d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-352dc19c-zxvds\" (UID: \"3d6cc957-61d3-4294-b534-b16da18a25d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-352dc19c-zxvds"
Jul 21 02:04:57.612558 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:57.612526    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-352dc19c-zxvds"
Jul 21 02:04:57.743629 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:57.743566    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-352dc19c-zxvds"]
Jul 21 02:04:57.746374 ip-10-0-129-219 kubenswrapper[2575]: W0721 02:04:57.746348    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d6cc957_61d3_4294_b534_b16da18a25d5.slice/crio-40b0c24261d6e9aeba59d97e9e31c08310c82c6d254f763666e1c57501ba489f WatchSource:0}: Error finding container 40b0c24261d6e9aeba59d97e9e31c08310c82c6d254f763666e1c57501ba489f: Status 404 returned error can't find the container with id 40b0c24261d6e9aeba59d97e9e31c08310c82c6d254f763666e1c57501ba489f
Jul 21 02:04:58.690549 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:58.690457    2575 generic.go:356] "Generic (PLEG): container finished" podID="3d6cc957-61d3-4294-b534-b16da18a25d5" containerID="1c1dd4ae911d4fc2e2a1261595fe59abc9eed435d123c7aa0634614cc2964095" exitCode=0
Jul 21 02:04:58.690925 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:58.690540    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-352dc19c-zxvds" event={"ID":"3d6cc957-61d3-4294-b534-b16da18a25d5","Type":"ContainerDied","Data":"1c1dd4ae911d4fc2e2a1261595fe59abc9eed435d123c7aa0634614cc2964095"}
Jul 21 02:04:58.690925 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:58.690598    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-352dc19c-zxvds" event={"ID":"3d6cc957-61d3-4294-b534-b16da18a25d5","Type":"ContainerStarted","Data":"40b0c24261d6e9aeba59d97e9e31c08310c82c6d254f763666e1c57501ba489f"}
Jul 21 02:04:59.823795 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:59.823771    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-352dc19c-zxvds"
Jul 21 02:04:59.971151 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:59.971036    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d6cc957-61d3-4294-b534-b16da18a25d5-maas-api-service-ca\") pod \"3d6cc957-61d3-4294-b534-b16da18a25d5\" (UID: \"3d6cc957-61d3-4294-b534-b16da18a25d5\") "
Jul 21 02:04:59.971404 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:04:59.971380    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d6cc957-61d3-4294-b534-b16da18a25d5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3d6cc957-61d3-4294-b534-b16da18a25d5" (UID: "3d6cc957-61d3-4294-b534-b16da18a25d5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 02:05:00.072504 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:00.072466    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d6cc957-61d3-4294-b534-b16da18a25d5-maas-api-service-ca\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:05:00.700711 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:00.700677    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-352dc19c-zxvds"
Jul 21 02:05:00.700943 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:00.700680    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-352dc19c-zxvds" event={"ID":"3d6cc957-61d3-4294-b534-b16da18a25d5","Type":"ContainerDied","Data":"40b0c24261d6e9aeba59d97e9e31c08310c82c6d254f763666e1c57501ba489f"}
Jul 21 02:05:00.700943 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:00.700799    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40b0c24261d6e9aeba59d97e9e31c08310c82c6d254f763666e1c57501ba489f"
Jul 21 02:05:07.743989 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:07.743954    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:05:12.363107 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:12.363066    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-352dc19c-zxvds"]
Jul 21 02:05:12.366054 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:12.366027    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-352dc19c-zxvds"]
Jul 21 02:05:12.367974 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:12.367948    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:05:13.025917 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:13.025884    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d6cc957-61d3-4294-b534-b16da18a25d5" path="/var/lib/kubelet/pods/3d6cc957-61d3-4294-b534-b16da18a25d5/volumes"
Jul 21 02:05:13.611102 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:13.611068    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:05:13.833136 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:13.833097    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:05:14.057283 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:14.057255    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:05:14.409738 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:14.409651    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:05:14.532233 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:14.532199    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:05:16.636818 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:16.636789    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:05:16.709284 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:16.709250    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:05:16.777359 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:16.777323    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:05:17.035481 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:17.035448    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:05:17.321721 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:17.321637    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:05:17.422235 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:17.422202    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:05:20.135401 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:20.135364    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:05:20.235370 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:20.235336    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:05:29.777853 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:29.777820    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:05:29.830733 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:29.830707    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:05:38.084142 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:38.084109    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:05:38.394929 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:38.394845    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:05:41.548388 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:41.548355    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8d4fc276-zrjgx"]
Jul 21 02:05:41.548857 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:41.548721    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d6cc957-61d3-4294-b534-b16da18a25d5" containerName="revoke-keys"
Jul 21 02:05:41.548857 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:41.548733    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d6cc957-61d3-4294-b534-b16da18a25d5" containerName="revoke-keys"
Jul 21 02:05:41.548857 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:41.548805    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d6cc957-61d3-4294-b534-b16da18a25d5" containerName="revoke-keys"
Jul 21 02:05:41.551912 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:41.551895    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8d4fc276-zrjgx"
Jul 21 02:05:41.554211 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:41.554189    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 21 02:05:41.554369 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:41.554213    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x582z\""
Jul 21 02:05:41.558911 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:41.558846    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8d4fc276-zrjgx"]
Jul 21 02:05:41.636553 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:41.636519    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22bc8fce-da42-462f-a7ee-8eeaec90f5c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-8d4fc276-zrjgx\" (UID: \"22bc8fce-da42-462f-a7ee-8eeaec90f5c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8d4fc276-zrjgx"
Jul 21 02:05:41.737323 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:41.737284    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22bc8fce-da42-462f-a7ee-8eeaec90f5c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-8d4fc276-zrjgx\" (UID: \"22bc8fce-da42-462f-a7ee-8eeaec90f5c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8d4fc276-zrjgx"
Jul 21 02:05:41.737904 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:41.737881    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22bc8fce-da42-462f-a7ee-8eeaec90f5c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-8d4fc276-zrjgx\" (UID: \"22bc8fce-da42-462f-a7ee-8eeaec90f5c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8d4fc276-zrjgx"
Jul 21 02:05:41.863578 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:41.863477    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8d4fc276-zrjgx"
Jul 21 02:05:41.990131 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:41.990105    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8d4fc276-zrjgx"]
Jul 21 02:05:41.992343 ip-10-0-129-219 kubenswrapper[2575]: W0721 02:05:41.992313    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22bc8fce_da42_462f_a7ee_8eeaec90f5c2.slice/crio-be2dc200671059af75af1b116e35e1eabdcb119b5d8c1dbe598b47764dc4aafb WatchSource:0}: Error finding container be2dc200671059af75af1b116e35e1eabdcb119b5d8c1dbe598b47764dc4aafb: Status 404 returned error can't find the container with id be2dc200671059af75af1b116e35e1eabdcb119b5d8c1dbe598b47764dc4aafb
Jul 21 02:05:42.894704 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:42.894668    2575 generic.go:356] "Generic (PLEG): container finished" podID="22bc8fce-da42-462f-a7ee-8eeaec90f5c2" containerID="4bced88bb29cc701d50ba855c49714272bceaf189bdcac753524b88bc2047618" exitCode=0
Jul 21 02:05:42.895068 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:42.894751    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8d4fc276-zrjgx" event={"ID":"22bc8fce-da42-462f-a7ee-8eeaec90f5c2","Type":"ContainerDied","Data":"4bced88bb29cc701d50ba855c49714272bceaf189bdcac753524b88bc2047618"}
Jul 21 02:05:42.895068 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:42.894805    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8d4fc276-zrjgx" event={"ID":"22bc8fce-da42-462f-a7ee-8eeaec90f5c2","Type":"ContainerStarted","Data":"be2dc200671059af75af1b116e35e1eabdcb119b5d8c1dbe598b47764dc4aafb"}
Jul 21 02:05:44.041437 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:44.041405    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8d4fc276-zrjgx"
Jul 21 02:05:44.159628 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:44.159546    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22bc8fce-da42-462f-a7ee-8eeaec90f5c2-maas-api-service-ca\") pod \"22bc8fce-da42-462f-a7ee-8eeaec90f5c2\" (UID: \"22bc8fce-da42-462f-a7ee-8eeaec90f5c2\") "
Jul 21 02:05:44.159892 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:44.159870    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22bc8fce-da42-462f-a7ee-8eeaec90f5c2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "22bc8fce-da42-462f-a7ee-8eeaec90f5c2" (UID: "22bc8fce-da42-462f-a7ee-8eeaec90f5c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 02:05:44.260425 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:44.260382    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22bc8fce-da42-462f-a7ee-8eeaec90f5c2-maas-api-service-ca\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:05:44.906311 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:44.906281    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8d4fc276-zrjgx"
Jul 21 02:05:44.906481 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:44.906280    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8d4fc276-zrjgx" event={"ID":"22bc8fce-da42-462f-a7ee-8eeaec90f5c2","Type":"ContainerDied","Data":"be2dc200671059af75af1b116e35e1eabdcb119b5d8c1dbe598b47764dc4aafb"}
Jul 21 02:05:44.906481 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:44.906416    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be2dc200671059af75af1b116e35e1eabdcb119b5d8c1dbe598b47764dc4aafb"
Jul 21 02:05:50.920856 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:50.920823    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:05:52.017991 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:52.017957    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:05:56.625258 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:56.625219    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8d4fc276-zrjgx"]
Jul 21 02:05:56.627567 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:56.627542    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8d4fc276-zrjgx"]
Jul 21 02:05:56.628533 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:56.628509    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:05:56.798569 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:56.798542    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:05:57.025698 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:57.025669    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22bc8fce-da42-462f-a7ee-8eeaec90f5c2" path="/var/lib/kubelet/pods/22bc8fce-da42-462f-a7ee-8eeaec90f5c2/volumes"
Jul 21 02:05:57.256299 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:57.256268    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:05:57.527763 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:57.527736    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:05:57.625264 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:57.625223    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:05:57.850722 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:57.850634    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:05:59.189362 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:05:59.189315    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:06:05.507915 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:06:05.507885    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:06:05.610962 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:06:05.610920    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:06:05.708600 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:06:05.708548    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:06:05.808273 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:06:05.808232    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:06:11.366011 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:06:11.365962    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:06:37.358033 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:06:37.357996    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:06:43.649406 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:06:43.649365    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:06:43.750191 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:06:43.750157    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:06:43.851623 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:06:43.851574    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:06:59.791075 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:06:59.791035    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:07:06.010180 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:06.010146    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:07:06.117424 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:06.117386    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:07:06.223909 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:06.223872    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:07:16.759628 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:16.759582    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:07:17.590982 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:17.590950    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:07:24.533255 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:24.533220    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:07:24.638594 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:24.638561    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:07:24.740359 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:24.740320    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:07:24.838003 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:24.837920    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:07:25.608442 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:25.608408    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:07:25.708182 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:25.708147    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:07:25.807780 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:25.807746    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:07:27.433581 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:27.433544    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:07:33.709212 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:33.709179    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:07:33.807310 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:33.807268    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:07:34.455909 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:34.455876    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:07:41.678059 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:41.678027    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:07:41.779320 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:41.779291    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:07:41.952910 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:41.952813    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:07:42.307985 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:42.307959    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:07:42.771804 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:42.771771    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:07:45.219513 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:45.219478    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:07:45.309687 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:45.309659    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:07:45.412284 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:45.412256    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:07:45.522408 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:45.522383    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:07:45.993181 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:45.993102    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:07:46.092014 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:46.091978    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:07:49.811381 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:49.811346    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:07:50.501677 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:50.501644    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:07:50.667018 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:50.666981    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:07:50.975391 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:50.975354    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:07:57.424170 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:57.424136    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:07:57.526713 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:57.526682    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:07:57.790593 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:57.790564    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1e560e69-gqxts"]
Jul 21 02:07:57.790929 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:57.790917    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22bc8fce-da42-462f-a7ee-8eeaec90f5c2" containerName="revoke-keys"
Jul 21 02:07:57.790973 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:57.790931    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="22bc8fce-da42-462f-a7ee-8eeaec90f5c2" containerName="revoke-keys"
Jul 21 02:07:57.791008 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:57.790999    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="22bc8fce-da42-462f-a7ee-8eeaec90f5c2" containerName="revoke-keys"
Jul 21 02:07:57.793973 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:57.793957    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1e560e69-gqxts"
Jul 21 02:07:57.796585 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:57.796565    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x582z\""
Jul 21 02:07:57.796883 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:57.796869    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 21 02:07:57.807313 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:57.807288    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1e560e69-gqxts"]
Jul 21 02:07:57.913785 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:57.913750    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e1bb63-15ea-4707-bfa3-48dc71aa24a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-1e560e69-gqxts\" (UID: \"33e1bb63-15ea-4707-bfa3-48dc71aa24a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1e560e69-gqxts"
Jul 21 02:07:58.014473 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:58.014441    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e1bb63-15ea-4707-bfa3-48dc71aa24a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-1e560e69-gqxts\" (UID: \"33e1bb63-15ea-4707-bfa3-48dc71aa24a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1e560e69-gqxts"
Jul 21 02:07:58.015101 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:58.015081    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e1bb63-15ea-4707-bfa3-48dc71aa24a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-1e560e69-gqxts\" (UID: \"33e1bb63-15ea-4707-bfa3-48dc71aa24a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1e560e69-gqxts"
Jul 21 02:07:58.104104 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:58.104002    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1e560e69-gqxts"
Jul 21 02:07:58.464676 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:58.464569    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1e560e69-gqxts"]
Jul 21 02:07:58.468474 ip-10-0-129-219 kubenswrapper[2575]: W0721 02:07:58.468430    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e1bb63_15ea_4707_bfa3_48dc71aa24a4.slice/crio-6cde47cc9f7d1098ed581406fdb6c88e10648dae3db871f73783ed3b7e05cc77 WatchSource:0}: Error finding container 6cde47cc9f7d1098ed581406fdb6c88e10648dae3db871f73783ed3b7e05cc77: Status 404 returned error can't find the container with id 6cde47cc9f7d1098ed581406fdb6c88e10648dae3db871f73783ed3b7e05cc77
Jul 21 02:07:58.524432 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:58.524397    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1e560e69-gqxts" event={"ID":"33e1bb63-15ea-4707-bfa3-48dc71aa24a4","Type":"ContainerStarted","Data":"6cde47cc9f7d1098ed581406fdb6c88e10648dae3db871f73783ed3b7e05cc77"}
Jul 21 02:07:59.529744 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:59.529710    2575 generic.go:356] "Generic (PLEG): container finished" podID="33e1bb63-15ea-4707-bfa3-48dc71aa24a4" containerID="9ad2cb93ef72ab5d66ef8f8f6cca4c4520c75d19f0caf74d5732b6c5ae3dc8a2" exitCode=0
Jul 21 02:07:59.530106 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:07:59.529768    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1e560e69-gqxts" event={"ID":"33e1bb63-15ea-4707-bfa3-48dc71aa24a4","Type":"ContainerDied","Data":"9ad2cb93ef72ab5d66ef8f8f6cca4c4520c75d19f0caf74d5732b6c5ae3dc8a2"}
Jul 21 02:08:00.658016 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:00.657994    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1e560e69-gqxts"
Jul 21 02:08:00.743991 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:00.743959    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e1bb63-15ea-4707-bfa3-48dc71aa24a4-maas-api-service-ca\") pod \"33e1bb63-15ea-4707-bfa3-48dc71aa24a4\" (UID: \"33e1bb63-15ea-4707-bfa3-48dc71aa24a4\") "
Jul 21 02:08:00.744327 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:00.744305    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e1bb63-15ea-4707-bfa3-48dc71aa24a4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "33e1bb63-15ea-4707-bfa3-48dc71aa24a4" (UID: "33e1bb63-15ea-4707-bfa3-48dc71aa24a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 02:08:00.845665 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:00.845567    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e1bb63-15ea-4707-bfa3-48dc71aa24a4-maas-api-service-ca\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:08:01.539426 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:01.539396    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1e560e69-gqxts"
Jul 21 02:08:01.539634 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:01.539446    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1e560e69-gqxts" event={"ID":"33e1bb63-15ea-4707-bfa3-48dc71aa24a4","Type":"ContainerDied","Data":"6cde47cc9f7d1098ed581406fdb6c88e10648dae3db871f73783ed3b7e05cc77"}
Jul 21 02:08:01.539634 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:01.539495    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cde47cc9f7d1098ed581406fdb6c88e10648dae3db871f73783ed3b7e05cc77"
Jul 21 02:08:08.319859 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:08.319820    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:08:12.940870 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:12.940811    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:08:12.951395 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:12.951371    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1e560e69-gqxts"]
Jul 21 02:08:12.958405 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:12.958382    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1e560e69-gqxts"]
Jul 21 02:08:13.025506 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:13.025480    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33e1bb63-15ea-4707-bfa3-48dc71aa24a4" path="/var/lib/kubelet/pods/33e1bb63-15ea-4707-bfa3-48dc71aa24a4/volumes"
Jul 21 02:08:13.410732 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:13.410696    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:08:20.400413 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:20.400378    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:08:20.555304 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:20.555268    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:08:20.717256 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:20.717226    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:08:21.048334 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:21.048268    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:08:21.243607 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:21.243568    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:08:21.349232 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:21.349104    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:08:21.449649 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:21.449583    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:08:27.278005 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:27.277971    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:08:27.348579 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:27.348551    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:08:27.454076 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:27.454007    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:08:27.556435 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:27.556328    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:08:27.648265 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:27.648237    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:08:33.489858 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:33.489811    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:08:33.608266 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:33.608216    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:08:40.439177 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:40.439145    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:08:40.794504 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:40.794473    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:08:41.203285 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:41.203195    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:08:44.230474 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:44.230429    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:08:45.657484 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:45.657449    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:08:48.739817 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:48.739785    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:08:55.692088 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:55.692050    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5"]
Jul 21 02:08:55.692466 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:55.692412    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33e1bb63-15ea-4707-bfa3-48dc71aa24a4" containerName="revoke-keys"
Jul 21 02:08:55.692466 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:55.692425    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="33e1bb63-15ea-4707-bfa3-48dc71aa24a4" containerName="revoke-keys"
Jul 21 02:08:55.692563 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:55.692492    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="33e1bb63-15ea-4707-bfa3-48dc71aa24a4" containerName="revoke-keys"
Jul 21 02:08:55.695304 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:55.695287    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5"
Jul 21 02:08:55.703911 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:55.703884    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f3d1a053\"/\"openshift-service-ca.crt\""
Jul 21 02:08:55.704265 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:55.703889    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f3d1a053\"/\"kube-root-ca.crt\""
Jul 21 02:08:55.704548 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:55.703920    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f3d1a053\"/\"e2e-lifecycle-model-f3d1a053-kserve-self-signed-certs\""
Jul 21 02:08:55.704765 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:55.703974    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f3d1a053\"/\"default-dockercfg-4f85f\""
Jul 21 02:08:55.706826 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:55.706804    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5"]
Jul 21 02:08:55.726450 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:55.726430    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:08:55.825254 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:55.825215    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/043d4019-94fa-4e9b-a619-a7f683f3bf9c-dshm\") pod \"e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5\" (UID: \"043d4019-94fa-4e9b-a619-a7f683f3bf9c\") " pod="ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5"
Jul 21 02:08:55.825444 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:55.825272    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/043d4019-94fa-4e9b-a619-a7f683f3bf9c-model-cache\") pod \"e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5\" (UID: \"043d4019-94fa-4e9b-a619-a7f683f3bf9c\") " pod="ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5"
Jul 21 02:08:55.825444 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:55.825352    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/043d4019-94fa-4e9b-a619-a7f683f3bf9c-tls-certs\") pod \"e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5\" (UID: \"043d4019-94fa-4e9b-a619-a7f683f3bf9c\") " pod="ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5"
Jul 21 02:08:55.825444 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:55.825397    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/043d4019-94fa-4e9b-a619-a7f683f3bf9c-home\") pod \"e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5\" (UID: \"043d4019-94fa-4e9b-a619-a7f683f3bf9c\") " pod="ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5"
Jul 21 02:08:55.825444 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:55.825417    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/043d4019-94fa-4e9b-a619-a7f683f3bf9c-tmp-dir\") pod \"e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5\" (UID: \"043d4019-94fa-4e9b-a619-a7f683f3bf9c\") " pod="ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5"
Jul 21 02:08:55.825444 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:55.825439    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qcqr\" (UniqueName: \"kubernetes.io/projected/043d4019-94fa-4e9b-a619-a7f683f3bf9c-kube-api-access-2qcqr\") pod \"e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5\" (UID: \"043d4019-94fa-4e9b-a619-a7f683f3bf9c\") " pod="ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5"
Jul 21 02:08:55.926331 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:55.926300    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/043d4019-94fa-4e9b-a619-a7f683f3bf9c-home\") pod \"e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5\" (UID: \"043d4019-94fa-4e9b-a619-a7f683f3bf9c\") " pod="ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5"
Jul 21 02:08:55.926331 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:55.926330    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/043d4019-94fa-4e9b-a619-a7f683f3bf9c-tmp-dir\") pod \"e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5\" (UID: \"043d4019-94fa-4e9b-a619-a7f683f3bf9c\") " pod="ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5"
Jul 21 02:08:55.926563 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:55.926348    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qcqr\" (UniqueName: \"kubernetes.io/projected/043d4019-94fa-4e9b-a619-a7f683f3bf9c-kube-api-access-2qcqr\") pod \"e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5\" (UID: \"043d4019-94fa-4e9b-a619-a7f683f3bf9c\") " pod="ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5"
Jul 21 02:08:55.926563 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:55.926443    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/043d4019-94fa-4e9b-a619-a7f683f3bf9c-dshm\") pod \"e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5\" (UID: \"043d4019-94fa-4e9b-a619-a7f683f3bf9c\") " pod="ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5"
Jul 21 02:08:55.926563 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:55.926502    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/043d4019-94fa-4e9b-a619-a7f683f3bf9c-model-cache\") pod \"e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5\" (UID: \"043d4019-94fa-4e9b-a619-a7f683f3bf9c\") " pod="ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5"
Jul 21 02:08:55.926757 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:55.926591    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/043d4019-94fa-4e9b-a619-a7f683f3bf9c-tls-certs\") pod \"e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5\" (UID: \"043d4019-94fa-4e9b-a619-a7f683f3bf9c\") " pod="ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5"
Jul 21 02:08:55.926816 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:55.926760    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/043d4019-94fa-4e9b-a619-a7f683f3bf9c-home\") pod \"e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5\" (UID: \"043d4019-94fa-4e9b-a619-a7f683f3bf9c\") " pod="ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5"
Jul 21 02:08:55.926816 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:55.926804    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/043d4019-94fa-4e9b-a619-a7f683f3bf9c-tmp-dir\") pod \"e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5\" (UID: \"043d4019-94fa-4e9b-a619-a7f683f3bf9c\") " pod="ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5"
Jul 21 02:08:55.926949 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:55.926927    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/043d4019-94fa-4e9b-a619-a7f683f3bf9c-model-cache\") pod \"e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5\" (UID: \"043d4019-94fa-4e9b-a619-a7f683f3bf9c\") " pod="ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5"
Jul 21 02:08:55.928638 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:55.928596    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/043d4019-94fa-4e9b-a619-a7f683f3bf9c-dshm\") pod \"e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5\" (UID: \"043d4019-94fa-4e9b-a619-a7f683f3bf9c\") " pod="ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5"
Jul 21 02:08:55.928895 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:55.928880    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/043d4019-94fa-4e9b-a619-a7f683f3bf9c-tls-certs\") pod \"e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5\" (UID: \"043d4019-94fa-4e9b-a619-a7f683f3bf9c\") " pod="ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5"
Jul 21 02:08:55.934905 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:55.934886    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qcqr\" (UniqueName: \"kubernetes.io/projected/043d4019-94fa-4e9b-a619-a7f683f3bf9c-kube-api-access-2qcqr\") pod \"e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5\" (UID: \"043d4019-94fa-4e9b-a619-a7f683f3bf9c\") " pod="ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5"
Jul 21 02:08:56.011938 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:56.011871    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5"
Jul 21 02:08:56.139453 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:56.139422    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5"]
Jul 21 02:08:56.142863 ip-10-0-129-219 kubenswrapper[2575]: W0721 02:08:56.142823    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod043d4019_94fa_4e9b_a619_a7f683f3bf9c.slice/crio-44453f9fc0afff89b1f30542f9cbef0b025fbbededa99b02547a7ef460bdadf8 WatchSource:0}: Error finding container 44453f9fc0afff89b1f30542f9cbef0b025fbbededa99b02547a7ef460bdadf8: Status 404 returned error can't find the container with id 44453f9fc0afff89b1f30542f9cbef0b025fbbededa99b02547a7ef460bdadf8
Jul 21 02:08:56.802535 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:56.802496    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5" event={"ID":"043d4019-94fa-4e9b-a619-a7f683f3bf9c","Type":"ContainerStarted","Data":"29fdcbac923874a9ded2fa74f89f131ce2c993a1cae59c9b50d812dfab79fa34"}
Jul 21 02:08:56.802935 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:56.802544    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5" event={"ID":"043d4019-94fa-4e9b-a619-a7f683f3bf9c","Type":"ContainerStarted","Data":"44453f9fc0afff89b1f30542f9cbef0b025fbbededa99b02547a7ef460bdadf8"}
Jul 21 02:08:56.819652 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:08:56.819588    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5" podStartSLOduration=1.819573454 podStartE2EDuration="1.819573454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:08:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:08:56.818167913 +0000 UTC m=+1956.415808260" watchObservedRunningTime="2026-07-21 02:08:56.819573454 +0000 UTC m=+1956.417213804"
Jul 21 02:09:06.012774 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:09:06.012715    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5"
Jul 21 02:09:06.013187 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:09:06.012797    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5"
Jul 21 02:09:06.024949 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:09:06.024918    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5"
Jul 21 02:09:06.859201 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:09:06.859171    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5"
Jul 21 02:09:22.280702 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:09:22.280669    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d1a053-dknq7"]
Jul 21 02:09:22.284253 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:09:22.284236    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d1a053-dknq7"
Jul 21 02:09:22.286562 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:09:22.286543    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x582z\""
Jul 21 02:09:22.286697 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:09:22.286577    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 21 02:09:22.290745 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:09:22.290708    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d1a053-dknq7"]
Jul 21 02:09:22.366772 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:09:22.366741    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/359f268a-4292-4319-b716-63b6d395cb10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f3d1a053-dknq7\" (UID: \"359f268a-4292-4319-b716-63b6d395cb10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d1a053-dknq7"
Jul 21 02:09:22.467453 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:09:22.467412    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/359f268a-4292-4319-b716-63b6d395cb10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f3d1a053-dknq7\" (UID: \"359f268a-4292-4319-b716-63b6d395cb10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d1a053-dknq7"
Jul 21 02:09:22.468088 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:09:22.468066    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/359f268a-4292-4319-b716-63b6d395cb10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f3d1a053-dknq7\" (UID: \"359f268a-4292-4319-b716-63b6d395cb10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d1a053-dknq7"
Jul 21 02:09:22.597078 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:09:22.597000    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d1a053-dknq7"
Jul 21 02:09:22.729243 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:09:22.729211    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d1a053-dknq7"]
Jul 21 02:09:22.732572 ip-10-0-129-219 kubenswrapper[2575]: W0721 02:09:22.732531    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod359f268a_4292_4319_b716_63b6d395cb10.slice/crio-43e3af8ef682b911df01079af0201da661f304991018e03e6c23127a6ca39623 WatchSource:0}: Error finding container 43e3af8ef682b911df01079af0201da661f304991018e03e6c23127a6ca39623: Status 404 returned error can't find the container with id 43e3af8ef682b911df01079af0201da661f304991018e03e6c23127a6ca39623
Jul 21 02:09:22.927298 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:09:22.927214    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d1a053-dknq7" event={"ID":"359f268a-4292-4319-b716-63b6d395cb10","Type":"ContainerStarted","Data":"43e3af8ef682b911df01079af0201da661f304991018e03e6c23127a6ca39623"}
Jul 21 02:09:23.932854 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:09:23.932820    2575 generic.go:356] "Generic (PLEG): container finished" podID="359f268a-4292-4319-b716-63b6d395cb10" containerID="8b1b7c70a54434cfe6bffd9980cfd85ab8c9bf022962a7de0a0ef2883258b9e6" exitCode=0
Jul 21 02:09:23.933270 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:09:23.932904    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d1a053-dknq7" event={"ID":"359f268a-4292-4319-b716-63b6d395cb10","Type":"ContainerDied","Data":"8b1b7c70a54434cfe6bffd9980cfd85ab8c9bf022962a7de0a0ef2883258b9e6"}
Jul 21 02:09:25.066038 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:09:25.066017    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d1a053-dknq7"
Jul 21 02:09:25.090623 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:09:25.090582    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/359f268a-4292-4319-b716-63b6d395cb10-maas-api-service-ca\") pod \"359f268a-4292-4319-b716-63b6d395cb10\" (UID: \"359f268a-4292-4319-b716-63b6d395cb10\") "
Jul 21 02:09:25.090900 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:09:25.090873    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/359f268a-4292-4319-b716-63b6d395cb10-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "359f268a-4292-4319-b716-63b6d395cb10" (UID: "359f268a-4292-4319-b716-63b6d395cb10"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 02:09:25.090970 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:09:25.090956    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/359f268a-4292-4319-b716-63b6d395cb10-maas-api-service-ca\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:09:25.943050 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:09:25.943020    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d1a053-dknq7"
Jul 21 02:09:25.943313 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:09:25.943017    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d1a053-dknq7" event={"ID":"359f268a-4292-4319-b716-63b6d395cb10","Type":"ContainerDied","Data":"43e3af8ef682b911df01079af0201da661f304991018e03e6c23127a6ca39623"}
Jul 21 02:09:25.943313 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:09:25.943158    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43e3af8ef682b911df01079af0201da661f304991018e03e6c23127a6ca39623"
Jul 21 02:09:37.822319 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:09:37.822286    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:09:42.355227 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:09:42.355196    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d1a053-dknq7"]
Jul 21 02:09:42.358665 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:09:42.358639    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3d1a053-dknq7"]
Jul 21 02:09:42.359012 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:09:42.358992    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:09:43.025345 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:09:43.025312    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="359f268a-4292-4319-b716-63b6d395cb10" path="/var/lib/kubelet/pods/359f268a-4292-4319-b716-63b6d395cb10/volumes"
Jul 21 02:09:44.740192 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:09:44.740159    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:09:49.715438 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:09:49.715400    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5"]
Jul 21 02:09:49.811276 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:09:49.811247    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:10:04.735587 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:04.735541    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5" podUID="043d4019-94fa-4e9b-a619-a7f683f3bf9c" containerName="main" containerID="cri-o://29fdcbac923874a9ded2fa74f89f131ce2c993a1cae59c9b50d812dfab79fa34" gracePeriod=45
Jul 21 02:10:04.990359 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:04.990302    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5"
Jul 21 02:10:05.048014 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:05.047983    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/043d4019-94fa-4e9b-a619-a7f683f3bf9c-home\") pod \"043d4019-94fa-4e9b-a619-a7f683f3bf9c\" (UID: \"043d4019-94fa-4e9b-a619-a7f683f3bf9c\") "
Jul 21 02:10:05.048164 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:05.048040    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/043d4019-94fa-4e9b-a619-a7f683f3bf9c-model-cache\") pod \"043d4019-94fa-4e9b-a619-a7f683f3bf9c\" (UID: \"043d4019-94fa-4e9b-a619-a7f683f3bf9c\") "
Jul 21 02:10:05.048164 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:05.048067    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/043d4019-94fa-4e9b-a619-a7f683f3bf9c-tmp-dir\") pod \"043d4019-94fa-4e9b-a619-a7f683f3bf9c\" (UID: \"043d4019-94fa-4e9b-a619-a7f683f3bf9c\") "
Jul 21 02:10:05.048164 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:05.048144    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qcqr\" (UniqueName: \"kubernetes.io/projected/043d4019-94fa-4e9b-a619-a7f683f3bf9c-kube-api-access-2qcqr\") pod \"043d4019-94fa-4e9b-a619-a7f683f3bf9c\" (UID: \"043d4019-94fa-4e9b-a619-a7f683f3bf9c\") "
Jul 21 02:10:05.048322 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:05.048181    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/043d4019-94fa-4e9b-a619-a7f683f3bf9c-tls-certs\") pod \"043d4019-94fa-4e9b-a619-a7f683f3bf9c\" (UID: \"043d4019-94fa-4e9b-a619-a7f683f3bf9c\") "
Jul 21 02:10:05.048322 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:05.048221    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/043d4019-94fa-4e9b-a619-a7f683f3bf9c-dshm\") pod \"043d4019-94fa-4e9b-a619-a7f683f3bf9c\" (UID: \"043d4019-94fa-4e9b-a619-a7f683f3bf9c\") "
Jul 21 02:10:05.048322 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:05.048234    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/043d4019-94fa-4e9b-a619-a7f683f3bf9c-home" (OuterVolumeSpecName: "home") pod "043d4019-94fa-4e9b-a619-a7f683f3bf9c" (UID: "043d4019-94fa-4e9b-a619-a7f683f3bf9c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:10:05.048322 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:05.048261    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/043d4019-94fa-4e9b-a619-a7f683f3bf9c-model-cache" (OuterVolumeSpecName: "model-cache") pod "043d4019-94fa-4e9b-a619-a7f683f3bf9c" (UID: "043d4019-94fa-4e9b-a619-a7f683f3bf9c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:10:05.048494 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:05.048393    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/043d4019-94fa-4e9b-a619-a7f683f3bf9c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "043d4019-94fa-4e9b-a619-a7f683f3bf9c" (UID: "043d4019-94fa-4e9b-a619-a7f683f3bf9c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:10:05.048564 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:05.048490    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/043d4019-94fa-4e9b-a619-a7f683f3bf9c-home\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:10:05.048564 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:05.048507    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/043d4019-94fa-4e9b-a619-a7f683f3bf9c-model-cache\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:10:05.048564 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:05.048521    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/043d4019-94fa-4e9b-a619-a7f683f3bf9c-tmp-dir\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:10:05.050493 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:05.050470    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/043d4019-94fa-4e9b-a619-a7f683f3bf9c-kube-api-access-2qcqr" (OuterVolumeSpecName: "kube-api-access-2qcqr") pod "043d4019-94fa-4e9b-a619-a7f683f3bf9c" (UID: "043d4019-94fa-4e9b-a619-a7f683f3bf9c"). InnerVolumeSpecName "kube-api-access-2qcqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 02:10:05.050747 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:05.050724    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/043d4019-94fa-4e9b-a619-a7f683f3bf9c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "043d4019-94fa-4e9b-a619-a7f683f3bf9c" (UID: "043d4019-94fa-4e9b-a619-a7f683f3bf9c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 02:10:05.050839 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:05.050727    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/043d4019-94fa-4e9b-a619-a7f683f3bf9c-dshm" (OuterVolumeSpecName: "dshm") pod "043d4019-94fa-4e9b-a619-a7f683f3bf9c" (UID: "043d4019-94fa-4e9b-a619-a7f683f3bf9c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:10:05.135853 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:05.135821    2575 generic.go:356] "Generic (PLEG): container finished" podID="043d4019-94fa-4e9b-a619-a7f683f3bf9c" containerID="29fdcbac923874a9ded2fa74f89f131ce2c993a1cae59c9b50d812dfab79fa34" exitCode=0
Jul 21 02:10:05.136004 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:05.135881    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5"
Jul 21 02:10:05.136004 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:05.135899    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5" event={"ID":"043d4019-94fa-4e9b-a619-a7f683f3bf9c","Type":"ContainerDied","Data":"29fdcbac923874a9ded2fa74f89f131ce2c993a1cae59c9b50d812dfab79fa34"}
Jul 21 02:10:05.136004 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:05.135949    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5" event={"ID":"043d4019-94fa-4e9b-a619-a7f683f3bf9c","Type":"ContainerDied","Data":"44453f9fc0afff89b1f30542f9cbef0b025fbbededa99b02547a7ef460bdadf8"}
Jul 21 02:10:05.136004 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:05.135972    2575 scope.go:117] "RemoveContainer" containerID="29fdcbac923874a9ded2fa74f89f131ce2c993a1cae59c9b50d812dfab79fa34"
Jul 21 02:10:05.144802 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:05.144784    2575 scope.go:117] "RemoveContainer" containerID="29fdcbac923874a9ded2fa74f89f131ce2c993a1cae59c9b50d812dfab79fa34"
Jul 21 02:10:05.145028 ip-10-0-129-219 kubenswrapper[2575]: E0721 02:10:05.145012    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29fdcbac923874a9ded2fa74f89f131ce2c993a1cae59c9b50d812dfab79fa34\": container with ID starting with 29fdcbac923874a9ded2fa74f89f131ce2c993a1cae59c9b50d812dfab79fa34 not found: ID does not exist" containerID="29fdcbac923874a9ded2fa74f89f131ce2c993a1cae59c9b50d812dfab79fa34"
Jul 21 02:10:05.145070 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:05.145033    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29fdcbac923874a9ded2fa74f89f131ce2c993a1cae59c9b50d812dfab79fa34"} err="failed to get container status \"29fdcbac923874a9ded2fa74f89f131ce2c993a1cae59c9b50d812dfab79fa34\": rpc error: code = NotFound desc = could not find container \"29fdcbac923874a9ded2fa74f89f131ce2c993a1cae59c9b50d812dfab79fa34\": container with ID starting with 29fdcbac923874a9ded2fa74f89f131ce2c993a1cae59c9b50d812dfab79fa34 not found: ID does not exist"
Jul 21 02:10:05.149295 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:05.149279    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qcqr\" (UniqueName: \"kubernetes.io/projected/043d4019-94fa-4e9b-a619-a7f683f3bf9c-kube-api-access-2qcqr\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:10:05.149360 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:05.149297    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/043d4019-94fa-4e9b-a619-a7f683f3bf9c-tls-certs\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:10:05.149360 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:05.149306    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/043d4019-94fa-4e9b-a619-a7f683f3bf9c-dshm\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:10:05.157051 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:05.157029    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5"]
Jul 21 02:10:05.161752 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:05.161732    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-f3d1a053/e2e-lifecycle-model-f3d1a053-kserve-b65fcc7f5-7jxq5"]
Jul 21 02:10:07.025118 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:07.025080    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="043d4019-94fa-4e9b-a619-a7f683f3bf9c" path="/var/lib/kubelet/pods/043d4019-94fa-4e9b-a619-a7f683f3bf9c/volumes"
Jul 21 02:10:20.916458 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:20.916414    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:10:21.017832 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:21.017795    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:10:21.091728 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:21.091698    2575 scope.go:117] "RemoveContainer" containerID="8cb5fadf6ff64992f11158ea5c5de4622da316b70042db83ab5103afdc64e534"
Jul 21 02:10:26.111592 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:26.111554    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:10:26.952193 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:26.952156    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:10:32.571003 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:32.570965    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:10:32.671540 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:32.671508    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:10:32.867988 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:32.867904    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:10:33.096327 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:33.096269    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:10:34.232418 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:34.232387    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:10:34.324430 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:34.324400    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:10:34.425314 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:34.425284    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:10:35.192261 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:35.192230    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:10:35.292589 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:35.292557    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:10:35.402245 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:35.402217    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:10:36.871878 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:36.871846    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:10:40.329761 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:40.329727    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:10:42.910290 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:42.910221    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:10:43.010538 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:43.010508    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:10:43.623769 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:43.623738    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:10:49.978713 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:49.978679    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:10:50.086809 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:50.086771    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:10:50.193728 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:10:50.193691    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:11:00.740096 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:00.740066    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:11:01.064543 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:01.064512    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:11:21.115225 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:21.115195    2575 scope.go:117] "RemoveContainer" containerID="1c1dd4ae911d4fc2e2a1261595fe59abc9eed435d123c7aa0634614cc2964095"
Jul 21 02:11:22.038748 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:22.038714    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:11:28.444549 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:28.444508    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:11:28.543829 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:28.543789    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:11:28.810583 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:28.810549    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:11:29.244968 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:29.244875    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:11:29.573755 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:29.573725    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:11:32.027569 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:32.027534    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:11:32.122883 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:32.122848    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:11:32.225246 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:32.225210    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:11:32.341078 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:32.341002    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:11:32.742140 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:32.742055    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:11:32.841147 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:32.841109    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:11:36.556255 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:36.556220    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:11:36.948911 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:36.948828    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:11:37.239113 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:37.239032    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:11:37.581935 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:37.581906    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:11:40.092909 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:40.092873    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:11:40.181381 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:40.181349    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:11:40.354309 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:40.354228    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:11:40.701663 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:40.701555    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:11:40.851056 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:40.851024    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:11:40.949861 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:40.949827    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:11:44.564901 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:44.564866    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:11:51.392639 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:51.392581    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b"]
Jul 21 02:11:51.393050 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:51.392957    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="043d4019-94fa-4e9b-a619-a7f683f3bf9c" containerName="main"
Jul 21 02:11:51.393050 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:51.392972    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="043d4019-94fa-4e9b-a619-a7f683f3bf9c" containerName="main"
Jul 21 02:11:51.393050 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:51.392986    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="359f268a-4292-4319-b716-63b6d395cb10" containerName="revoke-keys"
Jul 21 02:11:51.393050 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:51.392992    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="359f268a-4292-4319-b716-63b6d395cb10" containerName="revoke-keys"
Jul 21 02:11:51.393189 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:51.393061    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="043d4019-94fa-4e9b-a619-a7f683f3bf9c" containerName="main"
Jul 21 02:11:51.393189 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:51.393070    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="359f268a-4292-4319-b716-63b6d395cb10" containerName="revoke-keys"
Jul 21 02:11:51.396093 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:51.396078    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b"
Jul 21 02:11:51.398728 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:51.398707    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0eaad6\"/\"openshift-service-ca.crt\""
Jul 21 02:11:51.399535 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:51.399517    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0eaad6\"/\"test-model-0eaad6-kserve-self-signed-certs\""
Jul 21 02:11:51.399583 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:51.399517    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0eaad6\"/\"default-dockercfg-w9w7n\""
Jul 21 02:11:51.399650 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:51.399549    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0eaad6\"/\"kube-root-ca.crt\""
Jul 21 02:11:51.406167 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:51.406147    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b"]
Jul 21 02:11:51.468413 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:51.468387    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/103efd76-a8c7-4b48-95d9-f84bf59fe903-dshm\") pod \"test-model-0eaad6-kserve-dfc6dc9ff-bjt8b\" (UID: \"103efd76-a8c7-4b48-95d9-f84bf59fe903\") " pod="ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b"
Jul 21 02:11:51.468561 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:51.468474    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/103efd76-a8c7-4b48-95d9-f84bf59fe903-tls-certs\") pod \"test-model-0eaad6-kserve-dfc6dc9ff-bjt8b\" (UID: \"103efd76-a8c7-4b48-95d9-f84bf59fe903\") " pod="ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b"
Jul 21 02:11:51.468561 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:51.468512    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/103efd76-a8c7-4b48-95d9-f84bf59fe903-tmp-dir\") pod \"test-model-0eaad6-kserve-dfc6dc9ff-bjt8b\" (UID: \"103efd76-a8c7-4b48-95d9-f84bf59fe903\") " pod="ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b"
Jul 21 02:11:51.468703 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:51.468564    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/103efd76-a8c7-4b48-95d9-f84bf59fe903-model-cache\") pod \"test-model-0eaad6-kserve-dfc6dc9ff-bjt8b\" (UID: \"103efd76-a8c7-4b48-95d9-f84bf59fe903\") " pod="ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b"
Jul 21 02:11:51.468703 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:51.468601    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/103efd76-a8c7-4b48-95d9-f84bf59fe903-home\") pod \"test-model-0eaad6-kserve-dfc6dc9ff-bjt8b\" (UID: \"103efd76-a8c7-4b48-95d9-f84bf59fe903\") " pod="ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b"
Jul 21 02:11:51.468703 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:51.468682    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ppkr\" (UniqueName: \"kubernetes.io/projected/103efd76-a8c7-4b48-95d9-f84bf59fe903-kube-api-access-7ppkr\") pod \"test-model-0eaad6-kserve-dfc6dc9ff-bjt8b\" (UID: \"103efd76-a8c7-4b48-95d9-f84bf59fe903\") " pod="ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b"
Jul 21 02:11:51.541596 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:51.541568    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:11:51.569914 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:51.569883    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/103efd76-a8c7-4b48-95d9-f84bf59fe903-model-cache\") pod \"test-model-0eaad6-kserve-dfc6dc9ff-bjt8b\" (UID: \"103efd76-a8c7-4b48-95d9-f84bf59fe903\") " pod="ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b"
Jul 21 02:11:51.570028 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:51.569928    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/103efd76-a8c7-4b48-95d9-f84bf59fe903-home\") pod \"test-model-0eaad6-kserve-dfc6dc9ff-bjt8b\" (UID: \"103efd76-a8c7-4b48-95d9-f84bf59fe903\") " pod="ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b"
Jul 21 02:11:51.570028 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:51.569964    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ppkr\" (UniqueName: \"kubernetes.io/projected/103efd76-a8c7-4b48-95d9-f84bf59fe903-kube-api-access-7ppkr\") pod \"test-model-0eaad6-kserve-dfc6dc9ff-bjt8b\" (UID: \"103efd76-a8c7-4b48-95d9-f84bf59fe903\") " pod="ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b"
Jul 21 02:11:51.570149 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:51.570043    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/103efd76-a8c7-4b48-95d9-f84bf59fe903-dshm\") pod \"test-model-0eaad6-kserve-dfc6dc9ff-bjt8b\" (UID: \"103efd76-a8c7-4b48-95d9-f84bf59fe903\") " pod="ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b"
Jul 21 02:11:51.570149 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:51.570091    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/103efd76-a8c7-4b48-95d9-f84bf59fe903-tls-certs\") pod \"test-model-0eaad6-kserve-dfc6dc9ff-bjt8b\" (UID: \"103efd76-a8c7-4b48-95d9-f84bf59fe903\") " pod="ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b"
Jul 21 02:11:51.570149 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:51.570121    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/103efd76-a8c7-4b48-95d9-f84bf59fe903-tmp-dir\") pod \"test-model-0eaad6-kserve-dfc6dc9ff-bjt8b\" (UID: \"103efd76-a8c7-4b48-95d9-f84bf59fe903\") " pod="ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b"
Jul 21 02:11:51.570332 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:51.570306    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/103efd76-a8c7-4b48-95d9-f84bf59fe903-model-cache\") pod \"test-model-0eaad6-kserve-dfc6dc9ff-bjt8b\" (UID: \"103efd76-a8c7-4b48-95d9-f84bf59fe903\") " pod="ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b"
Jul 21 02:11:51.570445 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:51.570373    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/103efd76-a8c7-4b48-95d9-f84bf59fe903-home\") pod \"test-model-0eaad6-kserve-dfc6dc9ff-bjt8b\" (UID: \"103efd76-a8c7-4b48-95d9-f84bf59fe903\") " pod="ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b"
Jul 21 02:11:51.570445 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:51.570438    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/103efd76-a8c7-4b48-95d9-f84bf59fe903-tmp-dir\") pod \"test-model-0eaad6-kserve-dfc6dc9ff-bjt8b\" (UID: \"103efd76-a8c7-4b48-95d9-f84bf59fe903\") " pod="ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b"
Jul 21 02:11:51.572515 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:51.572495    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/103efd76-a8c7-4b48-95d9-f84bf59fe903-dshm\") pod \"test-model-0eaad6-kserve-dfc6dc9ff-bjt8b\" (UID: \"103efd76-a8c7-4b48-95d9-f84bf59fe903\") " pod="ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b"
Jul 21 02:11:51.572694 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:51.572678    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/103efd76-a8c7-4b48-95d9-f84bf59fe903-tls-certs\") pod \"test-model-0eaad6-kserve-dfc6dc9ff-bjt8b\" (UID: \"103efd76-a8c7-4b48-95d9-f84bf59fe903\") " pod="ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b"
Jul 21 02:11:51.585941 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:51.585919    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ppkr\" (UniqueName: \"kubernetes.io/projected/103efd76-a8c7-4b48-95d9-f84bf59fe903-kube-api-access-7ppkr\") pod \"test-model-0eaad6-kserve-dfc6dc9ff-bjt8b\" (UID: \"103efd76-a8c7-4b48-95d9-f84bf59fe903\") " pod="ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b"
Jul 21 02:11:51.705149 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:51.705072    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b"
Jul 21 02:11:51.833187 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:51.833161    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b"]
Jul 21 02:11:51.834934 ip-10-0-129-219 kubenswrapper[2575]: W0721 02:11:51.834905    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod103efd76_a8c7_4b48_95d9_f84bf59fe903.slice/crio-107a353f8d570bb6a5ca761c5b54072775d02c634bbe5cf3e8b03740ab87e889 WatchSource:0}: Error finding container 107a353f8d570bb6a5ca761c5b54072775d02c634bbe5cf3e8b03740ab87e889: Status 404 returned error can't find the container with id 107a353f8d570bb6a5ca761c5b54072775d02c634bbe5cf3e8b03740ab87e889
Jul 21 02:11:52.622442 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:52.622409    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b" event={"ID":"103efd76-a8c7-4b48-95d9-f84bf59fe903","Type":"ContainerStarted","Data":"a79e6674c24930793992e3b4f7af82c261bc3ae737e255fb6ea7dbaca73645d5"}
Jul 21 02:11:52.622442 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:52.622454    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b" event={"ID":"103efd76-a8c7-4b48-95d9-f84bf59fe903","Type":"ContainerStarted","Data":"107a353f8d570bb6a5ca761c5b54072775d02c634bbe5cf3e8b03740ab87e889"}
Jul 21 02:11:52.638743 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:52.638690    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b" podStartSLOduration=1.638674322 podStartE2EDuration="1.638674322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:11:52.638291457 +0000 UTC m=+2132.235931820" watchObservedRunningTime="2026-07-21 02:11:52.638674322 +0000 UTC m=+2132.236314673"
Jul 21 02:11:53.670707 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:53.670675    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:11:53.774826 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:11:53.774793    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:12:01.706144 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:01.706111    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b"
Jul 21 02:12:01.706144 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:01.706154    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b"
Jul 21 02:12:01.718498 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:01.718466    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b"
Jul 21 02:12:01.769044 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:01.769009    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:12:01.871121 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:01.871080    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:12:02.677521 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:02.677491    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b"
Jul 21 02:12:09.148534 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:09.148502    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:12:14.082206 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:14.082129    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:12:14.183386 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:14.183356    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:12:18.504946 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:18.504914    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl"]
Jul 21 02:12:18.508260 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:18.508244    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl"
Jul 21 02:12:18.511396 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:18.511359    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-27bb16\"/\"default-dockercfg-mm5mv\""
Jul 21 02:12:18.511396 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:18.511389    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-27bb16\"/\"kube-root-ca.crt\""
Jul 21 02:12:18.511642 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:18.511391    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-27bb16\"/\"test-model-27bb16-kserve-self-signed-certs\""
Jul 21 02:12:18.511642 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:18.511463    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-27bb16\"/\"openshift-service-ca.crt\""
Jul 21 02:12:18.516111 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:18.516090    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl"]
Jul 21 02:12:18.531881 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:18.531856    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:12:18.597902 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:18.597869    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4255a365-a904-4a55-86af-e7261234f010-tmp-dir\") pod \"test-model-27bb16-kserve-84d98cd849-wfqdl\" (UID: \"4255a365-a904-4a55-86af-e7261234f010\") " pod="ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl"
Jul 21 02:12:18.598030 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:18.597909    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6bzh\" (UniqueName: \"kubernetes.io/projected/4255a365-a904-4a55-86af-e7261234f010-kube-api-access-z6bzh\") pod \"test-model-27bb16-kserve-84d98cd849-wfqdl\" (UID: \"4255a365-a904-4a55-86af-e7261234f010\") " pod="ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl"
Jul 21 02:12:18.598030 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:18.597948    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4255a365-a904-4a55-86af-e7261234f010-home\") pod \"test-model-27bb16-kserve-84d98cd849-wfqdl\" (UID: \"4255a365-a904-4a55-86af-e7261234f010\") " pod="ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl"
Jul 21 02:12:18.598030 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:18.598017    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4255a365-a904-4a55-86af-e7261234f010-tls-certs\") pod \"test-model-27bb16-kserve-84d98cd849-wfqdl\" (UID: \"4255a365-a904-4a55-86af-e7261234f010\") " pod="ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl"
Jul 21 02:12:18.598140 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:18.598055    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4255a365-a904-4a55-86af-e7261234f010-dshm\") pod \"test-model-27bb16-kserve-84d98cd849-wfqdl\" (UID: \"4255a365-a904-4a55-86af-e7261234f010\") " pod="ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl"
Jul 21 02:12:18.598140 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:18.598092    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4255a365-a904-4a55-86af-e7261234f010-model-cache\") pod \"test-model-27bb16-kserve-84d98cd849-wfqdl\" (UID: \"4255a365-a904-4a55-86af-e7261234f010\") " pod="ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl"
Jul 21 02:12:18.699386 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:18.699357    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4255a365-a904-4a55-86af-e7261234f010-tls-certs\") pod \"test-model-27bb16-kserve-84d98cd849-wfqdl\" (UID: \"4255a365-a904-4a55-86af-e7261234f010\") " pod="ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl"
Jul 21 02:12:18.699386 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:18.699388    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4255a365-a904-4a55-86af-e7261234f010-dshm\") pod \"test-model-27bb16-kserve-84d98cd849-wfqdl\" (UID: \"4255a365-a904-4a55-86af-e7261234f010\") " pod="ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl"
Jul 21 02:12:18.699592 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:18.699416    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4255a365-a904-4a55-86af-e7261234f010-model-cache\") pod \"test-model-27bb16-kserve-84d98cd849-wfqdl\" (UID: \"4255a365-a904-4a55-86af-e7261234f010\") " pod="ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl"
Jul 21 02:12:18.699592 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:18.699457    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4255a365-a904-4a55-86af-e7261234f010-tmp-dir\") pod \"test-model-27bb16-kserve-84d98cd849-wfqdl\" (UID: \"4255a365-a904-4a55-86af-e7261234f010\") " pod="ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl"
Jul 21 02:12:18.699592 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:18.699477    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6bzh\" (UniqueName: \"kubernetes.io/projected/4255a365-a904-4a55-86af-e7261234f010-kube-api-access-z6bzh\") pod \"test-model-27bb16-kserve-84d98cd849-wfqdl\" (UID: \"4255a365-a904-4a55-86af-e7261234f010\") " pod="ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl"
Jul 21 02:12:18.699592 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:18.699511    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4255a365-a904-4a55-86af-e7261234f010-home\") pod \"test-model-27bb16-kserve-84d98cd849-wfqdl\" (UID: \"4255a365-a904-4a55-86af-e7261234f010\") " pod="ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl"
Jul 21 02:12:18.699919 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:18.699891    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4255a365-a904-4a55-86af-e7261234f010-model-cache\") pod \"test-model-27bb16-kserve-84d98cd849-wfqdl\" (UID: \"4255a365-a904-4a55-86af-e7261234f010\") " pod="ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl"
Jul 21 02:12:18.700019 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:18.699924    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4255a365-a904-4a55-86af-e7261234f010-tmp-dir\") pod \"test-model-27bb16-kserve-84d98cd849-wfqdl\" (UID: \"4255a365-a904-4a55-86af-e7261234f010\") " pod="ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl"
Jul 21 02:12:18.700019 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:18.699951    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4255a365-a904-4a55-86af-e7261234f010-home\") pod \"test-model-27bb16-kserve-84d98cd849-wfqdl\" (UID: \"4255a365-a904-4a55-86af-e7261234f010\") " pod="ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl"
Jul 21 02:12:18.701556 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:18.701536    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4255a365-a904-4a55-86af-e7261234f010-dshm\") pod \"test-model-27bb16-kserve-84d98cd849-wfqdl\" (UID: \"4255a365-a904-4a55-86af-e7261234f010\") " pod="ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl"
Jul 21 02:12:18.701903 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:18.701888    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4255a365-a904-4a55-86af-e7261234f010-tls-certs\") pod \"test-model-27bb16-kserve-84d98cd849-wfqdl\" (UID: \"4255a365-a904-4a55-86af-e7261234f010\") " pod="ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl"
Jul 21 02:12:18.707031 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:18.707013    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6bzh\" (UniqueName: \"kubernetes.io/projected/4255a365-a904-4a55-86af-e7261234f010-kube-api-access-z6bzh\") pod \"test-model-27bb16-kserve-84d98cd849-wfqdl\" (UID: \"4255a365-a904-4a55-86af-e7261234f010\") " pod="ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl"
Jul 21 02:12:18.818284 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:18.818258    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl"
Jul 21 02:12:18.951568 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:18.951539    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl"]
Jul 21 02:12:18.954729 ip-10-0-129-219 kubenswrapper[2575]: W0721 02:12:18.954705    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4255a365_a904_4a55_86af_e7261234f010.slice/crio-71c5a1bc8ca67660c7fab4f16d009a36fa6b8fad689980305d8c39df7d60d33c WatchSource:0}: Error finding container 71c5a1bc8ca67660c7fab4f16d009a36fa6b8fad689980305d8c39df7d60d33c: Status 404 returned error can't find the container with id 71c5a1bc8ca67660c7fab4f16d009a36fa6b8fad689980305d8c39df7d60d33c
Jul 21 02:12:19.745275 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:19.745246    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl" event={"ID":"4255a365-a904-4a55-86af-e7261234f010","Type":"ContainerStarted","Data":"91a922a4a7d793642a1e6ef6aef707a00b870ac2c5d0e73a25f9e81b1413e950"}
Jul 21 02:12:19.745667 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:19.745287    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl" event={"ID":"4255a365-a904-4a55-86af-e7261234f010","Type":"ContainerStarted","Data":"71c5a1bc8ca67660c7fab4f16d009a36fa6b8fad689980305d8c39df7d60d33c"}
Jul 21 02:12:19.764256 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:19.764206    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl" podStartSLOduration=1.764191865 podStartE2EDuration="1.764191865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:12:19.762785257 +0000 UTC m=+2159.360425607" watchObservedRunningTime="2026-07-21 02:12:19.764191865 +0000 UTC m=+2159.361832216"
Jul 21 02:12:21.133008 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:21.132977    2575 scope.go:117] "RemoveContainer" containerID="4bced88bb29cc701d50ba855c49714272bceaf189bdcac753524b88bc2047618"
Jul 21 02:12:28.818871 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:28.818836    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl"
Jul 21 02:12:28.819475 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:28.818973    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl"
Jul 21 02:12:28.831295 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:28.831269    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl"
Jul 21 02:12:29.805630 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:29.805580    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl"
Jul 21 02:12:46.786270 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:12:46.786228    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:13:17.586667 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:17.586586    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:13:17.687129 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:17.687098    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:13:19.649727 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:19.649699    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b"]
Jul 21 02:13:19.671742 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:19.671711    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:13:20.609497 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:20.609462    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl"]
Jul 21 02:13:20.629791 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:20.629763    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:13:20.768591 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:20.768560    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0eaad6-64sr6"]
Jul 21 02:13:20.772017 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:20.772001    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0eaad6-64sr6"
Jul 21 02:13:20.774887 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:20.774869    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x582z\""
Jul 21 02:13:20.775247 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:20.775232    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 21 02:13:20.793063 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:20.793042    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0eaad6-64sr6"]
Jul 21 02:13:20.937346 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:20.937256    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87f67d63-be3b-4b46-9bae-4e5b57f27696-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0eaad6-64sr6\" (UID: \"87f67d63-be3b-4b46-9bae-4e5b57f27696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0eaad6-64sr6"
Jul 21 02:13:21.038186 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:21.038154    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87f67d63-be3b-4b46-9bae-4e5b57f27696-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0eaad6-64sr6\" (UID: \"87f67d63-be3b-4b46-9bae-4e5b57f27696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0eaad6-64sr6"
Jul 21 02:13:21.040937 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:21.040918    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 21 02:13:21.049442 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:21.049422    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87f67d63-be3b-4b46-9bae-4e5b57f27696-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0eaad6-64sr6\" (UID: \"87f67d63-be3b-4b46-9bae-4e5b57f27696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0eaad6-64sr6"
Jul 21 02:13:21.084142 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:21.084123    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x582z\""
Jul 21 02:13:21.092851 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:21.092836    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0eaad6-64sr6"
Jul 21 02:13:21.224302 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:21.224272    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0eaad6-64sr6"]
Jul 21 02:13:21.226129 ip-10-0-129-219 kubenswrapper[2575]: W0721 02:13:21.226104    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87f67d63_be3b_4b46_9bae_4e5b57f27696.slice/crio-872279f3d8c4681af8f55caca695e1eaa9812b16d2f7c8c6b43b048e19a0b484 WatchSource:0}: Error finding container 872279f3d8c4681af8f55caca695e1eaa9812b16d2f7c8c6b43b048e19a0b484: Status 404 returned error can't find the container with id 872279f3d8c4681af8f55caca695e1eaa9812b16d2f7c8c6b43b048e19a0b484
Jul 21 02:13:21.339478 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:21.339432    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:13:21.440574 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:21.440547    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:13:22.022266 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:22.022234    2575 generic.go:356] "Generic (PLEG): container finished" podID="87f67d63-be3b-4b46-9bae-4e5b57f27696" containerID="babf8b78a2fac6d056b6ad156a5c497b35363b297b1a79cdf30fc0ab2914ec57" exitCode=0
Jul 21 02:13:22.022601 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:22.022318    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0eaad6-64sr6" event={"ID":"87f67d63-be3b-4b46-9bae-4e5b57f27696","Type":"ContainerDied","Data":"babf8b78a2fac6d056b6ad156a5c497b35363b297b1a79cdf30fc0ab2914ec57"}
Jul 21 02:13:22.022601 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:22.022371    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0eaad6-64sr6" event={"ID":"87f67d63-be3b-4b46-9bae-4e5b57f27696","Type":"ContainerStarted","Data":"872279f3d8c4681af8f55caca695e1eaa9812b16d2f7c8c6b43b048e19a0b484"}
Jul 21 02:13:23.154070 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:23.154046    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0eaad6-64sr6"
Jul 21 02:13:23.259102 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:23.259070    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87f67d63-be3b-4b46-9bae-4e5b57f27696-maas-api-service-ca\") pod \"87f67d63-be3b-4b46-9bae-4e5b57f27696\" (UID: \"87f67d63-be3b-4b46-9bae-4e5b57f27696\") "
Jul 21 02:13:23.259448 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:23.259427    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87f67d63-be3b-4b46-9bae-4e5b57f27696-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "87f67d63-be3b-4b46-9bae-4e5b57f27696" (UID: "87f67d63-be3b-4b46-9bae-4e5b57f27696"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 02:13:23.360510 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:23.360438    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87f67d63-be3b-4b46-9bae-4e5b57f27696-maas-api-service-ca\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:13:24.034417 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:24.034387    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0eaad6-64sr6"
Jul 21 02:13:24.034576 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:24.034386    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0eaad6-64sr6" event={"ID":"87f67d63-be3b-4b46-9bae-4e5b57f27696","Type":"ContainerDied","Data":"872279f3d8c4681af8f55caca695e1eaa9812b16d2f7c8c6b43b048e19a0b484"}
Jul 21 02:13:24.034576 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:24.034503    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="872279f3d8c4681af8f55caca695e1eaa9812b16d2f7c8c6b43b048e19a0b484"
Jul 21 02:13:31.201548 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:31.201514    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:13:34.669957 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:34.669898    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b" podUID="103efd76-a8c7-4b48-95d9-f84bf59fe903" containerName="main" containerID="cri-o://a79e6674c24930793992e3b4f7af82c261bc3ae737e255fb6ea7dbaca73645d5" gracePeriod=45
Jul 21 02:13:34.918224 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:34.918200    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b"
Jul 21 02:13:34.961918 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:34.961817    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/103efd76-a8c7-4b48-95d9-f84bf59fe903-home\") pod \"103efd76-a8c7-4b48-95d9-f84bf59fe903\" (UID: \"103efd76-a8c7-4b48-95d9-f84bf59fe903\") "
Jul 21 02:13:34.961918 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:34.961884    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/103efd76-a8c7-4b48-95d9-f84bf59fe903-dshm\") pod \"103efd76-a8c7-4b48-95d9-f84bf59fe903\" (UID: \"103efd76-a8c7-4b48-95d9-f84bf59fe903\") "
Jul 21 02:13:34.962140 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:34.961933    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ppkr\" (UniqueName: \"kubernetes.io/projected/103efd76-a8c7-4b48-95d9-f84bf59fe903-kube-api-access-7ppkr\") pod \"103efd76-a8c7-4b48-95d9-f84bf59fe903\" (UID: \"103efd76-a8c7-4b48-95d9-f84bf59fe903\") "
Jul 21 02:13:34.962140 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:34.962014    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/103efd76-a8c7-4b48-95d9-f84bf59fe903-tls-certs\") pod \"103efd76-a8c7-4b48-95d9-f84bf59fe903\" (UID: \"103efd76-a8c7-4b48-95d9-f84bf59fe903\") "
Jul 21 02:13:34.962140 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:34.962044    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/103efd76-a8c7-4b48-95d9-f84bf59fe903-model-cache\") pod \"103efd76-a8c7-4b48-95d9-f84bf59fe903\" (UID: \"103efd76-a8c7-4b48-95d9-f84bf59fe903\") "
Jul 21 02:13:34.962140 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:34.962084    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/103efd76-a8c7-4b48-95d9-f84bf59fe903-tmp-dir\") pod \"103efd76-a8c7-4b48-95d9-f84bf59fe903\" (UID: \"103efd76-a8c7-4b48-95d9-f84bf59fe903\") "
Jul 21 02:13:34.962352 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:34.962146    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103efd76-a8c7-4b48-95d9-f84bf59fe903-home" (OuterVolumeSpecName: "home") pod "103efd76-a8c7-4b48-95d9-f84bf59fe903" (UID: "103efd76-a8c7-4b48-95d9-f84bf59fe903"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:13:34.962404 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:34.962375    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103efd76-a8c7-4b48-95d9-f84bf59fe903-model-cache" (OuterVolumeSpecName: "model-cache") pod "103efd76-a8c7-4b48-95d9-f84bf59fe903" (UID: "103efd76-a8c7-4b48-95d9-f84bf59fe903"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:13:34.962454 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:34.962394    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/103efd76-a8c7-4b48-95d9-f84bf59fe903-home\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:13:34.962503 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:34.962447    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103efd76-a8c7-4b48-95d9-f84bf59fe903-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "103efd76-a8c7-4b48-95d9-f84bf59fe903" (UID: "103efd76-a8c7-4b48-95d9-f84bf59fe903"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:13:34.964174 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:34.964151    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103efd76-a8c7-4b48-95d9-f84bf59fe903-kube-api-access-7ppkr" (OuterVolumeSpecName: "kube-api-access-7ppkr") pod "103efd76-a8c7-4b48-95d9-f84bf59fe903" (UID: "103efd76-a8c7-4b48-95d9-f84bf59fe903"). InnerVolumeSpecName "kube-api-access-7ppkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 02:13:34.964282 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:34.964180    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103efd76-a8c7-4b48-95d9-f84bf59fe903-dshm" (OuterVolumeSpecName: "dshm") pod "103efd76-a8c7-4b48-95d9-f84bf59fe903" (UID: "103efd76-a8c7-4b48-95d9-f84bf59fe903"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:13:34.964282 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:34.964237    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/103efd76-a8c7-4b48-95d9-f84bf59fe903-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "103efd76-a8c7-4b48-95d9-f84bf59fe903" (UID: "103efd76-a8c7-4b48-95d9-f84bf59fe903"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 02:13:35.064167 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:35.064132    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ppkr\" (UniqueName: \"kubernetes.io/projected/103efd76-a8c7-4b48-95d9-f84bf59fe903-kube-api-access-7ppkr\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:13:35.064167 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:35.064161    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/103efd76-a8c7-4b48-95d9-f84bf59fe903-tls-certs\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:13:35.064167 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:35.064170    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/103efd76-a8c7-4b48-95d9-f84bf59fe903-model-cache\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:13:35.064167 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:35.064178    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/103efd76-a8c7-4b48-95d9-f84bf59fe903-tmp-dir\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:13:35.064517 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:35.064187    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/103efd76-a8c7-4b48-95d9-f84bf59fe903-dshm\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:13:35.089233 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:35.089197    2575 generic.go:356] "Generic (PLEG): container finished" podID="103efd76-a8c7-4b48-95d9-f84bf59fe903" containerID="a79e6674c24930793992e3b4f7af82c261bc3ae737e255fb6ea7dbaca73645d5" exitCode=0
Jul 21 02:13:35.089401 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:35.089242    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b"
Jul 21 02:13:35.089401 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:35.089283    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b" event={"ID":"103efd76-a8c7-4b48-95d9-f84bf59fe903","Type":"ContainerDied","Data":"a79e6674c24930793992e3b4f7af82c261bc3ae737e255fb6ea7dbaca73645d5"}
Jul 21 02:13:35.089401 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:35.089332    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b" event={"ID":"103efd76-a8c7-4b48-95d9-f84bf59fe903","Type":"ContainerDied","Data":"107a353f8d570bb6a5ca761c5b54072775d02c634bbe5cf3e8b03740ab87e889"}
Jul 21 02:13:35.089401 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:35.089354    2575 scope.go:117] "RemoveContainer" containerID="a79e6674c24930793992e3b4f7af82c261bc3ae737e255fb6ea7dbaca73645d5"
Jul 21 02:13:35.105488 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:35.105455    2575 scope.go:117] "RemoveContainer" containerID="a79e6674c24930793992e3b4f7af82c261bc3ae737e255fb6ea7dbaca73645d5"
Jul 21 02:13:35.105821 ip-10-0-129-219 kubenswrapper[2575]: E0721 02:13:35.105802    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a79e6674c24930793992e3b4f7af82c261bc3ae737e255fb6ea7dbaca73645d5\": container with ID starting with a79e6674c24930793992e3b4f7af82c261bc3ae737e255fb6ea7dbaca73645d5 not found: ID does not exist" containerID="a79e6674c24930793992e3b4f7af82c261bc3ae737e255fb6ea7dbaca73645d5"
Jul 21 02:13:35.105972 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:35.105938    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a79e6674c24930793992e3b4f7af82c261bc3ae737e255fb6ea7dbaca73645d5"} err="failed to get container status \"a79e6674c24930793992e3b4f7af82c261bc3ae737e255fb6ea7dbaca73645d5\": rpc error: code = NotFound desc = could not find container \"a79e6674c24930793992e3b4f7af82c261bc3ae737e255fb6ea7dbaca73645d5\": container with ID starting with a79e6674c24930793992e3b4f7af82c261bc3ae737e255fb6ea7dbaca73645d5 not found: ID does not exist"
Jul 21 02:13:35.109798 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:35.109749    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b"]
Jul 21 02:13:35.111596 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:35.111570    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-0eaad6/test-model-0eaad6-kserve-dfc6dc9ff-bjt8b"]
Jul 21 02:13:35.629743 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:35.629705    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl" podUID="4255a365-a904-4a55-86af-e7261234f010" containerName="main" containerID="cri-o://91a922a4a7d793642a1e6ef6aef707a00b870ac2c5d0e73a25f9e81b1413e950" gracePeriod=45
Jul 21 02:13:35.829491 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:35.829458    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:13:35.842958 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:35.842923    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0eaad6-64sr6"]
Jul 21 02:13:35.845803 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:35.845777    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0eaad6-64sr6"]
Jul 21 02:13:35.883489 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:35.883434    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl"
Jul 21 02:13:35.971803 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:35.971768    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4255a365-a904-4a55-86af-e7261234f010-home\") pod \"4255a365-a904-4a55-86af-e7261234f010\" (UID: \"4255a365-a904-4a55-86af-e7261234f010\") "
Jul 21 02:13:35.971803 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:35.971805    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4255a365-a904-4a55-86af-e7261234f010-model-cache\") pod \"4255a365-a904-4a55-86af-e7261234f010\" (UID: \"4255a365-a904-4a55-86af-e7261234f010\") "
Jul 21 02:13:35.972060 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:35.971892    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4255a365-a904-4a55-86af-e7261234f010-dshm\") pod \"4255a365-a904-4a55-86af-e7261234f010\" (UID: \"4255a365-a904-4a55-86af-e7261234f010\") "
Jul 21 02:13:35.972060 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:35.971927    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6bzh\" (UniqueName: \"kubernetes.io/projected/4255a365-a904-4a55-86af-e7261234f010-kube-api-access-z6bzh\") pod \"4255a365-a904-4a55-86af-e7261234f010\" (UID: \"4255a365-a904-4a55-86af-e7261234f010\") "
Jul 21 02:13:35.972060 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:35.971955    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4255a365-a904-4a55-86af-e7261234f010-tmp-dir\") pod \"4255a365-a904-4a55-86af-e7261234f010\" (UID: \"4255a365-a904-4a55-86af-e7261234f010\") "
Jul 21 02:13:35.972060 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:35.971972    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4255a365-a904-4a55-86af-e7261234f010-tls-certs\") pod \"4255a365-a904-4a55-86af-e7261234f010\" (UID: \"4255a365-a904-4a55-86af-e7261234f010\") "
Jul 21 02:13:35.972238 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:35.972119    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4255a365-a904-4a55-86af-e7261234f010-home" (OuterVolumeSpecName: "home") pod "4255a365-a904-4a55-86af-e7261234f010" (UID: "4255a365-a904-4a55-86af-e7261234f010"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:13:35.972275 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:35.972222    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4255a365-a904-4a55-86af-e7261234f010-model-cache" (OuterVolumeSpecName: "model-cache") pod "4255a365-a904-4a55-86af-e7261234f010" (UID: "4255a365-a904-4a55-86af-e7261234f010"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:13:35.972275 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:35.972259    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4255a365-a904-4a55-86af-e7261234f010-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4255a365-a904-4a55-86af-e7261234f010" (UID: "4255a365-a904-4a55-86af-e7261234f010"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:13:35.974147 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:35.974107    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4255a365-a904-4a55-86af-e7261234f010-dshm" (OuterVolumeSpecName: "dshm") pod "4255a365-a904-4a55-86af-e7261234f010" (UID: "4255a365-a904-4a55-86af-e7261234f010"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:13:35.974147 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:35.974130    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4255a365-a904-4a55-86af-e7261234f010-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4255a365-a904-4a55-86af-e7261234f010" (UID: "4255a365-a904-4a55-86af-e7261234f010"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 02:13:35.974286 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:35.974145    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4255a365-a904-4a55-86af-e7261234f010-kube-api-access-z6bzh" (OuterVolumeSpecName: "kube-api-access-z6bzh") pod "4255a365-a904-4a55-86af-e7261234f010" (UID: "4255a365-a904-4a55-86af-e7261234f010"). InnerVolumeSpecName "kube-api-access-z6bzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 02:13:36.073290 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:36.073253    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4255a365-a904-4a55-86af-e7261234f010-dshm\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:13:36.073290 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:36.073286    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6bzh\" (UniqueName: \"kubernetes.io/projected/4255a365-a904-4a55-86af-e7261234f010-kube-api-access-z6bzh\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:13:36.073290 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:36.073298    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4255a365-a904-4a55-86af-e7261234f010-tmp-dir\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:13:36.073524 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:36.073308    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4255a365-a904-4a55-86af-e7261234f010-tls-certs\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:13:36.073524 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:36.073316    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4255a365-a904-4a55-86af-e7261234f010-home\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:13:36.073524 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:36.073323    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4255a365-a904-4a55-86af-e7261234f010-model-cache\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:13:36.095326 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:36.095296    2575 generic.go:356] "Generic (PLEG): container finished" podID="4255a365-a904-4a55-86af-e7261234f010" containerID="91a922a4a7d793642a1e6ef6aef707a00b870ac2c5d0e73a25f9e81b1413e950" exitCode=0
Jul 21 02:13:36.095467 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:36.095342    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl"
Jul 21 02:13:36.095467 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:36.095382    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl" event={"ID":"4255a365-a904-4a55-86af-e7261234f010","Type":"ContainerDied","Data":"91a922a4a7d793642a1e6ef6aef707a00b870ac2c5d0e73a25f9e81b1413e950"}
Jul 21 02:13:36.095467 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:36.095423    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl" event={"ID":"4255a365-a904-4a55-86af-e7261234f010","Type":"ContainerDied","Data":"71c5a1bc8ca67660c7fab4f16d009a36fa6b8fad689980305d8c39df7d60d33c"}
Jul 21 02:13:36.095467 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:36.095445    2575 scope.go:117] "RemoveContainer" containerID="91a922a4a7d793642a1e6ef6aef707a00b870ac2c5d0e73a25f9e81b1413e950"
Jul 21 02:13:36.104708 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:36.104691    2575 scope.go:117] "RemoveContainer" containerID="91a922a4a7d793642a1e6ef6aef707a00b870ac2c5d0e73a25f9e81b1413e950"
Jul 21 02:13:36.104961 ip-10-0-129-219 kubenswrapper[2575]: E0721 02:13:36.104940    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91a922a4a7d793642a1e6ef6aef707a00b870ac2c5d0e73a25f9e81b1413e950\": container with ID starting with 91a922a4a7d793642a1e6ef6aef707a00b870ac2c5d0e73a25f9e81b1413e950 not found: ID does not exist" containerID="91a922a4a7d793642a1e6ef6aef707a00b870ac2c5d0e73a25f9e81b1413e950"
Jul 21 02:13:36.105053 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:36.104967    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91a922a4a7d793642a1e6ef6aef707a00b870ac2c5d0e73a25f9e81b1413e950"} err="failed to get container status \"91a922a4a7d793642a1e6ef6aef707a00b870ac2c5d0e73a25f9e81b1413e950\": rpc error: code = NotFound desc = could not find container \"91a922a4a7d793642a1e6ef6aef707a00b870ac2c5d0e73a25f9e81b1413e950\": container with ID starting with 91a922a4a7d793642a1e6ef6aef707a00b870ac2c5d0e73a25f9e81b1413e950 not found: ID does not exist"
Jul 21 02:13:36.116012 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:36.115987    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl"]
Jul 21 02:13:36.119528 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:36.119505    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-27bb16/test-model-27bb16-kserve-84d98cd849-wfqdl"]
Jul 21 02:13:36.228345 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:36.228270    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:13:37.026010 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:37.025978    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="103efd76-a8c7-4b48-95d9-f84bf59fe903" path="/var/lib/kubelet/pods/103efd76-a8c7-4b48-95d9-f84bf59fe903/volumes"
Jul 21 02:13:37.026393 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:37.026372    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4255a365-a904-4a55-86af-e7261234f010" path="/var/lib/kubelet/pods/4255a365-a904-4a55-86af-e7261234f010/volumes"
Jul 21 02:13:37.026698 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:37.026683    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87f67d63-be3b-4b46-9bae-4e5b57f27696" path="/var/lib/kubelet/pods/87f67d63-be3b-4b46-9bae-4e5b57f27696/volumes"
Jul 21 02:13:42.744955 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:42.744880    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:13:42.843200 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:42.843164    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:13:42.932802 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:42.932763    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:13:42.933195 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:42.933174    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27bb16-zzkrw"]
Jul 21 02:13:42.933542 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:42.933527    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87f67d63-be3b-4b46-9bae-4e5b57f27696" containerName="revoke-keys"
Jul 21 02:13:42.933640 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:42.933544    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="87f67d63-be3b-4b46-9bae-4e5b57f27696" containerName="revoke-keys"
Jul 21 02:13:42.933640 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:42.933582    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="103efd76-a8c7-4b48-95d9-f84bf59fe903" containerName="main"
Jul 21 02:13:42.933640 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:42.933592    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="103efd76-a8c7-4b48-95d9-f84bf59fe903" containerName="main"
Jul 21 02:13:42.933640 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:42.933635    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4255a365-a904-4a55-86af-e7261234f010" containerName="main"
Jul 21 02:13:42.933840 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:42.933645    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4255a365-a904-4a55-86af-e7261234f010" containerName="main"
Jul 21 02:13:42.933840 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:42.933732    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="87f67d63-be3b-4b46-9bae-4e5b57f27696" containerName="revoke-keys"
Jul 21 02:13:42.933840 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:42.933747    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="103efd76-a8c7-4b48-95d9-f84bf59fe903" containerName="main"
Jul 21 02:13:42.933840 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:42.933756    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4255a365-a904-4a55-86af-e7261234f010" containerName="main"
Jul 21 02:13:42.936535 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:42.936517    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27bb16-zzkrw"
Jul 21 02:13:42.938882 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:42.938865    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x582z\""
Jul 21 02:13:42.938958 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:42.938927    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 21 02:13:42.944827 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:42.944792    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27bb16-zzkrw"]
Jul 21 02:13:43.033285 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:43.033254    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d64b8a91-1a17-4b5b-aa3b-2240729abaad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-27bb16-zzkrw\" (UID: \"d64b8a91-1a17-4b5b-aa3b-2240729abaad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27bb16-zzkrw"
Jul 21 02:13:43.133877 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:43.133844    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d64b8a91-1a17-4b5b-aa3b-2240729abaad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-27bb16-zzkrw\" (UID: \"d64b8a91-1a17-4b5b-aa3b-2240729abaad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27bb16-zzkrw"
Jul 21 02:13:43.134441 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:43.134422    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d64b8a91-1a17-4b5b-aa3b-2240729abaad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-27bb16-zzkrw\" (UID: \"d64b8a91-1a17-4b5b-aa3b-2240729abaad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27bb16-zzkrw"
Jul 21 02:13:43.247335 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:43.247295    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27bb16-zzkrw"
Jul 21 02:13:43.385680 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:43.385627    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27bb16-zzkrw"]
Jul 21 02:13:43.387903 ip-10-0-129-219 kubenswrapper[2575]: W0721 02:13:43.387876    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd64b8a91_1a17_4b5b_aa3b_2240729abaad.slice/crio-89804d845926225ef672c8215d4a5e2add33636c27326162bd3bbcbf314f7569 WatchSource:0}: Error finding container 89804d845926225ef672c8215d4a5e2add33636c27326162bd3bbcbf314f7569: Status 404 returned error can't find the container with id 89804d845926225ef672c8215d4a5e2add33636c27326162bd3bbcbf314f7569
Jul 21 02:13:43.520276 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:43.520243    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:13:43.618813 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:43.618743    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:13:44.134818 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:44.134786    2575 generic.go:356] "Generic (PLEG): container finished" podID="d64b8a91-1a17-4b5b-aa3b-2240729abaad" containerID="11214f230f6908518ae0adc4b90a036261985ac6951a4cfaa0e863ee287bf99e" exitCode=0
Jul 21 02:13:44.135142 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:44.134866    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27bb16-zzkrw" event={"ID":"d64b8a91-1a17-4b5b-aa3b-2240729abaad","Type":"ContainerDied","Data":"11214f230f6908518ae0adc4b90a036261985ac6951a4cfaa0e863ee287bf99e"}
Jul 21 02:13:44.135142 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:44.134913    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27bb16-zzkrw" event={"ID":"d64b8a91-1a17-4b5b-aa3b-2240729abaad","Type":"ContainerStarted","Data":"89804d845926225ef672c8215d4a5e2add33636c27326162bd3bbcbf314f7569"}
Jul 21 02:13:45.265622 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:45.265593    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27bb16-zzkrw"
Jul 21 02:13:45.351077 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:45.351037    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d64b8a91-1a17-4b5b-aa3b-2240729abaad-maas-api-service-ca\") pod \"d64b8a91-1a17-4b5b-aa3b-2240729abaad\" (UID: \"d64b8a91-1a17-4b5b-aa3b-2240729abaad\") "
Jul 21 02:13:45.351364 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:45.351343    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d64b8a91-1a17-4b5b-aa3b-2240729abaad-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d64b8a91-1a17-4b5b-aa3b-2240729abaad" (UID: "d64b8a91-1a17-4b5b-aa3b-2240729abaad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 02:13:45.452515 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:45.452438    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d64b8a91-1a17-4b5b-aa3b-2240729abaad-maas-api-service-ca\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:13:46.145414 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:46.145377    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27bb16-zzkrw" event={"ID":"d64b8a91-1a17-4b5b-aa3b-2240729abaad","Type":"ContainerDied","Data":"89804d845926225ef672c8215d4a5e2add33636c27326162bd3bbcbf314f7569"}
Jul 21 02:13:46.145414 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:46.145417    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27bb16-zzkrw"
Jul 21 02:13:46.145635 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:46.145425    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89804d845926225ef672c8215d4a5e2add33636c27326162bd3bbcbf314f7569"
Jul 21 02:13:53.384588 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:53.384556    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:13:58.022651 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:58.022607    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27bb16-zzkrw"]
Jul 21 02:13:58.026834 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:58.026809    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-27bb16-zzkrw"]
Jul 21 02:13:58.028604 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:58.028581    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:13:58.429869 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:58.429782    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:13:59.025794 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:13:59.025760    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d64b8a91-1a17-4b5b-aa3b-2240729abaad" path="/var/lib/kubelet/pods/d64b8a91-1a17-4b5b-aa3b-2240729abaad/volumes"
Jul 21 02:14:04.502458 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:04.502429    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:04.646182 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:04.646150    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:04.752026 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:04.751996    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:04.992735 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:04.992703    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:05.384031 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:05.383998    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:05.665889 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:05.665808    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:05.954213 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:05.954134    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:08.782746 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:08.782711    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:08.881862 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:08.881837    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:08.997400 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:08.997359    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:09.412659 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:09.412604    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:09.635974 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:09.635945    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:09.729564 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:09.729478    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:13.323464 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:13.323428    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:13.822918 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:13.822885    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:14.136257 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:14.136176    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:14.278760 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:14.278707    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:16.856484 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:16.856450    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:16.958802 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:16.958766    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:17.055624 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:17.055578    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:17.220390 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:17.220311    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:17.645936 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:17.645906    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:17.747337 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:17.747308    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:21.160329 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:21.160297    2575 scope.go:117] "RemoveContainer" containerID="9ad2cb93ef72ab5d66ef8f8f6cca4c4520c75d19f0caf74d5732b6c5ae3dc8a2"
Jul 21 02:14:21.367533 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:21.367500    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:29.535042 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:29.535012    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-91b59b-nd48b"]
Jul 21 02:14:29.535492 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:29.535354    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d64b8a91-1a17-4b5b-aa3b-2240729abaad" containerName="revoke-keys"
Jul 21 02:14:29.535492 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:29.535365    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d64b8a91-1a17-4b5b-aa3b-2240729abaad" containerName="revoke-keys"
Jul 21 02:14:29.535492 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:29.535415    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d64b8a91-1a17-4b5b-aa3b-2240729abaad" containerName="revoke-keys"
Jul 21 02:14:29.538258 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:29.538242    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-91b59b-nd48b"
Jul 21 02:14:29.540648 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:29.540605    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x582z\""
Jul 21 02:14:29.540766 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:29.540681    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 21 02:14:29.545429 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:29.545408    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-91b59b-nd48b"]
Jul 21 02:14:29.636302 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:29.636268    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7262fb4c-c9f4-4d35-872c-293d15cd3e14-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-91b59b-nd48b\" (UID: \"7262fb4c-c9f4-4d35-872c-293d15cd3e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-91b59b-nd48b"
Jul 21 02:14:29.737742 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:29.737703    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7262fb4c-c9f4-4d35-872c-293d15cd3e14-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-91b59b-nd48b\" (UID: \"7262fb4c-c9f4-4d35-872c-293d15cd3e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-91b59b-nd48b"
Jul 21 02:14:29.738309 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:29.738287    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7262fb4c-c9f4-4d35-872c-293d15cd3e14-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-91b59b-nd48b\" (UID: \"7262fb4c-c9f4-4d35-872c-293d15cd3e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-91b59b-nd48b"
Jul 21 02:14:29.850137 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:29.850042    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-91b59b-nd48b"
Jul 21 02:14:29.976473 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:29.976445    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-91b59b-nd48b"]
Jul 21 02:14:29.978296 ip-10-0-129-219 kubenswrapper[2575]: W0721 02:14:29.978271    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7262fb4c_c9f4_4d35_872c_293d15cd3e14.slice/crio-1bea634313d8fdd0d2b3b32f7e1c0cc8d6fc36f82aea0f59ad29e3d2755de8bc WatchSource:0}: Error finding container 1bea634313d8fdd0d2b3b32f7e1c0cc8d6fc36f82aea0f59ad29e3d2755de8bc: Status 404 returned error can't find the container with id 1bea634313d8fdd0d2b3b32f7e1c0cc8d6fc36f82aea0f59ad29e3d2755de8bc
Jul 21 02:14:30.052498 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:30.052461    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:30.160088 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:30.160008    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:30.349475 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:30.349445    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-91b59b-nd48b" event={"ID":"7262fb4c-c9f4-4d35-872c-293d15cd3e14","Type":"ContainerStarted","Data":"5ec5b5322bbc8ab1fbb5af370d5b643b608e9faf720aad5379da407611a3a930"}
Jul 21 02:14:30.349596 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:30.349528    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-91b59b-nd48b" event={"ID":"7262fb4c-c9f4-4d35-872c-293d15cd3e14","Type":"ContainerStarted","Data":"1bea634313d8fdd0d2b3b32f7e1c0cc8d6fc36f82aea0f59ad29e3d2755de8bc"}
Jul 21 02:14:30.363772 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:30.363701    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-91b59b-nd48b" podStartSLOduration=1.363682542 podStartE2EDuration="1.363682542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:14:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:14:30.363324218 +0000 UTC m=+2289.960964588" watchObservedRunningTime="2026-07-21 02:14:30.363682542 +0000 UTC m=+2289.961322896"
Jul 21 02:14:31.355925 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:31.355883    2575 generic.go:356] "Generic (PLEG): container finished" podID="7262fb4c-c9f4-4d35-872c-293d15cd3e14" containerID="5ec5b5322bbc8ab1fbb5af370d5b643b608e9faf720aad5379da407611a3a930" exitCode=0
Jul 21 02:14:31.356346 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:31.355935    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-91b59b-nd48b" event={"ID":"7262fb4c-c9f4-4d35-872c-293d15cd3e14","Type":"ContainerDied","Data":"5ec5b5322bbc8ab1fbb5af370d5b643b608e9faf720aad5379da407611a3a930"}
Jul 21 02:14:32.487063 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:32.487039    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-91b59b-nd48b"
Jul 21 02:14:32.560188 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:32.560161    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7262fb4c-c9f4-4d35-872c-293d15cd3e14-maas-api-service-ca\") pod \"7262fb4c-c9f4-4d35-872c-293d15cd3e14\" (UID: \"7262fb4c-c9f4-4d35-872c-293d15cd3e14\") "
Jul 21 02:14:32.560501 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:32.560479    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7262fb4c-c9f4-4d35-872c-293d15cd3e14-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7262fb4c-c9f4-4d35-872c-293d15cd3e14" (UID: "7262fb4c-c9f4-4d35-872c-293d15cd3e14"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 02:14:32.660980 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:32.660902    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7262fb4c-c9f4-4d35-872c-293d15cd3e14-maas-api-service-ca\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:14:33.365816 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:33.365745    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-91b59b-nd48b"
Jul 21 02:14:33.365944 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:33.365741    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-91b59b-nd48b" event={"ID":"7262fb4c-c9f4-4d35-872c-293d15cd3e14","Type":"ContainerDied","Data":"1bea634313d8fdd0d2b3b32f7e1c0cc8d6fc36f82aea0f59ad29e3d2755de8bc"}
Jul 21 02:14:33.365944 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:33.365867    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bea634313d8fdd0d2b3b32f7e1c0cc8d6fc36f82aea0f59ad29e3d2755de8bc"
Jul 21 02:14:38.552675 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:38.552627    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:38.655534 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:38.655502    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:40.244958 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:40.244913    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:40.346898 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:40.346864    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:45.519212 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:45.519151    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:49.637373 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:49.637337    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-91b59b-nd48b"]
Jul 21 02:14:49.640546 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:49.640521    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-91b59b-nd48b"]
Jul 21 02:14:49.649124 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:49.649103    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:50.060569 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:50.060537    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:51.025438 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:51.025403    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7262fb4c-c9f4-4d35-872c-293d15cd3e14" path="/var/lib/kubelet/pods/7262fb4c-c9f4-4d35-872c-293d15cd3e14/volumes"
Jul 21 02:14:56.141360 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:56.141327    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:56.241083 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:56.241050    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:56.423467 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:56.423388    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:56.647421 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:56.647391    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:56.764505 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:56.764432    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-120127-bxw6p"]
Jul 21 02:14:56.764825 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:56.764812    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7262fb4c-c9f4-4d35-872c-293d15cd3e14" containerName="revoke-keys"
Jul 21 02:14:56.764876 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:56.764827    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7262fb4c-c9f4-4d35-872c-293d15cd3e14" containerName="revoke-keys"
Jul 21 02:14:56.764912 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:56.764904    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7262fb4c-c9f4-4d35-872c-293d15cd3e14" containerName="revoke-keys"
Jul 21 02:14:56.767910 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:56.767895    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-120127-bxw6p"
Jul 21 02:14:56.770128 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:56.770104    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x582z\""
Jul 21 02:14:56.770233 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:56.770131    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 21 02:14:56.771011 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:56.770994    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38edd61d-d756-463a-b800-04d1375cb0f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-120127-bxw6p\" (UID: \"38edd61d-d756-463a-b800-04d1375cb0f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-120127-bxw6p"
Jul 21 02:14:56.775113 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:56.775087    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-120127-bxw6p"]
Jul 21 02:14:56.871576 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:56.871540    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38edd61d-d756-463a-b800-04d1375cb0f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-120127-bxw6p\" (UID: \"38edd61d-d756-463a-b800-04d1375cb0f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-120127-bxw6p"
Jul 21 02:14:56.872257 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:56.872237    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38edd61d-d756-463a-b800-04d1375cb0f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-120127-bxw6p\" (UID: \"38edd61d-d756-463a-b800-04d1375cb0f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-120127-bxw6p"
Jul 21 02:14:57.079151 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:57.079125    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-120127-bxw6p"
Jul 21 02:14:57.210547 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:57.210523    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-120127-bxw6p"]
Jul 21 02:14:57.212434 ip-10-0-129-219 kubenswrapper[2575]: W0721 02:14:57.212405    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38edd61d_d756_463a_b800_04d1375cb0f8.slice/crio-fb03ab8f54734007126c47e9a2041ca0d8697a7262e90c7df1bf8e442531e155 WatchSource:0}: Error finding container fb03ab8f54734007126c47e9a2041ca0d8697a7262e90c7df1bf8e442531e155: Status 404 returned error can't find the container with id fb03ab8f54734007126c47e9a2041ca0d8697a7262e90c7df1bf8e442531e155
Jul 21 02:14:57.445780 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:57.445746    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:57.481520 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:57.481493    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-120127-bxw6p" event={"ID":"38edd61d-d756-463a-b800-04d1375cb0f8","Type":"ContainerStarted","Data":"799d2b1d74a05b4b846782437cf2df7b2922d75c5bccbb38753ca2633f6aec10"}
Jul 21 02:14:57.481648 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:57.481546    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-120127-bxw6p" event={"ID":"38edd61d-d756-463a-b800-04d1375cb0f8","Type":"ContainerStarted","Data":"fb03ab8f54734007126c47e9a2041ca0d8697a7262e90c7df1bf8e442531e155"}
Jul 21 02:14:57.499905 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:57.499859    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-120127-bxw6p" podStartSLOduration=1.499845612 podStartE2EDuration="1.499845612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:14:57.49912476 +0000 UTC m=+2317.096765129" watchObservedRunningTime="2026-07-21 02:14:57.499845612 +0000 UTC m=+2317.097485962"
Jul 21 02:14:57.541480 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:57.541456    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:14:58.487349 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:58.487310    2575 generic.go:356] "Generic (PLEG): container finished" podID="38edd61d-d756-463a-b800-04d1375cb0f8" containerID="799d2b1d74a05b4b846782437cf2df7b2922d75c5bccbb38753ca2633f6aec10" exitCode=0
Jul 21 02:14:58.487732 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:58.487360    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-120127-bxw6p" event={"ID":"38edd61d-d756-463a-b800-04d1375cb0f8","Type":"ContainerDied","Data":"799d2b1d74a05b4b846782437cf2df7b2922d75c5bccbb38753ca2633f6aec10"}
Jul 21 02:14:59.622574 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:59.622548    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-120127-bxw6p"
Jul 21 02:14:59.688451 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:59.688426    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38edd61d-d756-463a-b800-04d1375cb0f8-maas-api-service-ca\") pod \"38edd61d-d756-463a-b800-04d1375cb0f8\" (UID: \"38edd61d-d756-463a-b800-04d1375cb0f8\") "
Jul 21 02:14:59.688772 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:59.688751    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38edd61d-d756-463a-b800-04d1375cb0f8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "38edd61d-d756-463a-b800-04d1375cb0f8" (UID: "38edd61d-d756-463a-b800-04d1375cb0f8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 02:14:59.789339 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:14:59.789309    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38edd61d-d756-463a-b800-04d1375cb0f8-maas-api-service-ca\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:15:00.140686 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:00.140573    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29743335-szfh2"]
Jul 21 02:15:00.141002 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:00.140986    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38edd61d-d756-463a-b800-04d1375cb0f8" containerName="revoke-keys"
Jul 21 02:15:00.141050 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:00.141004    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="38edd61d-d756-463a-b800-04d1375cb0f8" containerName="revoke-keys"
Jul 21 02:15:00.141086 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:00.141070    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="38edd61d-d756-463a-b800-04d1375cb0f8" containerName="revoke-keys"
Jul 21 02:15:00.143949 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:00.143932    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-120127-29743335-pck6x"]
Jul 21 02:15:00.144061 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:00.144041    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743335-szfh2"
Jul 21 02:15:00.146967 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:00.146946    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-120127-29743335-pck6x"
Jul 21 02:15:00.153313 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:00.153287    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-120127-29743335-pck6x"]
Jul 21 02:15:00.156046 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:00.156026    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29743335-szfh2"]
Jul 21 02:15:00.156725 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:00.156707    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743335-szfh2"
Jul 21 02:15:00.163890 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:00.163874    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-120127-29743335-pck6x"
Jul 21 02:15:00.300802 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:00.300752    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-120127-29743335-pck6x"]
Jul 21 02:15:00.304904 ip-10-0-129-219 kubenswrapper[2575]: W0721 02:15:00.304875    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6291b22d_0eb3_4f2a_aa5a_0da93be3a543.slice/crio-e1714d939dd6c17e608e4829ac910b66ff31604a8014c9b04336e848fc950314 WatchSource:0}: Error finding container e1714d939dd6c17e608e4829ac910b66ff31604a8014c9b04336e848fc950314: Status 404 returned error can't find the container with id e1714d939dd6c17e608e4829ac910b66ff31604a8014c9b04336e848fc950314
Jul 21 02:15:00.322221 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:00.322193    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29743335-szfh2"]
Jul 21 02:15:00.323504 ip-10-0-129-219 kubenswrapper[2575]: W0721 02:15:00.323478    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f62600_525d_4655_9e47_d3ef36c9fcba.slice/crio-137e62a24449900d217cc452c94bebaa26a182853482dfa48825233eeb6a0d4f WatchSource:0}: Error finding container 137e62a24449900d217cc452c94bebaa26a182853482dfa48825233eeb6a0d4f: Status 404 returned error can't find the container with id 137e62a24449900d217cc452c94bebaa26a182853482dfa48825233eeb6a0d4f
Jul 21 02:15:00.497093 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:00.497025    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-120127-29743335-pck6x" event={"ID":"6291b22d-0eb3-4f2a-aa5a-0da93be3a543","Type":"ContainerStarted","Data":"e1714d939dd6c17e608e4829ac910b66ff31604a8014c9b04336e848fc950314"}
Jul 21 02:15:00.498333 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:00.498310    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-120127-bxw6p"
Jul 21 02:15:00.498454 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:00.498308    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-120127-bxw6p" event={"ID":"38edd61d-d756-463a-b800-04d1375cb0f8","Type":"ContainerDied","Data":"fb03ab8f54734007126c47e9a2041ca0d8697a7262e90c7df1bf8e442531e155"}
Jul 21 02:15:00.498454 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:00.498443    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb03ab8f54734007126c47e9a2041ca0d8697a7262e90c7df1bf8e442531e155"
Jul 21 02:15:00.499632 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:00.499594    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743335-szfh2" event={"ID":"77f62600-525d-4655-9e47-d3ef36c9fcba","Type":"ContainerStarted","Data":"137e62a24449900d217cc452c94bebaa26a182853482dfa48825233eeb6a0d4f"}
Jul 21 02:15:01.505940 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:01.505850    2575 generic.go:356] "Generic (PLEG): container finished" podID="77f62600-525d-4655-9e47-d3ef36c9fcba" containerID="7170af4291e2a9cf14df533298943f9ef571fcb8b5579ab4f7ab50408f211d74" exitCode=0
Jul 21 02:15:01.505940 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:01.505880    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743335-szfh2" event={"ID":"77f62600-525d-4655-9e47-d3ef36c9fcba","Type":"ContainerDied","Data":"7170af4291e2a9cf14df533298943f9ef571fcb8b5579ab4f7ab50408f211d74"}
Jul 21 02:15:01.513356 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:01.513325    2575 generic.go:356] "Generic (PLEG): container finished" podID="6291b22d-0eb3-4f2a-aa5a-0da93be3a543" containerID="ce84ca459bc4fd3e2472aa6c7a605f29fe6494b6030d9b2f0a9181e9bd748a35" exitCode=0
Jul 21 02:15:01.513508 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:01.513409    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-120127-29743335-pck6x" event={"ID":"6291b22d-0eb3-4f2a-aa5a-0da93be3a543","Type":"ContainerDied","Data":"ce84ca459bc4fd3e2472aa6c7a605f29fe6494b6030d9b2f0a9181e9bd748a35"}
Jul 21 02:15:02.654063 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:02.654036    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743335-szfh2"
Jul 21 02:15:02.656655 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:02.656600    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-120127-29743335-pck6x"
Jul 21 02:15:03.523325 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:03.523296    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743335-szfh2"
Jul 21 02:15:03.523325 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:03.523314    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743335-szfh2" event={"ID":"77f62600-525d-4655-9e47-d3ef36c9fcba","Type":"ContainerDied","Data":"137e62a24449900d217cc452c94bebaa26a182853482dfa48825233eeb6a0d4f"}
Jul 21 02:15:03.523563 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:03.523361    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="137e62a24449900d217cc452c94bebaa26a182853482dfa48825233eeb6a0d4f"
Jul 21 02:15:03.524494 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:03.524475    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-120127-29743335-pck6x"
Jul 21 02:15:03.524605 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:03.524494    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-120127-29743335-pck6x" event={"ID":"6291b22d-0eb3-4f2a-aa5a-0da93be3a543","Type":"ContainerDied","Data":"e1714d939dd6c17e608e4829ac910b66ff31604a8014c9b04336e848fc950314"}
Jul 21 02:15:03.524605 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:03.524530    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1714d939dd6c17e608e4829ac910b66ff31604a8014c9b04336e848fc950314"
Jul 21 02:15:12.282668 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:12.282569    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-120127-29743335-pck6x"]
Jul 21 02:15:12.287368 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:12.287338    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-120127-29743335-pck6x"]
Jul 21 02:15:12.287902 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:12.287883    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:15:13.025892 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:13.025854    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6291b22d-0eb3-4f2a-aa5a-0da93be3a543" path="/var/lib/kubelet/pods/6291b22d-0eb3-4f2a-aa5a-0da93be3a543/volumes"
Jul 21 02:15:16.848692 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:16.848658    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-120127-bxw6p"]
Jul 21 02:15:16.851423 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:16.851400    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-120127-bxw6p"]
Jul 21 02:15:16.853970 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:16.853947    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:15:17.026150 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:17.026119    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38edd61d-d756-463a-b800-04d1375cb0f8" path="/var/lib/kubelet/pods/38edd61d-d756-463a-b800-04d1375cb0f8/volumes"
Jul 21 02:15:17.266809 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:17.266728    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:15:21.787778 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:21.787740    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:15:21.888575 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:21.888538    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:15:23.123638 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:23.123578    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:15:23.246702 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:23.246669    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:15:23.402138 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:23.402043    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:15:23.634919 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:23.634891    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:15:24.008757 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:24.008723    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:15:24.349493 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:24.349465    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:15:24.660738 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:24.660657    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:15:27.213446 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:27.213411    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:15:27.304497 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:27.304465    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:15:27.385133 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:27.385105    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:15:27.552379 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:27.552347    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:15:27.891917 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:27.891839    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:15:28.016657 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:28.016603    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:15:31.721132 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:31.721084    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:15:32.049261 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:32.049227    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:15:32.390221 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:32.390139    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:15:32.653191 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:32.653099    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:15:35.213113 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:35.213082    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:15:35.302121 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:35.302086    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:15:35.387671 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:35.387638    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:15:35.482141 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:35.482061    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:15:35.948986 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:35.948949    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:15:36.052360 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:36.052322    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:15:39.749924 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:39.749892    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:15:48.982462 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:48.982427    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:15:49.084086 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:49.084053    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:15:51.591366 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:51.591334    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46"]
Jul 21 02:15:51.591748 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:51.591702    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6291b22d-0eb3-4f2a-aa5a-0da93be3a543" containerName="cleanup"
Jul 21 02:15:51.591748 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:51.591715    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6291b22d-0eb3-4f2a-aa5a-0da93be3a543" containerName="cleanup"
Jul 21 02:15:51.591748 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:51.591731    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77f62600-525d-4655-9e47-d3ef36c9fcba" containerName="cleanup"
Jul 21 02:15:51.591748 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:51.591737    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="77f62600-525d-4655-9e47-d3ef36c9fcba" containerName="cleanup"
Jul 21 02:15:51.591878 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:51.591790    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6291b22d-0eb3-4f2a-aa5a-0da93be3a543" containerName="cleanup"
Jul 21 02:15:51.591878 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:51.591803    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="77f62600-525d-4655-9e47-d3ef36c9fcba" containerName="cleanup"
Jul 21 02:15:51.594928 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:51.594909    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46"
Jul 21 02:15:51.597369 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:51.597348    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-748a4d\"/\"default-dockercfg-ks6mr\""
Jul 21 02:15:51.598773 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:51.598753    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-748a4d\"/\"kube-root-ca.crt\""
Jul 21 02:15:51.598889 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:51.598811    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-748a4d\"/\"auth-test-model-748a4d-kserve-self-signed-certs\""
Jul 21 02:15:51.598889 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:51.598866    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-748a4d\"/\"openshift-service-ca.crt\""
Jul 21 02:15:51.610048 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:51.610027    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46"]
Jul 21 02:15:51.627412 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:51.627382    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cfcj\" (UniqueName: \"kubernetes.io/projected/3d53d86d-4b5e-415b-bb7d-805078a19d17-kube-api-access-2cfcj\") pod \"auth-test-model-748a4d-kserve-69cf6f6d95-4mj46\" (UID: \"3d53d86d-4b5e-415b-bb7d-805078a19d17\") " pod="ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46"
Jul 21 02:15:51.627510 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:51.627429    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d53d86d-4b5e-415b-bb7d-805078a19d17-tmp-dir\") pod \"auth-test-model-748a4d-kserve-69cf6f6d95-4mj46\" (UID: \"3d53d86d-4b5e-415b-bb7d-805078a19d17\") " pod="ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46"
Jul 21 02:15:51.627510 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:51.627484    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d53d86d-4b5e-415b-bb7d-805078a19d17-home\") pod \"auth-test-model-748a4d-kserve-69cf6f6d95-4mj46\" (UID: \"3d53d86d-4b5e-415b-bb7d-805078a19d17\") " pod="ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46"
Jul 21 02:15:51.627585 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:51.627518    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d53d86d-4b5e-415b-bb7d-805078a19d17-tls-certs\") pod \"auth-test-model-748a4d-kserve-69cf6f6d95-4mj46\" (UID: \"3d53d86d-4b5e-415b-bb7d-805078a19d17\") " pod="ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46"
Jul 21 02:15:51.627585 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:51.627548    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d53d86d-4b5e-415b-bb7d-805078a19d17-model-cache\") pod \"auth-test-model-748a4d-kserve-69cf6f6d95-4mj46\" (UID: \"3d53d86d-4b5e-415b-bb7d-805078a19d17\") " pod="ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46"
Jul 21 02:15:51.627585 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:51.627574    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d53d86d-4b5e-415b-bb7d-805078a19d17-dshm\") pod \"auth-test-model-748a4d-kserve-69cf6f6d95-4mj46\" (UID: \"3d53d86d-4b5e-415b-bb7d-805078a19d17\") " pod="ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46"
Jul 21 02:15:51.633265 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:51.633237    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:15:51.728369 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:51.728338    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d53d86d-4b5e-415b-bb7d-805078a19d17-home\") pod \"auth-test-model-748a4d-kserve-69cf6f6d95-4mj46\" (UID: \"3d53d86d-4b5e-415b-bb7d-805078a19d17\") " pod="ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46"
Jul 21 02:15:51.728369 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:51.728374    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d53d86d-4b5e-415b-bb7d-805078a19d17-tls-certs\") pod \"auth-test-model-748a4d-kserve-69cf6f6d95-4mj46\" (UID: \"3d53d86d-4b5e-415b-bb7d-805078a19d17\") " pod="ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46"
Jul 21 02:15:51.728571 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:51.728396    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d53d86d-4b5e-415b-bb7d-805078a19d17-model-cache\") pod \"auth-test-model-748a4d-kserve-69cf6f6d95-4mj46\" (UID: \"3d53d86d-4b5e-415b-bb7d-805078a19d17\") " pod="ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46"
Jul 21 02:15:51.728571 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:51.728413    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d53d86d-4b5e-415b-bb7d-805078a19d17-dshm\") pod \"auth-test-model-748a4d-kserve-69cf6f6d95-4mj46\" (UID: \"3d53d86d-4b5e-415b-bb7d-805078a19d17\") " pod="ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46"
Jul 21 02:15:51.728571 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:51.728453    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cfcj\" (UniqueName: \"kubernetes.io/projected/3d53d86d-4b5e-415b-bb7d-805078a19d17-kube-api-access-2cfcj\") pod \"auth-test-model-748a4d-kserve-69cf6f6d95-4mj46\" (UID: \"3d53d86d-4b5e-415b-bb7d-805078a19d17\") " pod="ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46"
Jul 21 02:15:51.728571 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:51.728485    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d53d86d-4b5e-415b-bb7d-805078a19d17-tmp-dir\") pod \"auth-test-model-748a4d-kserve-69cf6f6d95-4mj46\" (UID: \"3d53d86d-4b5e-415b-bb7d-805078a19d17\") " pod="ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46"
Jul 21 02:15:51.728876 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:51.728822    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d53d86d-4b5e-415b-bb7d-805078a19d17-home\") pod \"auth-test-model-748a4d-kserve-69cf6f6d95-4mj46\" (UID: \"3d53d86d-4b5e-415b-bb7d-805078a19d17\") " pod="ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46"
Jul 21 02:15:51.728876 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:51.728861    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d53d86d-4b5e-415b-bb7d-805078a19d17-model-cache\") pod \"auth-test-model-748a4d-kserve-69cf6f6d95-4mj46\" (UID: \"3d53d86d-4b5e-415b-bb7d-805078a19d17\") " pod="ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46"
Jul 21 02:15:51.728992 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:51.728905    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d53d86d-4b5e-415b-bb7d-805078a19d17-tmp-dir\") pod \"auth-test-model-748a4d-kserve-69cf6f6d95-4mj46\" (UID: \"3d53d86d-4b5e-415b-bb7d-805078a19d17\") " pod="ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46"
Jul 21 02:15:51.730707 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:51.730684    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d53d86d-4b5e-415b-bb7d-805078a19d17-dshm\") pod \"auth-test-model-748a4d-kserve-69cf6f6d95-4mj46\" (UID: \"3d53d86d-4b5e-415b-bb7d-805078a19d17\") " pod="ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46"
Jul 21 02:15:51.730927 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:51.730907    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d53d86d-4b5e-415b-bb7d-805078a19d17-tls-certs\") pod \"auth-test-model-748a4d-kserve-69cf6f6d95-4mj46\" (UID: \"3d53d86d-4b5e-415b-bb7d-805078a19d17\") " pod="ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46"
Jul 21 02:15:51.735576 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:51.735554    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cfcj\" (UniqueName: \"kubernetes.io/projected/3d53d86d-4b5e-415b-bb7d-805078a19d17-kube-api-access-2cfcj\") pod \"auth-test-model-748a4d-kserve-69cf6f6d95-4mj46\" (UID: \"3d53d86d-4b5e-415b-bb7d-805078a19d17\") " pod="ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46"
Jul 21 02:15:51.904798 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:51.904715    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46"
Jul 21 02:15:52.033673 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:52.033638    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46"]
Jul 21 02:15:52.036529 ip-10-0-129-219 kubenswrapper[2575]: W0721 02:15:52.036498    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d53d86d_4b5e_415b_bb7d_805078a19d17.slice/crio-248715ca45c9f0a19cfd34a8e843067b549e4ce26e91d875c546793cad385513 WatchSource:0}: Error finding container 248715ca45c9f0a19cfd34a8e843067b549e4ce26e91d875c546793cad385513: Status 404 returned error can't find the container with id 248715ca45c9f0a19cfd34a8e843067b549e4ce26e91d875c546793cad385513
Jul 21 02:15:52.756091 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:52.756058    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46" event={"ID":"3d53d86d-4b5e-415b-bb7d-805078a19d17","Type":"ContainerStarted","Data":"f81dab4f9248ee184f3d52de5163151536df6566a59e9c75cbfe24be65eda8cd"}
Jul 21 02:15:52.756091 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:52.756097    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46" event={"ID":"3d53d86d-4b5e-415b-bb7d-805078a19d17","Type":"ContainerStarted","Data":"248715ca45c9f0a19cfd34a8e843067b549e4ce26e91d875c546793cad385513"}
Jul 21 02:15:52.772843 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:52.772787    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46" podStartSLOduration=1.77276644 podStartE2EDuration="1.77276644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:15:52.772380601 +0000 UTC m=+2372.370020947" watchObservedRunningTime="2026-07-21 02:15:52.77276644 +0000 UTC m=+2372.370406792"
Jul 21 02:15:56.758984 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:56.758928    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:15:56.858130 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:15:56.858088    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:16:01.905708 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:01.905670    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46"
Jul 21 02:16:01.906168 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:01.905722    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46"
Jul 21 02:16:01.918840 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:01.918811    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46"
Jul 21 02:16:02.817966 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:02.817937    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46"
Jul 21 02:16:21.199796 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:21.199762    2575 scope.go:117] "RemoveContainer" containerID="8b1b7c70a54434cfe6bffd9980cfd85ab8c9bf022962a7de0a0ef2883258b9e6"
Jul 21 02:16:32.605034 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:32.605000    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n"]
Jul 21 02:16:32.608291 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:32.608274    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n"
Jul 21 02:16:32.610743 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:32.610722    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c6538a\"/\"kube-root-ca.crt\""
Jul 21 02:16:32.611473 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:32.611449    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c6538a\"/\"default-dockercfg-zpbgz\""
Jul 21 02:16:32.611473 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:32.611468    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c6538a\"/\"openshift-service-ca.crt\""
Jul 21 02:16:32.611606 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:32.611465    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c6538a\"/\"auth-test-model-c6538a-kserve-self-signed-certs\""
Jul 21 02:16:32.622741 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:32.622715    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n"]
Jul 21 02:16:32.648594 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:32.648562    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:16:32.794485 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:32.794457    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d91a3c26-d335-4889-ab81-374be282ef3e-dshm\") pod \"auth-test-model-c6538a-kserve-5b59f74cf-fqz2n\" (UID: \"d91a3c26-d335-4889-ab81-374be282ef3e\") " pod="ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n"
Jul 21 02:16:32.794485 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:32.794489    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d91a3c26-d335-4889-ab81-374be282ef3e-tls-certs\") pod \"auth-test-model-c6538a-kserve-5b59f74cf-fqz2n\" (UID: \"d91a3c26-d335-4889-ab81-374be282ef3e\") " pod="ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n"
Jul 21 02:16:32.794769 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:32.794524    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d91a3c26-d335-4889-ab81-374be282ef3e-tmp-dir\") pod \"auth-test-model-c6538a-kserve-5b59f74cf-fqz2n\" (UID: \"d91a3c26-d335-4889-ab81-374be282ef3e\") " pod="ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n"
Jul 21 02:16:32.794769 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:32.794599    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d91a3c26-d335-4889-ab81-374be282ef3e-home\") pod \"auth-test-model-c6538a-kserve-5b59f74cf-fqz2n\" (UID: \"d91a3c26-d335-4889-ab81-374be282ef3e\") " pod="ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n"
Jul 21 02:16:32.794769 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:32.794696    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74qj2\" (UniqueName: \"kubernetes.io/projected/d91a3c26-d335-4889-ab81-374be282ef3e-kube-api-access-74qj2\") pod \"auth-test-model-c6538a-kserve-5b59f74cf-fqz2n\" (UID: \"d91a3c26-d335-4889-ab81-374be282ef3e\") " pod="ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n"
Jul 21 02:16:32.794769 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:32.794730    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d91a3c26-d335-4889-ab81-374be282ef3e-model-cache\") pod \"auth-test-model-c6538a-kserve-5b59f74cf-fqz2n\" (UID: \"d91a3c26-d335-4889-ab81-374be282ef3e\") " pod="ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n"
Jul 21 02:16:32.895903 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:32.895808    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d91a3c26-d335-4889-ab81-374be282ef3e-dshm\") pod \"auth-test-model-c6538a-kserve-5b59f74cf-fqz2n\" (UID: \"d91a3c26-d335-4889-ab81-374be282ef3e\") " pod="ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n"
Jul 21 02:16:32.895903 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:32.895847    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d91a3c26-d335-4889-ab81-374be282ef3e-tls-certs\") pod \"auth-test-model-c6538a-kserve-5b59f74cf-fqz2n\" (UID: \"d91a3c26-d335-4889-ab81-374be282ef3e\") " pod="ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n"
Jul 21 02:16:32.895903 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:32.895891    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d91a3c26-d335-4889-ab81-374be282ef3e-tmp-dir\") pod \"auth-test-model-c6538a-kserve-5b59f74cf-fqz2n\" (UID: \"d91a3c26-d335-4889-ab81-374be282ef3e\") " pod="ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n"
Jul 21 02:16:32.896161 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:32.895922    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d91a3c26-d335-4889-ab81-374be282ef3e-home\") pod \"auth-test-model-c6538a-kserve-5b59f74cf-fqz2n\" (UID: \"d91a3c26-d335-4889-ab81-374be282ef3e\") " pod="ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n"
Jul 21 02:16:32.896161 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:32.895964    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74qj2\" (UniqueName: \"kubernetes.io/projected/d91a3c26-d335-4889-ab81-374be282ef3e-kube-api-access-74qj2\") pod \"auth-test-model-c6538a-kserve-5b59f74cf-fqz2n\" (UID: \"d91a3c26-d335-4889-ab81-374be282ef3e\") " pod="ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n"
Jul 21 02:16:32.896161 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:32.895991    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d91a3c26-d335-4889-ab81-374be282ef3e-model-cache\") pod \"auth-test-model-c6538a-kserve-5b59f74cf-fqz2n\" (UID: \"d91a3c26-d335-4889-ab81-374be282ef3e\") " pod="ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n"
Jul 21 02:16:32.896326 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:32.896305    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d91a3c26-d335-4889-ab81-374be282ef3e-tmp-dir\") pod \"auth-test-model-c6538a-kserve-5b59f74cf-fqz2n\" (UID: \"d91a3c26-d335-4889-ab81-374be282ef3e\") " pod="ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n"
Jul 21 02:16:32.896394 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:32.896372    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d91a3c26-d335-4889-ab81-374be282ef3e-home\") pod \"auth-test-model-c6538a-kserve-5b59f74cf-fqz2n\" (UID: \"d91a3c26-d335-4889-ab81-374be282ef3e\") " pod="ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n"
Jul 21 02:16:32.896449 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:32.896406    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d91a3c26-d335-4889-ab81-374be282ef3e-model-cache\") pod \"auth-test-model-c6538a-kserve-5b59f74cf-fqz2n\" (UID: \"d91a3c26-d335-4889-ab81-374be282ef3e\") " pod="ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n"
Jul 21 02:16:32.898170 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:32.898142    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d91a3c26-d335-4889-ab81-374be282ef3e-dshm\") pod \"auth-test-model-c6538a-kserve-5b59f74cf-fqz2n\" (UID: \"d91a3c26-d335-4889-ab81-374be282ef3e\") " pod="ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n"
Jul 21 02:16:32.898445 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:32.898427    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d91a3c26-d335-4889-ab81-374be282ef3e-tls-certs\") pod \"auth-test-model-c6538a-kserve-5b59f74cf-fqz2n\" (UID: \"d91a3c26-d335-4889-ab81-374be282ef3e\") " pod="ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n"
Jul 21 02:16:32.904739 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:32.904721    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74qj2\" (UniqueName: \"kubernetes.io/projected/d91a3c26-d335-4889-ab81-374be282ef3e-kube-api-access-74qj2\") pod \"auth-test-model-c6538a-kserve-5b59f74cf-fqz2n\" (UID: \"d91a3c26-d335-4889-ab81-374be282ef3e\") " pod="ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n"
Jul 21 02:16:32.916706 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:32.916681    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n"
Jul 21 02:16:33.050313 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:33.050287    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n"]
Jul 21 02:16:33.053044 ip-10-0-129-219 kubenswrapper[2575]: W0721 02:16:33.053020    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd91a3c26_d335_4889_ab81_374be282ef3e.slice/crio-4558899a76d3a9e828154fe82f8aa6625eb41ff5ffb9144973629560edf3e6a6 WatchSource:0}: Error finding container 4558899a76d3a9e828154fe82f8aa6625eb41ff5ffb9144973629560edf3e6a6: Status 404 returned error can't find the container with id 4558899a76d3a9e828154fe82f8aa6625eb41ff5ffb9144973629560edf3e6a6
Jul 21 02:16:33.963716 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:33.963681    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n" event={"ID":"d91a3c26-d335-4889-ab81-374be282ef3e","Type":"ContainerStarted","Data":"4be4a4d544c4ba3607a9f54afbeb7578ac80019b3717c68a0e8278180c34f705"}
Jul 21 02:16:33.963716 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:33.963724    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n" event={"ID":"d91a3c26-d335-4889-ab81-374be282ef3e","Type":"ContainerStarted","Data":"4558899a76d3a9e828154fe82f8aa6625eb41ff5ffb9144973629560edf3e6a6"}
Jul 21 02:16:33.978773 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:33.978722    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n" podStartSLOduration=1.97870511 podStartE2EDuration="1.97870511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:16:33.978470402 +0000 UTC m=+2413.576110753" watchObservedRunningTime="2026-07-21 02:16:33.97870511 +0000 UTC m=+2413.576345461"
Jul 21 02:16:42.917368 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:42.917261    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n"
Jul 21 02:16:42.917368 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:42.917329    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n"
Jul 21 02:16:42.929979 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:42.929954    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n"
Jul 21 02:16:43.017039 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:16:43.017014    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n"
Jul 21 02:18:17.349550 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:17.349500    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:18:38.180583 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:38.180551    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46"]
Jul 21 02:18:38.201713 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:38.201672    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:18:39.127840 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:39.127806    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n"]
Jul 21 02:18:39.150971 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:39.150936    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:18:52.403024 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:52.402986    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9"]
Jul 21 02:18:52.406368 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:52.406351    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9"
Jul 21 02:18:52.408543 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:52.408523    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-748a4d\"/\"sub-test-model-748a4d-kserve-self-signed-certs\""
Jul 21 02:18:52.414957 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:52.414932    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9"]
Jul 21 02:18:52.490249 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:52.490213    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:18:52.517729 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:52.517691    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4cd3379-3258-44ad-a737-cbe411468dd6-tls-certs\") pod \"sub-test-model-748a4d-kserve-786c89cd8c-858q9\" (UID: \"d4cd3379-3258-44ad-a737-cbe411468dd6\") " pod="ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9"
Jul 21 02:18:52.517900 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:52.517737    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pskl6\" (UniqueName: \"kubernetes.io/projected/d4cd3379-3258-44ad-a737-cbe411468dd6-kube-api-access-pskl6\") pod \"sub-test-model-748a4d-kserve-786c89cd8c-858q9\" (UID: \"d4cd3379-3258-44ad-a737-cbe411468dd6\") " pod="ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9"
Jul 21 02:18:52.517900 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:52.517762    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4cd3379-3258-44ad-a737-cbe411468dd6-dshm\") pod \"sub-test-model-748a4d-kserve-786c89cd8c-858q9\" (UID: \"d4cd3379-3258-44ad-a737-cbe411468dd6\") " pod="ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9"
Jul 21 02:18:52.517900 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:52.517783    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4cd3379-3258-44ad-a737-cbe411468dd6-tmp-dir\") pod \"sub-test-model-748a4d-kserve-786c89cd8c-858q9\" (UID: \"d4cd3379-3258-44ad-a737-cbe411468dd6\") " pod="ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9"
Jul 21 02:18:52.517900 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:52.517817    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4cd3379-3258-44ad-a737-cbe411468dd6-model-cache\") pod \"sub-test-model-748a4d-kserve-786c89cd8c-858q9\" (UID: \"d4cd3379-3258-44ad-a737-cbe411468dd6\") " pod="ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9"
Jul 21 02:18:52.518071 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:52.517898    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4cd3379-3258-44ad-a737-cbe411468dd6-home\") pod \"sub-test-model-748a4d-kserve-786c89cd8c-858q9\" (UID: \"d4cd3379-3258-44ad-a737-cbe411468dd6\") " pod="ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9"
Jul 21 02:18:52.619127 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:52.619092    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4cd3379-3258-44ad-a737-cbe411468dd6-tls-certs\") pod \"sub-test-model-748a4d-kserve-786c89cd8c-858q9\" (UID: \"d4cd3379-3258-44ad-a737-cbe411468dd6\") " pod="ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9"
Jul 21 02:18:52.619127 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:52.619133    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pskl6\" (UniqueName: \"kubernetes.io/projected/d4cd3379-3258-44ad-a737-cbe411468dd6-kube-api-access-pskl6\") pod \"sub-test-model-748a4d-kserve-786c89cd8c-858q9\" (UID: \"d4cd3379-3258-44ad-a737-cbe411468dd6\") " pod="ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9"
Jul 21 02:18:52.619372 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:52.619153    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4cd3379-3258-44ad-a737-cbe411468dd6-dshm\") pod \"sub-test-model-748a4d-kserve-786c89cd8c-858q9\" (UID: \"d4cd3379-3258-44ad-a737-cbe411468dd6\") " pod="ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9"
Jul 21 02:18:52.619372 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:52.619173    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4cd3379-3258-44ad-a737-cbe411468dd6-tmp-dir\") pod \"sub-test-model-748a4d-kserve-786c89cd8c-858q9\" (UID: \"d4cd3379-3258-44ad-a737-cbe411468dd6\") " pod="ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9"
Jul 21 02:18:52.619372 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:52.619208    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4cd3379-3258-44ad-a737-cbe411468dd6-model-cache\") pod \"sub-test-model-748a4d-kserve-786c89cd8c-858q9\" (UID: \"d4cd3379-3258-44ad-a737-cbe411468dd6\") " pod="ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9"
Jul 21 02:18:52.619372 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:52.619245    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4cd3379-3258-44ad-a737-cbe411468dd6-home\") pod \"sub-test-model-748a4d-kserve-786c89cd8c-858q9\" (UID: \"d4cd3379-3258-44ad-a737-cbe411468dd6\") " pod="ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9"
Jul 21 02:18:52.619666 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:52.619646    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4cd3379-3258-44ad-a737-cbe411468dd6-home\") pod \"sub-test-model-748a4d-kserve-786c89cd8c-858q9\" (UID: \"d4cd3379-3258-44ad-a737-cbe411468dd6\") " pod="ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9"
Jul 21 02:18:52.619747 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:52.619710    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4cd3379-3258-44ad-a737-cbe411468dd6-tmp-dir\") pod \"sub-test-model-748a4d-kserve-786c89cd8c-858q9\" (UID: \"d4cd3379-3258-44ad-a737-cbe411468dd6\") " pod="ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9"
Jul 21 02:18:52.619790 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:52.619742    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4cd3379-3258-44ad-a737-cbe411468dd6-model-cache\") pod \"sub-test-model-748a4d-kserve-786c89cd8c-858q9\" (UID: \"d4cd3379-3258-44ad-a737-cbe411468dd6\") " pod="ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9"
Jul 21 02:18:52.621459 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:52.621437    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4cd3379-3258-44ad-a737-cbe411468dd6-dshm\") pod \"sub-test-model-748a4d-kserve-786c89cd8c-858q9\" (UID: \"d4cd3379-3258-44ad-a737-cbe411468dd6\") " pod="ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9"
Jul 21 02:18:52.621727 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:52.621708    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4cd3379-3258-44ad-a737-cbe411468dd6-tls-certs\") pod \"sub-test-model-748a4d-kserve-786c89cd8c-858q9\" (UID: \"d4cd3379-3258-44ad-a737-cbe411468dd6\") " pod="ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9"
Jul 21 02:18:52.627134 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:52.627112    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pskl6\" (UniqueName: \"kubernetes.io/projected/d4cd3379-3258-44ad-a737-cbe411468dd6-kube-api-access-pskl6\") pod \"sub-test-model-748a4d-kserve-786c89cd8c-858q9\" (UID: \"d4cd3379-3258-44ad-a737-cbe411468dd6\") " pod="ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9"
Jul 21 02:18:52.716508 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:52.716444    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9"
Jul 21 02:18:52.845265 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:52.845240    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9"]
Jul 21 02:18:52.847290 ip-10-0-129-219 kubenswrapper[2575]: W0721 02:18:52.847258    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4cd3379_3258_44ad_a737_cbe411468dd6.slice/crio-17e5c415888d59cfa6181840f7a82d1cff67d5d92843664838f6b28ef0edec7c WatchSource:0}: Error finding container 17e5c415888d59cfa6181840f7a82d1cff67d5d92843664838f6b28ef0edec7c: Status 404 returned error can't find the container with id 17e5c415888d59cfa6181840f7a82d1cff67d5d92843664838f6b28ef0edec7c
Jul 21 02:18:53.200697 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:53.200590    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46" podUID="3d53d86d-4b5e-415b-bb7d-805078a19d17" containerName="main" containerID="cri-o://f81dab4f9248ee184f3d52de5163151536df6566a59e9c75cbfe24be65eda8cd" gracePeriod=45
Jul 21 02:18:53.436850 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:53.436826    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46"
Jul 21 02:18:53.528745 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:53.528723    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d53d86d-4b5e-415b-bb7d-805078a19d17-tmp-dir\") pod \"3d53d86d-4b5e-415b-bb7d-805078a19d17\" (UID: \"3d53d86d-4b5e-415b-bb7d-805078a19d17\") "
Jul 21 02:18:53.528930 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:53.528764    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d53d86d-4b5e-415b-bb7d-805078a19d17-home\") pod \"3d53d86d-4b5e-415b-bb7d-805078a19d17\" (UID: \"3d53d86d-4b5e-415b-bb7d-805078a19d17\") "
Jul 21 02:18:53.528930 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:53.528798    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d53d86d-4b5e-415b-bb7d-805078a19d17-tls-certs\") pod \"3d53d86d-4b5e-415b-bb7d-805078a19d17\" (UID: \"3d53d86d-4b5e-415b-bb7d-805078a19d17\") "
Jul 21 02:18:53.528930 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:53.528833    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d53d86d-4b5e-415b-bb7d-805078a19d17-dshm\") pod \"3d53d86d-4b5e-415b-bb7d-805078a19d17\" (UID: \"3d53d86d-4b5e-415b-bb7d-805078a19d17\") "
Jul 21 02:18:53.528930 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:53.528907    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cfcj\" (UniqueName: \"kubernetes.io/projected/3d53d86d-4b5e-415b-bb7d-805078a19d17-kube-api-access-2cfcj\") pod \"3d53d86d-4b5e-415b-bb7d-805078a19d17\" (UID: \"3d53d86d-4b5e-415b-bb7d-805078a19d17\") "
Jul 21 02:18:53.529151 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:53.528935    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d53d86d-4b5e-415b-bb7d-805078a19d17-model-cache\") pod \"3d53d86d-4b5e-415b-bb7d-805078a19d17\" (UID: \"3d53d86d-4b5e-415b-bb7d-805078a19d17\") "
Jul 21 02:18:53.529151 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:53.529000    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d53d86d-4b5e-415b-bb7d-805078a19d17-home" (OuterVolumeSpecName: "home") pod "3d53d86d-4b5e-415b-bb7d-805078a19d17" (UID: "3d53d86d-4b5e-415b-bb7d-805078a19d17"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:18:53.529151 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:53.529009    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d53d86d-4b5e-415b-bb7d-805078a19d17-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3d53d86d-4b5e-415b-bb7d-805078a19d17" (UID: "3d53d86d-4b5e-415b-bb7d-805078a19d17"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:18:53.529279 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:53.529198    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d53d86d-4b5e-415b-bb7d-805078a19d17-tmp-dir\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:18:53.529279 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:53.529217    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d53d86d-4b5e-415b-bb7d-805078a19d17-home\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:18:53.529279 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:53.529222    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d53d86d-4b5e-415b-bb7d-805078a19d17-model-cache" (OuterVolumeSpecName: "model-cache") pod "3d53d86d-4b5e-415b-bb7d-805078a19d17" (UID: "3d53d86d-4b5e-415b-bb7d-805078a19d17"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:18:53.531004 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:53.530983    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d53d86d-4b5e-415b-bb7d-805078a19d17-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d53d86d-4b5e-415b-bb7d-805078a19d17" (UID: "3d53d86d-4b5e-415b-bb7d-805078a19d17"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 02:18:53.531089 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:53.531013    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d53d86d-4b5e-415b-bb7d-805078a19d17-dshm" (OuterVolumeSpecName: "dshm") pod "3d53d86d-4b5e-415b-bb7d-805078a19d17" (UID: "3d53d86d-4b5e-415b-bb7d-805078a19d17"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:18:53.531089 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:53.531030    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d53d86d-4b5e-415b-bb7d-805078a19d17-kube-api-access-2cfcj" (OuterVolumeSpecName: "kube-api-access-2cfcj") pod "3d53d86d-4b5e-415b-bb7d-805078a19d17" (UID: "3d53d86d-4b5e-415b-bb7d-805078a19d17"). InnerVolumeSpecName "kube-api-access-2cfcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 02:18:53.629834 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:53.629810    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d53d86d-4b5e-415b-bb7d-805078a19d17-tls-certs\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:18:53.629947 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:53.629838    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d53d86d-4b5e-415b-bb7d-805078a19d17-dshm\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:18:53.629947 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:53.629849    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cfcj\" (UniqueName: \"kubernetes.io/projected/3d53d86d-4b5e-415b-bb7d-805078a19d17-kube-api-access-2cfcj\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:18:53.629947 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:53.629858    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d53d86d-4b5e-415b-bb7d-805078a19d17-model-cache\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:18:53.633782 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:53.633754    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9" event={"ID":"d4cd3379-3258-44ad-a737-cbe411468dd6","Type":"ContainerStarted","Data":"af6eba8a0111146fa592a361092bba2cfe8a60afba0361ff22f614ee4d1e4ca9"}
Jul 21 02:18:53.633894 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:53.633798    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9" event={"ID":"d4cd3379-3258-44ad-a737-cbe411468dd6","Type":"ContainerStarted","Data":"17e5c415888d59cfa6181840f7a82d1cff67d5d92843664838f6b28ef0edec7c"}
Jul 21 02:18:53.635093 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:53.635071    2575 generic.go:356] "Generic (PLEG): container finished" podID="3d53d86d-4b5e-415b-bb7d-805078a19d17" containerID="f81dab4f9248ee184f3d52de5163151536df6566a59e9c75cbfe24be65eda8cd" exitCode=0
Jul 21 02:18:53.635177 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:53.635113    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46"
Jul 21 02:18:53.635177 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:53.635142    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46" event={"ID":"3d53d86d-4b5e-415b-bb7d-805078a19d17","Type":"ContainerDied","Data":"f81dab4f9248ee184f3d52de5163151536df6566a59e9c75cbfe24be65eda8cd"}
Jul 21 02:18:53.635177 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:53.635167    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46" event={"ID":"3d53d86d-4b5e-415b-bb7d-805078a19d17","Type":"ContainerDied","Data":"248715ca45c9f0a19cfd34a8e843067b549e4ce26e91d875c546793cad385513"}
Jul 21 02:18:53.635334 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:53.635184    2575 scope.go:117] "RemoveContainer" containerID="f81dab4f9248ee184f3d52de5163151536df6566a59e9c75cbfe24be65eda8cd"
Jul 21 02:18:53.644033 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:53.644017    2575 scope.go:117] "RemoveContainer" containerID="f81dab4f9248ee184f3d52de5163151536df6566a59e9c75cbfe24be65eda8cd"
Jul 21 02:18:53.644272 ip-10-0-129-219 kubenswrapper[2575]: E0721 02:18:53.644252    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f81dab4f9248ee184f3d52de5163151536df6566a59e9c75cbfe24be65eda8cd\": container with ID starting with f81dab4f9248ee184f3d52de5163151536df6566a59e9c75cbfe24be65eda8cd not found: ID does not exist" containerID="f81dab4f9248ee184f3d52de5163151536df6566a59e9c75cbfe24be65eda8cd"
Jul 21 02:18:53.644348 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:53.644280    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f81dab4f9248ee184f3d52de5163151536df6566a59e9c75cbfe24be65eda8cd"} err="failed to get container status \"f81dab4f9248ee184f3d52de5163151536df6566a59e9c75cbfe24be65eda8cd\": rpc error: code = NotFound desc = could not find container \"f81dab4f9248ee184f3d52de5163151536df6566a59e9c75cbfe24be65eda8cd\": container with ID starting with f81dab4f9248ee184f3d52de5163151536df6566a59e9c75cbfe24be65eda8cd not found: ID does not exist"
Jul 21 02:18:53.651869 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:53.651830    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9" podStartSLOduration=1.651817066 podStartE2EDuration="1.651817066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:18:53.650366029 +0000 UTC m=+2553.248006379" watchObservedRunningTime="2026-07-21 02:18:53.651817066 +0000 UTC m=+2553.249457446"
Jul 21 02:18:53.666942 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:53.666922    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46"]
Jul 21 02:18:53.671140 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:53.671122    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-748a4d/auth-test-model-748a4d-kserve-69cf6f6d95-4mj46"]
Jul 21 02:18:54.147706 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:54.147664    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n" podUID="d91a3c26-d335-4889-ab81-374be282ef3e" containerName="main" containerID="cri-o://4be4a4d544c4ba3607a9f54afbeb7578ac80019b3717c68a0e8278180c34f705" gracePeriod=45
Jul 21 02:18:54.392756 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:54.392734    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n"
Jul 21 02:18:54.538437 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:54.538412    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d91a3c26-d335-4889-ab81-374be282ef3e-home\") pod \"d91a3c26-d335-4889-ab81-374be282ef3e\" (UID: \"d91a3c26-d335-4889-ab81-374be282ef3e\") "
Jul 21 02:18:54.538901 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:54.538488    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d91a3c26-d335-4889-ab81-374be282ef3e-model-cache\") pod \"d91a3c26-d335-4889-ab81-374be282ef3e\" (UID: \"d91a3c26-d335-4889-ab81-374be282ef3e\") "
Jul 21 02:18:54.538901 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:54.538518    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d91a3c26-d335-4889-ab81-374be282ef3e-dshm\") pod \"d91a3c26-d335-4889-ab81-374be282ef3e\" (UID: \"d91a3c26-d335-4889-ab81-374be282ef3e\") "
Jul 21 02:18:54.538901 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:54.538550    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d91a3c26-d335-4889-ab81-374be282ef3e-tmp-dir\") pod \"d91a3c26-d335-4889-ab81-374be282ef3e\" (UID: \"d91a3c26-d335-4889-ab81-374be282ef3e\") "
Jul 21 02:18:54.538901 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:54.538566    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74qj2\" (UniqueName: \"kubernetes.io/projected/d91a3c26-d335-4889-ab81-374be282ef3e-kube-api-access-74qj2\") pod \"d91a3c26-d335-4889-ab81-374be282ef3e\" (UID: \"d91a3c26-d335-4889-ab81-374be282ef3e\") "
Jul 21 02:18:54.538901 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:54.538585    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d91a3c26-d335-4889-ab81-374be282ef3e-tls-certs\") pod \"d91a3c26-d335-4889-ab81-374be282ef3e\" (UID: \"d91a3c26-d335-4889-ab81-374be282ef3e\") "
Jul 21 02:18:54.538901 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:54.538774    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91a3c26-d335-4889-ab81-374be282ef3e-home" (OuterVolumeSpecName: "home") pod "d91a3c26-d335-4889-ab81-374be282ef3e" (UID: "d91a3c26-d335-4889-ab81-374be282ef3e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:18:54.538901 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:54.538845    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91a3c26-d335-4889-ab81-374be282ef3e-model-cache" (OuterVolumeSpecName: "model-cache") pod "d91a3c26-d335-4889-ab81-374be282ef3e" (UID: "d91a3c26-d335-4889-ab81-374be282ef3e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:18:54.538901 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:54.538851    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91a3c26-d335-4889-ab81-374be282ef3e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d91a3c26-d335-4889-ab81-374be282ef3e" (UID: "d91a3c26-d335-4889-ab81-374be282ef3e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:18:54.538901 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:54.538881    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d91a3c26-d335-4889-ab81-374be282ef3e-home\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:18:54.540848 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:54.540824    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91a3c26-d335-4889-ab81-374be282ef3e-dshm" (OuterVolumeSpecName: "dshm") pod "d91a3c26-d335-4889-ab81-374be282ef3e" (UID: "d91a3c26-d335-4889-ab81-374be282ef3e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:18:54.540960 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:54.540901    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91a3c26-d335-4889-ab81-374be282ef3e-kube-api-access-74qj2" (OuterVolumeSpecName: "kube-api-access-74qj2") pod "d91a3c26-d335-4889-ab81-374be282ef3e" (UID: "d91a3c26-d335-4889-ab81-374be282ef3e"). InnerVolumeSpecName "kube-api-access-74qj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 02:18:54.541029 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:54.540988    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d91a3c26-d335-4889-ab81-374be282ef3e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d91a3c26-d335-4889-ab81-374be282ef3e" (UID: "d91a3c26-d335-4889-ab81-374be282ef3e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 02:18:54.639313 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:54.639286    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d91a3c26-d335-4889-ab81-374be282ef3e-model-cache\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:18:54.639313 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:54.639308    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d91a3c26-d335-4889-ab81-374be282ef3e-dshm\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:18:54.639313 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:54.639318    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d91a3c26-d335-4889-ab81-374be282ef3e-tmp-dir\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:18:54.639575 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:54.639327    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74qj2\" (UniqueName: \"kubernetes.io/projected/d91a3c26-d335-4889-ab81-374be282ef3e-kube-api-access-74qj2\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:18:54.639575 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:54.639335    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d91a3c26-d335-4889-ab81-374be282ef3e-tls-certs\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:18:54.641541 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:54.641511    2575 generic.go:356] "Generic (PLEG): container finished" podID="d91a3c26-d335-4889-ab81-374be282ef3e" containerID="4be4a4d544c4ba3607a9f54afbeb7578ac80019b3717c68a0e8278180c34f705" exitCode=0
Jul 21 02:18:54.641675 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:54.641558    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n"
Jul 21 02:18:54.641675 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:54.641602    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n" event={"ID":"d91a3c26-d335-4889-ab81-374be282ef3e","Type":"ContainerDied","Data":"4be4a4d544c4ba3607a9f54afbeb7578ac80019b3717c68a0e8278180c34f705"}
Jul 21 02:18:54.641675 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:54.641661    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n" event={"ID":"d91a3c26-d335-4889-ab81-374be282ef3e","Type":"ContainerDied","Data":"4558899a76d3a9e828154fe82f8aa6625eb41ff5ffb9144973629560edf3e6a6"}
Jul 21 02:18:54.641798 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:54.641685    2575 scope.go:117] "RemoveContainer" containerID="4be4a4d544c4ba3607a9f54afbeb7578ac80019b3717c68a0e8278180c34f705"
Jul 21 02:18:54.650814 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:54.650798    2575 scope.go:117] "RemoveContainer" containerID="4be4a4d544c4ba3607a9f54afbeb7578ac80019b3717c68a0e8278180c34f705"
Jul 21 02:18:54.651061 ip-10-0-129-219 kubenswrapper[2575]: E0721 02:18:54.651044    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4be4a4d544c4ba3607a9f54afbeb7578ac80019b3717c68a0e8278180c34f705\": container with ID starting with 4be4a4d544c4ba3607a9f54afbeb7578ac80019b3717c68a0e8278180c34f705 not found: ID does not exist" containerID="4be4a4d544c4ba3607a9f54afbeb7578ac80019b3717c68a0e8278180c34f705"
Jul 21 02:18:54.651128 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:54.651073    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4be4a4d544c4ba3607a9f54afbeb7578ac80019b3717c68a0e8278180c34f705"} err="failed to get container status \"4be4a4d544c4ba3607a9f54afbeb7578ac80019b3717c68a0e8278180c34f705\": rpc error: code = NotFound desc = could not find container \"4be4a4d544c4ba3607a9f54afbeb7578ac80019b3717c68a0e8278180c34f705\": container with ID starting with 4be4a4d544c4ba3607a9f54afbeb7578ac80019b3717c68a0e8278180c34f705 not found: ID does not exist"
Jul 21 02:18:54.663904 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:54.663882    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n"]
Jul 21 02:18:54.667413 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:54.667394    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c6538a/auth-test-model-c6538a-kserve-5b59f74cf-fqz2n"]
Jul 21 02:18:55.025643 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:55.025594    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d53d86d-4b5e-415b-bb7d-805078a19d17" path="/var/lib/kubelet/pods/3d53d86d-4b5e-415b-bb7d-805078a19d17/volumes"
Jul 21 02:18:55.026012 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:18:55.025999    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d91a3c26-d335-4889-ab81-374be282ef3e" path="/var/lib/kubelet/pods/d91a3c26-d335-4889-ab81-374be282ef3e/volumes"
Jul 21 02:19:02.717509 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:02.717475    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9"
Jul 21 02:19:02.717509 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:02.717518    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9"
Jul 21 02:19:02.730591 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:02.730509    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9"
Jul 21 02:19:03.699446 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:03.699417    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9"
Jul 21 02:19:10.980599 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:10.980555    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4"]
Jul 21 02:19:10.981153 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:10.981132    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d91a3c26-d335-4889-ab81-374be282ef3e" containerName="main"
Jul 21 02:19:10.981198 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:10.981158    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d91a3c26-d335-4889-ab81-374be282ef3e" containerName="main"
Jul 21 02:19:10.981198 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:10.981174    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d53d86d-4b5e-415b-bb7d-805078a19d17" containerName="main"
Jul 21 02:19:10.981198 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:10.981182    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d53d86d-4b5e-415b-bb7d-805078a19d17" containerName="main"
Jul 21 02:19:10.981293 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:10.981284    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d91a3c26-d335-4889-ab81-374be282ef3e" containerName="main"
Jul 21 02:19:10.981327 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:10.981298    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d53d86d-4b5e-415b-bb7d-805078a19d17" containerName="main"
Jul 21 02:19:10.984598 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:10.984578    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4"
Jul 21 02:19:10.988209 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:10.988187    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c6538a\"/\"default-dockercfg-zpbgz\""
Jul 21 02:19:10.988316 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:10.988211    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c6538a\"/\"openshift-service-ca.crt\""
Jul 21 02:19:10.988316 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:10.988253    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c6538a\"/\"kube-root-ca.crt\""
Jul 21 02:19:10.988316 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:10.988189    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c6538a\"/\"sub-test-model-c6538a-kserve-self-signed-certs\""
Jul 21 02:19:10.991689 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:10.991477    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21d35400-3f72-4177-ac59-d0ff4e53c9c7-tmp-dir\") pod \"sub-test-model-c6538a-kserve-89456cc6c-hcgk4\" (UID: \"21d35400-3f72-4177-ac59-d0ff4e53c9c7\") " pod="ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4"
Jul 21 02:19:10.991689 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:10.991526    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21d35400-3f72-4177-ac59-d0ff4e53c9c7-model-cache\") pod \"sub-test-model-c6538a-kserve-89456cc6c-hcgk4\" (UID: \"21d35400-3f72-4177-ac59-d0ff4e53c9c7\") " pod="ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4"
Jul 21 02:19:10.991689 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:10.991553    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21d35400-3f72-4177-ac59-d0ff4e53c9c7-tls-certs\") pod \"sub-test-model-c6538a-kserve-89456cc6c-hcgk4\" (UID: \"21d35400-3f72-4177-ac59-d0ff4e53c9c7\") " pod="ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4"
Jul 21 02:19:10.991689 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:10.991663    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21d35400-3f72-4177-ac59-d0ff4e53c9c7-dshm\") pod \"sub-test-model-c6538a-kserve-89456cc6c-hcgk4\" (UID: \"21d35400-3f72-4177-ac59-d0ff4e53c9c7\") " pod="ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4"
Jul 21 02:19:10.991925 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:10.991746    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21d35400-3f72-4177-ac59-d0ff4e53c9c7-home\") pod \"sub-test-model-c6538a-kserve-89456cc6c-hcgk4\" (UID: \"21d35400-3f72-4177-ac59-d0ff4e53c9c7\") " pod="ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4"
Jul 21 02:19:10.991925 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:10.991782    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw868\" (UniqueName: \"kubernetes.io/projected/21d35400-3f72-4177-ac59-d0ff4e53c9c7-kube-api-access-zw868\") pod \"sub-test-model-c6538a-kserve-89456cc6c-hcgk4\" (UID: \"21d35400-3f72-4177-ac59-d0ff4e53c9c7\") " pod="ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4"
Jul 21 02:19:10.993112 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:10.993090    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4"]
Jul 21 02:19:11.014184 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:11.014156    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:19:11.092183 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:11.092139    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21d35400-3f72-4177-ac59-d0ff4e53c9c7-home\") pod \"sub-test-model-c6538a-kserve-89456cc6c-hcgk4\" (UID: \"21d35400-3f72-4177-ac59-d0ff4e53c9c7\") " pod="ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4"
Jul 21 02:19:11.092363 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:11.092201    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw868\" (UniqueName: \"kubernetes.io/projected/21d35400-3f72-4177-ac59-d0ff4e53c9c7-kube-api-access-zw868\") pod \"sub-test-model-c6538a-kserve-89456cc6c-hcgk4\" (UID: \"21d35400-3f72-4177-ac59-d0ff4e53c9c7\") " pod="ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4"
Jul 21 02:19:11.092363 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:11.092236    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21d35400-3f72-4177-ac59-d0ff4e53c9c7-tmp-dir\") pod \"sub-test-model-c6538a-kserve-89456cc6c-hcgk4\" (UID: \"21d35400-3f72-4177-ac59-d0ff4e53c9c7\") " pod="ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4"
Jul 21 02:19:11.092363 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:11.092259    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21d35400-3f72-4177-ac59-d0ff4e53c9c7-model-cache\") pod \"sub-test-model-c6538a-kserve-89456cc6c-hcgk4\" (UID: \"21d35400-3f72-4177-ac59-d0ff4e53c9c7\") " pod="ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4"
Jul 21 02:19:11.092363 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:11.092275    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21d35400-3f72-4177-ac59-d0ff4e53c9c7-tls-certs\") pod \"sub-test-model-c6538a-kserve-89456cc6c-hcgk4\" (UID: \"21d35400-3f72-4177-ac59-d0ff4e53c9c7\") " pod="ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4"
Jul 21 02:19:11.092363 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:11.092319    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21d35400-3f72-4177-ac59-d0ff4e53c9c7-dshm\") pod \"sub-test-model-c6538a-kserve-89456cc6c-hcgk4\" (UID: \"21d35400-3f72-4177-ac59-d0ff4e53c9c7\") " pod="ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4"
Jul 21 02:19:11.092685 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:11.092636    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21d35400-3f72-4177-ac59-d0ff4e53c9c7-tmp-dir\") pod \"sub-test-model-c6538a-kserve-89456cc6c-hcgk4\" (UID: \"21d35400-3f72-4177-ac59-d0ff4e53c9c7\") " pod="ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4"
Jul 21 02:19:11.092685 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:11.092666    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21d35400-3f72-4177-ac59-d0ff4e53c9c7-home\") pod \"sub-test-model-c6538a-kserve-89456cc6c-hcgk4\" (UID: \"21d35400-3f72-4177-ac59-d0ff4e53c9c7\") " pod="ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4"
Jul 21 02:19:11.092788 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:11.092704    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21d35400-3f72-4177-ac59-d0ff4e53c9c7-model-cache\") pod \"sub-test-model-c6538a-kserve-89456cc6c-hcgk4\" (UID: \"21d35400-3f72-4177-ac59-d0ff4e53c9c7\") " pod="ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4"
Jul 21 02:19:11.094717 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:11.094683    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21d35400-3f72-4177-ac59-d0ff4e53c9c7-dshm\") pod \"sub-test-model-c6538a-kserve-89456cc6c-hcgk4\" (UID: \"21d35400-3f72-4177-ac59-d0ff4e53c9c7\") " pod="ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4"
Jul 21 02:19:11.094839 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:11.094824    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21d35400-3f72-4177-ac59-d0ff4e53c9c7-tls-certs\") pod \"sub-test-model-c6538a-kserve-89456cc6c-hcgk4\" (UID: \"21d35400-3f72-4177-ac59-d0ff4e53c9c7\") " pod="ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4"
Jul 21 02:19:11.100693 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:11.100670    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw868\" (UniqueName: \"kubernetes.io/projected/21d35400-3f72-4177-ac59-d0ff4e53c9c7-kube-api-access-zw868\") pod \"sub-test-model-c6538a-kserve-89456cc6c-hcgk4\" (UID: \"21d35400-3f72-4177-ac59-d0ff4e53c9c7\") " pod="ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4"
Jul 21 02:19:11.293906 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:11.293869    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4"
Jul 21 02:19:11.450514 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:11.450451    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4"]
Jul 21 02:19:11.454534 ip-10-0-129-219 kubenswrapper[2575]: W0721 02:19:11.454495    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21d35400_3f72_4177_ac59_d0ff4e53c9c7.slice/crio-75e05ad8f56d65cb642f85a8c6a0fad9902b888d3e1c6af5c9530238404af6d3 WatchSource:0}: Error finding container 75e05ad8f56d65cb642f85a8c6a0fad9902b888d3e1c6af5c9530238404af6d3: Status 404 returned error can't find the container with id 75e05ad8f56d65cb642f85a8c6a0fad9902b888d3e1c6af5c9530238404af6d3
Jul 21 02:19:11.727390 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:11.727300    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4" event={"ID":"21d35400-3f72-4177-ac59-d0ff4e53c9c7","Type":"ContainerStarted","Data":"fbd5d7dc59f38474263aacd321a266d8fb379f51d82743ce0ca3949d86e9dbcd"}
Jul 21 02:19:11.727390 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:11.727347    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4" event={"ID":"21d35400-3f72-4177-ac59-d0ff4e53c9c7","Type":"ContainerStarted","Data":"75e05ad8f56d65cb642f85a8c6a0fad9902b888d3e1c6af5c9530238404af6d3"}
Jul 21 02:19:11.746851 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:11.746798    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4" podStartSLOduration=1.746782553 podStartE2EDuration="1.746782553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:19:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:19:11.745960053 +0000 UTC m=+2571.343600404" watchObservedRunningTime="2026-07-21 02:19:11.746782553 +0000 UTC m=+2571.344422903"
Jul 21 02:19:21.294711 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:21.294684    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4"
Jul 21 02:19:21.295071 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:21.294879    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4"
Jul 21 02:19:21.307370 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:21.307349    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4"
Jul 21 02:19:21.787310 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:19:21.787281    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4"
Jul 21 02:20:02.745641 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:02.745583    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:20:02.807114 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:02.807072    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:20:04.657474 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:04.657433    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:20:11.190052 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:11.190020    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9"]
Jul 21 02:20:11.210871 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:11.210832    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:20:12.100143 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:12.100114    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4"]
Jul 21 02:20:12.121780 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:12.121748    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:20:17.221269 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:17.221224    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:20:17.283093 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:17.283052    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:20:21.251355 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:21.251320    2575 scope.go:117] "RemoveContainer" containerID="11214f230f6908518ae0adc4b90a036261985ac6951a4cfaa0e863ee287bf99e"
Jul 21 02:20:21.259152 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:21.259129    2575 scope.go:117] "RemoveContainer" containerID="babf8b78a2fac6d056b6ad156a5c497b35363b297b1a79cdf30fc0ab2914ec57"
Jul 21 02:20:22.628810 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:22.628777    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:20:22.728914 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:22.728874    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:20:24.936862 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:24.936827    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn"]
Jul 21 02:20:24.941304 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:24.941287    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn"
Jul 21 02:20:24.944355 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:24.944333    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-748a4d\"/\"rate-test-model-748a4d-kserve-self-signed-certs\""
Jul 21 02:20:24.967974 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:24.967951    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn"]
Jul 21 02:20:25.047364 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:25.047335    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-tls-certs\") pod \"rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn\" (UID: \"0fea9d62-d275-4ccb-80cd-0c4d31d8efef\") " pod="ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn"
Jul 21 02:20:25.047533 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:25.047374    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6kl9\" (UniqueName: \"kubernetes.io/projected/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-kube-api-access-p6kl9\") pod \"rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn\" (UID: \"0fea9d62-d275-4ccb-80cd-0c4d31d8efef\") " pod="ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn"
Jul 21 02:20:25.047533 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:25.047399    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-dshm\") pod \"rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn\" (UID: \"0fea9d62-d275-4ccb-80cd-0c4d31d8efef\") " pod="ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn"
Jul 21 02:20:25.047533 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:25.047417    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-model-cache\") pod \"rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn\" (UID: \"0fea9d62-d275-4ccb-80cd-0c4d31d8efef\") " pod="ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn"
Jul 21 02:20:25.047533 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:25.047523    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-tmp-dir\") pod \"rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn\" (UID: \"0fea9d62-d275-4ccb-80cd-0c4d31d8efef\") " pod="ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn"
Jul 21 02:20:25.047734 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:25.047559    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-home\") pod \"rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn\" (UID: \"0fea9d62-d275-4ccb-80cd-0c4d31d8efef\") " pod="ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn"
Jul 21 02:20:25.054718 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:25.054692    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:20:25.149092 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:25.149057    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-tls-certs\") pod \"rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn\" (UID: \"0fea9d62-d275-4ccb-80cd-0c4d31d8efef\") " pod="ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn"
Jul 21 02:20:25.149284 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:25.149115    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6kl9\" (UniqueName: \"kubernetes.io/projected/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-kube-api-access-p6kl9\") pod \"rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn\" (UID: \"0fea9d62-d275-4ccb-80cd-0c4d31d8efef\") " pod="ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn"
Jul 21 02:20:25.149284 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:25.149208    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-dshm\") pod \"rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn\" (UID: \"0fea9d62-d275-4ccb-80cd-0c4d31d8efef\") " pod="ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn"
Jul 21 02:20:25.149440 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:25.149398    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-model-cache\") pod \"rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn\" (UID: \"0fea9d62-d275-4ccb-80cd-0c4d31d8efef\") " pod="ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn"
Jul 21 02:20:25.149689 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:25.149664    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-tmp-dir\") pod \"rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn\" (UID: \"0fea9d62-d275-4ccb-80cd-0c4d31d8efef\") " pod="ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn"
Jul 21 02:20:25.149797 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:25.149714    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-home\") pod \"rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn\" (UID: \"0fea9d62-d275-4ccb-80cd-0c4d31d8efef\") " pod="ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn"
Jul 21 02:20:25.149797 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:25.149774    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-model-cache\") pod \"rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn\" (UID: \"0fea9d62-d275-4ccb-80cd-0c4d31d8efef\") " pod="ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn"
Jul 21 02:20:25.149967 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:25.149942    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-tmp-dir\") pod \"rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn\" (UID: \"0fea9d62-d275-4ccb-80cd-0c4d31d8efef\") " pod="ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn"
Jul 21 02:20:25.150143 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:25.150117    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-home\") pod \"rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn\" (UID: \"0fea9d62-d275-4ccb-80cd-0c4d31d8efef\") " pod="ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn"
Jul 21 02:20:25.151578 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:25.151554    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-dshm\") pod \"rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn\" (UID: \"0fea9d62-d275-4ccb-80cd-0c4d31d8efef\") " pod="ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn"
Jul 21 02:20:25.151775 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:25.151758    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-tls-certs\") pod \"rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn\" (UID: \"0fea9d62-d275-4ccb-80cd-0c4d31d8efef\") " pod="ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn"
Jul 21 02:20:25.157537 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:25.157520    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6kl9\" (UniqueName: \"kubernetes.io/projected/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-kube-api-access-p6kl9\") pod \"rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn\" (UID: \"0fea9d62-d275-4ccb-80cd-0c4d31d8efef\") " pod="ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn"
Jul 21 02:20:25.250914 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:25.250824    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn"
Jul 21 02:20:25.380295 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:25.380271    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn"]
Jul 21 02:20:25.383039 ip-10-0-129-219 kubenswrapper[2575]: W0721 02:20:25.383009    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0fea9d62_d275_4ccb_80cd_0c4d31d8efef.slice/crio-a2c35c1960751199dc13102ce85c8cdc0ad24b547bc0f811e8b8ad45cbd75166 WatchSource:0}: Error finding container a2c35c1960751199dc13102ce85c8cdc0ad24b547bc0f811e8b8ad45cbd75166: Status 404 returned error can't find the container with id a2c35c1960751199dc13102ce85c8cdc0ad24b547bc0f811e8b8ad45cbd75166
Jul 21 02:20:26.077309 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:26.077264    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn" event={"ID":"0fea9d62-d275-4ccb-80cd-0c4d31d8efef","Type":"ContainerStarted","Data":"91807af3ff1623a71f7385c31c348da023742818bbaa163984fd7449aed5f96c"}
Jul 21 02:20:26.077309 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:26.077313    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn" event={"ID":"0fea9d62-d275-4ccb-80cd-0c4d31d8efef","Type":"ContainerStarted","Data":"a2c35c1960751199dc13102ce85c8cdc0ad24b547bc0f811e8b8ad45cbd75166"}
Jul 21 02:20:26.094652 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:26.094587    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn" podStartSLOduration=2.094573427 podStartE2EDuration="2.094573427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:20:26.093659706 +0000 UTC m=+2645.691300051" watchObservedRunningTime="2026-07-21 02:20:26.094573427 +0000 UTC m=+2645.692213776"
Jul 21 02:20:26.209817 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:26.209780    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9" podUID="d4cd3379-3258-44ad-a737-cbe411468dd6" containerName="main" containerID="cri-o://af6eba8a0111146fa592a361092bba2cfe8a60afba0361ff22f614ee4d1e4ca9" gracePeriod=45
Jul 21 02:20:26.468487 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:26.468465    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9"
Jul 21 02:20:26.564196 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:26.564165    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4cd3379-3258-44ad-a737-cbe411468dd6-tls-certs\") pod \"d4cd3379-3258-44ad-a737-cbe411468dd6\" (UID: \"d4cd3379-3258-44ad-a737-cbe411468dd6\") "
Jul 21 02:20:26.564381 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:26.564223    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pskl6\" (UniqueName: \"kubernetes.io/projected/d4cd3379-3258-44ad-a737-cbe411468dd6-kube-api-access-pskl6\") pod \"d4cd3379-3258-44ad-a737-cbe411468dd6\" (UID: \"d4cd3379-3258-44ad-a737-cbe411468dd6\") "
Jul 21 02:20:26.564381 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:26.564248    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4cd3379-3258-44ad-a737-cbe411468dd6-home\") pod \"d4cd3379-3258-44ad-a737-cbe411468dd6\" (UID: \"d4cd3379-3258-44ad-a737-cbe411468dd6\") "
Jul 21 02:20:26.564504 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:26.564408    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4cd3379-3258-44ad-a737-cbe411468dd6-dshm\") pod \"d4cd3379-3258-44ad-a737-cbe411468dd6\" (UID: \"d4cd3379-3258-44ad-a737-cbe411468dd6\") "
Jul 21 02:20:26.564504 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:26.564458    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4cd3379-3258-44ad-a737-cbe411468dd6-tmp-dir\") pod \"d4cd3379-3258-44ad-a737-cbe411468dd6\" (UID: \"d4cd3379-3258-44ad-a737-cbe411468dd6\") "
Jul 21 02:20:26.564673 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:26.564529    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4cd3379-3258-44ad-a737-cbe411468dd6-model-cache\") pod \"d4cd3379-3258-44ad-a737-cbe411468dd6\" (UID: \"d4cd3379-3258-44ad-a737-cbe411468dd6\") "
Jul 21 02:20:26.564673 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:26.564458    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4cd3379-3258-44ad-a737-cbe411468dd6-home" (OuterVolumeSpecName: "home") pod "d4cd3379-3258-44ad-a737-cbe411468dd6" (UID: "d4cd3379-3258-44ad-a737-cbe411468dd6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:20:26.564673 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:26.564633    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4cd3379-3258-44ad-a737-cbe411468dd6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d4cd3379-3258-44ad-a737-cbe411468dd6" (UID: "d4cd3379-3258-44ad-a737-cbe411468dd6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:20:26.564837 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:26.564785    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4cd3379-3258-44ad-a737-cbe411468dd6-model-cache" (OuterVolumeSpecName: "model-cache") pod "d4cd3379-3258-44ad-a737-cbe411468dd6" (UID: "d4cd3379-3258-44ad-a737-cbe411468dd6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:20:26.564897 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:26.564887    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4cd3379-3258-44ad-a737-cbe411468dd6-home\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:20:26.564960 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:26.564898    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4cd3379-3258-44ad-a737-cbe411468dd6-tmp-dir\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:20:26.564960 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:26.564908    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4cd3379-3258-44ad-a737-cbe411468dd6-model-cache\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:20:26.566354 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:26.566329    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4cd3379-3258-44ad-a737-cbe411468dd6-dshm" (OuterVolumeSpecName: "dshm") pod "d4cd3379-3258-44ad-a737-cbe411468dd6" (UID: "d4cd3379-3258-44ad-a737-cbe411468dd6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:20:26.566476 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:26.566407    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4cd3379-3258-44ad-a737-cbe411468dd6-kube-api-access-pskl6" (OuterVolumeSpecName: "kube-api-access-pskl6") pod "d4cd3379-3258-44ad-a737-cbe411468dd6" (UID: "d4cd3379-3258-44ad-a737-cbe411468dd6"). InnerVolumeSpecName "kube-api-access-pskl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 02:20:26.566476 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:26.566451    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4cd3379-3258-44ad-a737-cbe411468dd6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d4cd3379-3258-44ad-a737-cbe411468dd6" (UID: "d4cd3379-3258-44ad-a737-cbe411468dd6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 02:20:26.666070 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:26.665998    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4cd3379-3258-44ad-a737-cbe411468dd6-tls-certs\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:20:26.666070 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:26.666021    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pskl6\" (UniqueName: \"kubernetes.io/projected/d4cd3379-3258-44ad-a737-cbe411468dd6-kube-api-access-pskl6\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:20:26.666070 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:26.666030    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4cd3379-3258-44ad-a737-cbe411468dd6-dshm\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:20:27.084765 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:27.084739    2575 generic.go:356] "Generic (PLEG): container finished" podID="d4cd3379-3258-44ad-a737-cbe411468dd6" containerID="af6eba8a0111146fa592a361092bba2cfe8a60afba0361ff22f614ee4d1e4ca9" exitCode=0
Jul 21 02:20:27.085244 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:27.084780    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9"
Jul 21 02:20:27.085244 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:27.084816    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9" event={"ID":"d4cd3379-3258-44ad-a737-cbe411468dd6","Type":"ContainerDied","Data":"af6eba8a0111146fa592a361092bba2cfe8a60afba0361ff22f614ee4d1e4ca9"}
Jul 21 02:20:27.085244 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:27.084855    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9" event={"ID":"d4cd3379-3258-44ad-a737-cbe411468dd6","Type":"ContainerDied","Data":"17e5c415888d59cfa6181840f7a82d1cff67d5d92843664838f6b28ef0edec7c"}
Jul 21 02:20:27.085244 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:27.084874    2575 scope.go:117] "RemoveContainer" containerID="af6eba8a0111146fa592a361092bba2cfe8a60afba0361ff22f614ee4d1e4ca9"
Jul 21 02:20:27.093570 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:27.093549    2575 scope.go:117] "RemoveContainer" containerID="af6eba8a0111146fa592a361092bba2cfe8a60afba0361ff22f614ee4d1e4ca9"
Jul 21 02:20:27.093827 ip-10-0-129-219 kubenswrapper[2575]: E0721 02:20:27.093810    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af6eba8a0111146fa592a361092bba2cfe8a60afba0361ff22f614ee4d1e4ca9\": container with ID starting with af6eba8a0111146fa592a361092bba2cfe8a60afba0361ff22f614ee4d1e4ca9 not found: ID does not exist" containerID="af6eba8a0111146fa592a361092bba2cfe8a60afba0361ff22f614ee4d1e4ca9"
Jul 21 02:20:27.093884 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:27.093834    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af6eba8a0111146fa592a361092bba2cfe8a60afba0361ff22f614ee4d1e4ca9"} err="failed to get container status \"af6eba8a0111146fa592a361092bba2cfe8a60afba0361ff22f614ee4d1e4ca9\": rpc error: code = NotFound desc = could not find container \"af6eba8a0111146fa592a361092bba2cfe8a60afba0361ff22f614ee4d1e4ca9\": container with ID starting with af6eba8a0111146fa592a361092bba2cfe8a60afba0361ff22f614ee4d1e4ca9 not found: ID does not exist"
Jul 21 02:20:27.101246 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:27.101225    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9"]
Jul 21 02:20:27.104733 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:27.104715    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-748a4d/sub-test-model-748a4d-kserve-786c89cd8c-858q9"]
Jul 21 02:20:27.119578 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:27.119553    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4" podUID="21d35400-3f72-4177-ac59-d0ff4e53c9c7" containerName="main" containerID="cri-o://fbd5d7dc59f38474263aacd321a266d8fb379f51d82743ce0ca3949d86e9dbcd" gracePeriod=45
Jul 21 02:20:27.373876 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:27.373851    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4"
Jul 21 02:20:27.473473 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:27.473441    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21d35400-3f72-4177-ac59-d0ff4e53c9c7-model-cache\") pod \"21d35400-3f72-4177-ac59-d0ff4e53c9c7\" (UID: \"21d35400-3f72-4177-ac59-d0ff4e53c9c7\") "
Jul 21 02:20:27.473473 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:27.473475    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21d35400-3f72-4177-ac59-d0ff4e53c9c7-home\") pod \"21d35400-3f72-4177-ac59-d0ff4e53c9c7\" (UID: \"21d35400-3f72-4177-ac59-d0ff4e53c9c7\") "
Jul 21 02:20:27.473699 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:27.473541    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21d35400-3f72-4177-ac59-d0ff4e53c9c7-tls-certs\") pod \"21d35400-3f72-4177-ac59-d0ff4e53c9c7\" (UID: \"21d35400-3f72-4177-ac59-d0ff4e53c9c7\") "
Jul 21 02:20:27.473699 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:27.473643    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21d35400-3f72-4177-ac59-d0ff4e53c9c7-tmp-dir\") pod \"21d35400-3f72-4177-ac59-d0ff4e53c9c7\" (UID: \"21d35400-3f72-4177-ac59-d0ff4e53c9c7\") "
Jul 21 02:20:27.473774 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:27.473702    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zw868\" (UniqueName: \"kubernetes.io/projected/21d35400-3f72-4177-ac59-d0ff4e53c9c7-kube-api-access-zw868\") pod \"21d35400-3f72-4177-ac59-d0ff4e53c9c7\" (UID: \"21d35400-3f72-4177-ac59-d0ff4e53c9c7\") "
Jul 21 02:20:27.473774 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:27.473746    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d35400-3f72-4177-ac59-d0ff4e53c9c7-model-cache" (OuterVolumeSpecName: "model-cache") pod "21d35400-3f72-4177-ac59-d0ff4e53c9c7" (UID: "21d35400-3f72-4177-ac59-d0ff4e53c9c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:20:27.473855 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:27.473781    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21d35400-3f72-4177-ac59-d0ff4e53c9c7-dshm\") pod \"21d35400-3f72-4177-ac59-d0ff4e53c9c7\" (UID: \"21d35400-3f72-4177-ac59-d0ff4e53c9c7\") "
Jul 21 02:20:27.473855 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:27.473822    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d35400-3f72-4177-ac59-d0ff4e53c9c7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "21d35400-3f72-4177-ac59-d0ff4e53c9c7" (UID: "21d35400-3f72-4177-ac59-d0ff4e53c9c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:20:27.473855 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:27.473832    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d35400-3f72-4177-ac59-d0ff4e53c9c7-home" (OuterVolumeSpecName: "home") pod "21d35400-3f72-4177-ac59-d0ff4e53c9c7" (UID: "21d35400-3f72-4177-ac59-d0ff4e53c9c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:20:27.474022 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:27.474006    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21d35400-3f72-4177-ac59-d0ff4e53c9c7-model-cache\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:20:27.474022 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:27.474017    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21d35400-3f72-4177-ac59-d0ff4e53c9c7-home\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:20:27.474126 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:27.474026    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21d35400-3f72-4177-ac59-d0ff4e53c9c7-tmp-dir\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:20:27.475706 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:27.475683    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d35400-3f72-4177-ac59-d0ff4e53c9c7-dshm" (OuterVolumeSpecName: "dshm") pod "21d35400-3f72-4177-ac59-d0ff4e53c9c7" (UID: "21d35400-3f72-4177-ac59-d0ff4e53c9c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:20:27.475706 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:27.475697    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21d35400-3f72-4177-ac59-d0ff4e53c9c7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "21d35400-3f72-4177-ac59-d0ff4e53c9c7" (UID: "21d35400-3f72-4177-ac59-d0ff4e53c9c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 02:20:27.475871 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:27.475736    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21d35400-3f72-4177-ac59-d0ff4e53c9c7-kube-api-access-zw868" (OuterVolumeSpecName: "kube-api-access-zw868") pod "21d35400-3f72-4177-ac59-d0ff4e53c9c7" (UID: "21d35400-3f72-4177-ac59-d0ff4e53c9c7"). InnerVolumeSpecName "kube-api-access-zw868". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 02:20:27.575407 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:27.575342    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw868\" (UniqueName: \"kubernetes.io/projected/21d35400-3f72-4177-ac59-d0ff4e53c9c7-kube-api-access-zw868\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:20:27.575407 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:27.575367    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21d35400-3f72-4177-ac59-d0ff4e53c9c7-dshm\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:20:27.575407 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:27.575375    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21d35400-3f72-4177-ac59-d0ff4e53c9c7-tls-certs\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:20:28.097578 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:28.097540    2575 generic.go:356] "Generic (PLEG): container finished" podID="21d35400-3f72-4177-ac59-d0ff4e53c9c7" containerID="fbd5d7dc59f38474263aacd321a266d8fb379f51d82743ce0ca3949d86e9dbcd" exitCode=0
Jul 21 02:20:28.097578 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:28.097581    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4" event={"ID":"21d35400-3f72-4177-ac59-d0ff4e53c9c7","Type":"ContainerDied","Data":"fbd5d7dc59f38474263aacd321a266d8fb379f51d82743ce0ca3949d86e9dbcd"}
Jul 21 02:20:28.098128 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:28.097604    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4"
Jul 21 02:20:28.098128 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:28.097641    2575 scope.go:117] "RemoveContainer" containerID="fbd5d7dc59f38474263aacd321a266d8fb379f51d82743ce0ca3949d86e9dbcd"
Jul 21 02:20:28.098128 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:28.097628    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4" event={"ID":"21d35400-3f72-4177-ac59-d0ff4e53c9c7","Type":"ContainerDied","Data":"75e05ad8f56d65cb642f85a8c6a0fad9902b888d3e1c6af5c9530238404af6d3"}
Jul 21 02:20:28.106577 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:28.106561    2575 scope.go:117] "RemoveContainer" containerID="fbd5d7dc59f38474263aacd321a266d8fb379f51d82743ce0ca3949d86e9dbcd"
Jul 21 02:20:28.106832 ip-10-0-129-219 kubenswrapper[2575]: E0721 02:20:28.106808    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbd5d7dc59f38474263aacd321a266d8fb379f51d82743ce0ca3949d86e9dbcd\": container with ID starting with fbd5d7dc59f38474263aacd321a266d8fb379f51d82743ce0ca3949d86e9dbcd not found: ID does not exist" containerID="fbd5d7dc59f38474263aacd321a266d8fb379f51d82743ce0ca3949d86e9dbcd"
Jul 21 02:20:28.106936 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:28.106836    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbd5d7dc59f38474263aacd321a266d8fb379f51d82743ce0ca3949d86e9dbcd"} err="failed to get container status \"fbd5d7dc59f38474263aacd321a266d8fb379f51d82743ce0ca3949d86e9dbcd\": rpc error: code = NotFound desc = could not find container \"fbd5d7dc59f38474263aacd321a266d8fb379f51d82743ce0ca3949d86e9dbcd\": container with ID starting with fbd5d7dc59f38474263aacd321a266d8fb379f51d82743ce0ca3949d86e9dbcd not found: ID does not exist"
Jul 21 02:20:28.120194 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:28.120170    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4"]
Jul 21 02:20:28.124148 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:28.124125    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c6538a/sub-test-model-c6538a-kserve-89456cc6c-hcgk4"]
Jul 21 02:20:29.026233 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:29.026199    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21d35400-3f72-4177-ac59-d0ff4e53c9c7" path="/var/lib/kubelet/pods/21d35400-3f72-4177-ac59-d0ff4e53c9c7/volumes"
Jul 21 02:20:29.026587 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:29.026574    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4cd3379-3258-44ad-a737-cbe411468dd6" path="/var/lib/kubelet/pods/d4cd3379-3258-44ad-a737-cbe411468dd6/volumes"
Jul 21 02:20:31.483490 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:31.483456    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:20:35.251249 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:35.251211    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn"
Jul 21 02:20:35.251700 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:35.251515    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn"
Jul 21 02:20:35.264088 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:35.264067    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn"
Jul 21 02:20:36.146956 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:36.146928    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn"
Jul 21 02:20:43.499112 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:43.499074    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q"]
Jul 21 02:20:43.499626 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:43.499594    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21d35400-3f72-4177-ac59-d0ff4e53c9c7" containerName="main"
Jul 21 02:20:43.499729 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:43.499636    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="21d35400-3f72-4177-ac59-d0ff4e53c9c7" containerName="main"
Jul 21 02:20:43.499729 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:43.499660    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4cd3379-3258-44ad-a737-cbe411468dd6" containerName="main"
Jul 21 02:20:43.499729 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:43.499669    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4cd3379-3258-44ad-a737-cbe411468dd6" containerName="main"
Jul 21 02:20:43.499896 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:43.499751    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="21d35400-3f72-4177-ac59-d0ff4e53c9c7" containerName="main"
Jul 21 02:20:43.499896 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:43.499766    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4cd3379-3258-44ad-a737-cbe411468dd6" containerName="main"
Jul 21 02:20:43.504460 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:43.504441    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q"
Jul 21 02:20:43.507450 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:43.507430    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c6538a\"/\"kube-root-ca.crt\""
Jul 21 02:20:43.507737 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:43.507715    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c6538a\"/\"openshift-service-ca.crt\""
Jul 21 02:20:43.507830 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:43.507740    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c6538a\"/\"default-dockercfg-zpbgz\""
Jul 21 02:20:43.507830 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:43.507804    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c6538a\"/\"rate-test-model-c6538a-kserve-self-signed-certs\""
Jul 21 02:20:43.512128 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:43.512104    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55cda727-c2ae-4273-9e3f-abf1bbc5676c-tmp-dir\") pod \"rate-test-model-c6538a-kserve-6cc784dcff-bf78q\" (UID: \"55cda727-c2ae-4273-9e3f-abf1bbc5676c\") " pod="ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q"
Jul 21 02:20:43.512289 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:43.512263    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55cda727-c2ae-4273-9e3f-abf1bbc5676c-dshm\") pod \"rate-test-model-c6538a-kserve-6cc784dcff-bf78q\" (UID: \"55cda727-c2ae-4273-9e3f-abf1bbc5676c\") " pod="ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q"
Jul 21 02:20:43.512428 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:43.512300    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55cda727-c2ae-4273-9e3f-abf1bbc5676c-tls-certs\") pod \"rate-test-model-c6538a-kserve-6cc784dcff-bf78q\" (UID: \"55cda727-c2ae-4273-9e3f-abf1bbc5676c\") " pod="ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q"
Jul 21 02:20:43.512428 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:43.512324    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87gxj\" (UniqueName: \"kubernetes.io/projected/55cda727-c2ae-4273-9e3f-abf1bbc5676c-kube-api-access-87gxj\") pod \"rate-test-model-c6538a-kserve-6cc784dcff-bf78q\" (UID: \"55cda727-c2ae-4273-9e3f-abf1bbc5676c\") " pod="ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q"
Jul 21 02:20:43.512428 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:43.512353    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55cda727-c2ae-4273-9e3f-abf1bbc5676c-model-cache\") pod \"rate-test-model-c6538a-kserve-6cc784dcff-bf78q\" (UID: \"55cda727-c2ae-4273-9e3f-abf1bbc5676c\") " pod="ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q"
Jul 21 02:20:43.512600 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:43.512440    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55cda727-c2ae-4273-9e3f-abf1bbc5676c-home\") pod \"rate-test-model-c6538a-kserve-6cc784dcff-bf78q\" (UID: \"55cda727-c2ae-4273-9e3f-abf1bbc5676c\") " pod="ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q"
Jul 21 02:20:43.513055 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:43.513034    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q"]
Jul 21 02:20:43.530659 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:43.530606    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:20:43.612889 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:43.612863    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55cda727-c2ae-4273-9e3f-abf1bbc5676c-home\") pod \"rate-test-model-c6538a-kserve-6cc784dcff-bf78q\" (UID: \"55cda727-c2ae-4273-9e3f-abf1bbc5676c\") " pod="ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q"
Jul 21 02:20:43.613014 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:43.612935    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55cda727-c2ae-4273-9e3f-abf1bbc5676c-tmp-dir\") pod \"rate-test-model-c6538a-kserve-6cc784dcff-bf78q\" (UID: \"55cda727-c2ae-4273-9e3f-abf1bbc5676c\") " pod="ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q"
Jul 21 02:20:43.613058 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:43.613026    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55cda727-c2ae-4273-9e3f-abf1bbc5676c-dshm\") pod \"rate-test-model-c6538a-kserve-6cc784dcff-bf78q\" (UID: \"55cda727-c2ae-4273-9e3f-abf1bbc5676c\") " pod="ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q"
Jul 21 02:20:43.613058 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:43.613055    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55cda727-c2ae-4273-9e3f-abf1bbc5676c-tls-certs\") pod \"rate-test-model-c6538a-kserve-6cc784dcff-bf78q\" (UID: \"55cda727-c2ae-4273-9e3f-abf1bbc5676c\") " pod="ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q"
Jul 21 02:20:43.613142 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:43.613077    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87gxj\" (UniqueName: \"kubernetes.io/projected/55cda727-c2ae-4273-9e3f-abf1bbc5676c-kube-api-access-87gxj\") pod \"rate-test-model-c6538a-kserve-6cc784dcff-bf78q\" (UID: \"55cda727-c2ae-4273-9e3f-abf1bbc5676c\") " pod="ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q"
Jul 21 02:20:43.613142 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:43.613107    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55cda727-c2ae-4273-9e3f-abf1bbc5676c-model-cache\") pod \"rate-test-model-c6538a-kserve-6cc784dcff-bf78q\" (UID: \"55cda727-c2ae-4273-9e3f-abf1bbc5676c\") " pod="ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q"
Jul 21 02:20:43.613301 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:43.613279    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55cda727-c2ae-4273-9e3f-abf1bbc5676c-home\") pod \"rate-test-model-c6538a-kserve-6cc784dcff-bf78q\" (UID: \"55cda727-c2ae-4273-9e3f-abf1bbc5676c\") " pod="ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q"
Jul 21 02:20:43.613377 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:43.613310    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55cda727-c2ae-4273-9e3f-abf1bbc5676c-tmp-dir\") pod \"rate-test-model-c6538a-kserve-6cc784dcff-bf78q\" (UID: \"55cda727-c2ae-4273-9e3f-abf1bbc5676c\") " pod="ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q"
Jul 21 02:20:43.613463 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:43.613447    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55cda727-c2ae-4273-9e3f-abf1bbc5676c-model-cache\") pod \"rate-test-model-c6538a-kserve-6cc784dcff-bf78q\" (UID: \"55cda727-c2ae-4273-9e3f-abf1bbc5676c\") " pod="ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q"
Jul 21 02:20:43.615235 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:43.615214    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55cda727-c2ae-4273-9e3f-abf1bbc5676c-dshm\") pod \"rate-test-model-c6538a-kserve-6cc784dcff-bf78q\" (UID: \"55cda727-c2ae-4273-9e3f-abf1bbc5676c\") " pod="ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q"
Jul 21 02:20:43.615406 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:43.615389    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55cda727-c2ae-4273-9e3f-abf1bbc5676c-tls-certs\") pod \"rate-test-model-c6538a-kserve-6cc784dcff-bf78q\" (UID: \"55cda727-c2ae-4273-9e3f-abf1bbc5676c\") " pod="ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q"
Jul 21 02:20:43.621542 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:43.621520    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87gxj\" (UniqueName: \"kubernetes.io/projected/55cda727-c2ae-4273-9e3f-abf1bbc5676c-kube-api-access-87gxj\") pod \"rate-test-model-c6538a-kserve-6cc784dcff-bf78q\" (UID: \"55cda727-c2ae-4273-9e3f-abf1bbc5676c\") " pod="ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q"
Jul 21 02:20:43.814862 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:43.814825    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q"
Jul 21 02:20:43.953999 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:43.953971    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q"]
Jul 21 02:20:43.956131 ip-10-0-129-219 kubenswrapper[2575]: W0721 02:20:43.956101    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55cda727_c2ae_4273_9e3f_abf1bbc5676c.slice/crio-d5bc899b387ad81f793fbcf8d4b0c3bb7cb6a46e7b29bd47f2833d855d7065f6 WatchSource:0}: Error finding container d5bc899b387ad81f793fbcf8d4b0c3bb7cb6a46e7b29bd47f2833d855d7065f6: Status 404 returned error can't find the container with id d5bc899b387ad81f793fbcf8d4b0c3bb7cb6a46e7b29bd47f2833d855d7065f6
Jul 21 02:20:44.174132 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:44.174040    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q" event={"ID":"55cda727-c2ae-4273-9e3f-abf1bbc5676c","Type":"ContainerStarted","Data":"1efd7c8e9a84350e4a865916b35537a21e2339d468b1ef4df95560d4fcd4395c"}
Jul 21 02:20:44.174132 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:44.174083    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q" event={"ID":"55cda727-c2ae-4273-9e3f-abf1bbc5676c","Type":"ContainerStarted","Data":"d5bc899b387ad81f793fbcf8d4b0c3bb7cb6a46e7b29bd47f2833d855d7065f6"}
Jul 21 02:20:44.190322 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:44.190270    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q" podStartSLOduration=1.190255759 podStartE2EDuration="1.190255759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:20:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:20:44.189064753 +0000 UTC m=+2663.786705101" watchObservedRunningTime="2026-07-21 02:20:44.190255759 +0000 UTC m=+2663.787896155"
Jul 21 02:20:49.848788 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:49.848738    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:20:49.950895 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:49.950853    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:20:53.815746 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:53.815714    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q"
Jul 21 02:20:53.815746 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:53.815757    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q"
Jul 21 02:20:53.828302 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:53.828271    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q"
Jul 21 02:20:54.232679 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:54.232581    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q"
Jul 21 02:20:57.649284 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:57.649252    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:20:57.753849 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:57.753815    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:20:59.693532 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:20:59.693501    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:21:21.280249 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:21.280215    2575 scope.go:117] "RemoveContainer" containerID="ce84ca459bc4fd3e2472aa6c7a605f29fe6494b6030d9b2f0a9181e9bd748a35"
Jul 21 02:21:21.288251 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:21.288228    2575 scope.go:117] "RemoveContainer" containerID="799d2b1d74a05b4b846782437cf2df7b2922d75c5bccbb38753ca2633f6aec10"
Jul 21 02:21:21.295477 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:21.295457    2575 scope.go:117] "RemoveContainer" containerID="5ec5b5322bbc8ab1fbb5af370d5b643b608e9faf720aad5379da407611a3a930"
Jul 21 02:21:25.472433 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:25.472398    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn"]
Jul 21 02:21:25.498753 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:25.498719    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:21:26.302675 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:26.302643    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q"]
Jul 21 02:21:26.322193 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:26.322150    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:21:40.492318 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:40.492281    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn" podUID="0fea9d62-d275-4ccb-80cd-0c4d31d8efef" containerName="main" containerID="cri-o://91807af3ff1623a71f7385c31c348da023742818bbaa163984fd7449aed5f96c" gracePeriod=45
Jul 21 02:21:40.739678 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:40.739650    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn"
Jul 21 02:21:40.808936 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:40.808909    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-tmp-dir\") pod \"0fea9d62-d275-4ccb-80cd-0c4d31d8efef\" (UID: \"0fea9d62-d275-4ccb-80cd-0c4d31d8efef\") "
Jul 21 02:21:40.808936 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:40.808939    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-model-cache\") pod \"0fea9d62-d275-4ccb-80cd-0c4d31d8efef\" (UID: \"0fea9d62-d275-4ccb-80cd-0c4d31d8efef\") "
Jul 21 02:21:40.809126 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:40.808966    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-home\") pod \"0fea9d62-d275-4ccb-80cd-0c4d31d8efef\" (UID: \"0fea9d62-d275-4ccb-80cd-0c4d31d8efef\") "
Jul 21 02:21:40.809126 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:40.808992    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-dshm\") pod \"0fea9d62-d275-4ccb-80cd-0c4d31d8efef\" (UID: \"0fea9d62-d275-4ccb-80cd-0c4d31d8efef\") "
Jul 21 02:21:40.809126 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:40.809056    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-tls-certs\") pod \"0fea9d62-d275-4ccb-80cd-0c4d31d8efef\" (UID: \"0fea9d62-d275-4ccb-80cd-0c4d31d8efef\") "
Jul 21 02:21:40.809126 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:40.809106    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6kl9\" (UniqueName: \"kubernetes.io/projected/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-kube-api-access-p6kl9\") pod \"0fea9d62-d275-4ccb-80cd-0c4d31d8efef\" (UID: \"0fea9d62-d275-4ccb-80cd-0c4d31d8efef\") "
Jul 21 02:21:40.809276 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:40.809194    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0fea9d62-d275-4ccb-80cd-0c4d31d8efef" (UID: "0fea9d62-d275-4ccb-80cd-0c4d31d8efef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:21:40.809276 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:40.809207    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-model-cache" (OuterVolumeSpecName: "model-cache") pod "0fea9d62-d275-4ccb-80cd-0c4d31d8efef" (UID: "0fea9d62-d275-4ccb-80cd-0c4d31d8efef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:21:40.809276 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:40.809214    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-home" (OuterVolumeSpecName: "home") pod "0fea9d62-d275-4ccb-80cd-0c4d31d8efef" (UID: "0fea9d62-d275-4ccb-80cd-0c4d31d8efef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:21:40.809429 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:40.809410    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-tmp-dir\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:21:40.809494 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:40.809441    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-model-cache\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:21:40.809494 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:40.809454    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-home\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:21:40.811107 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:40.811082    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-kube-api-access-p6kl9" (OuterVolumeSpecName: "kube-api-access-p6kl9") pod "0fea9d62-d275-4ccb-80cd-0c4d31d8efef" (UID: "0fea9d62-d275-4ccb-80cd-0c4d31d8efef"). InnerVolumeSpecName "kube-api-access-p6kl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 02:21:40.811213 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:40.811107    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0fea9d62-d275-4ccb-80cd-0c4d31d8efef" (UID: "0fea9d62-d275-4ccb-80cd-0c4d31d8efef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 02:21:40.811213 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:40.811165    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-dshm" (OuterVolumeSpecName: "dshm") pod "0fea9d62-d275-4ccb-80cd-0c4d31d8efef" (UID: "0fea9d62-d275-4ccb-80cd-0c4d31d8efef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:21:40.910421 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:40.910389    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-tls-certs\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:21:40.910421 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:40.910417    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6kl9\" (UniqueName: \"kubernetes.io/projected/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-kube-api-access-p6kl9\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:21:40.910421 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:40.910427    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fea9d62-d275-4ccb-80cd-0c4d31d8efef-dshm\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:21:41.322030 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:41.321965    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q" podUID="55cda727-c2ae-4273-9e3f-abf1bbc5676c" containerName="main" containerID="cri-o://1efd7c8e9a84350e4a865916b35537a21e2339d468b1ef4df95560d4fcd4395c" gracePeriod=45
Jul 21 02:21:41.437680 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:41.437642    2575 generic.go:356] "Generic (PLEG): container finished" podID="0fea9d62-d275-4ccb-80cd-0c4d31d8efef" containerID="91807af3ff1623a71f7385c31c348da023742818bbaa163984fd7449aed5f96c" exitCode=0
Jul 21 02:21:41.437854 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:41.437685    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn"
Jul 21 02:21:41.437854 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:41.437733    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn" event={"ID":"0fea9d62-d275-4ccb-80cd-0c4d31d8efef","Type":"ContainerDied","Data":"91807af3ff1623a71f7385c31c348da023742818bbaa163984fd7449aed5f96c"}
Jul 21 02:21:41.437854 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:41.437768    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn" event={"ID":"0fea9d62-d275-4ccb-80cd-0c4d31d8efef","Type":"ContainerDied","Data":"a2c35c1960751199dc13102ce85c8cdc0ad24b547bc0f811e8b8ad45cbd75166"}
Jul 21 02:21:41.437854 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:41.437789    2575 scope.go:117] "RemoveContainer" containerID="91807af3ff1623a71f7385c31c348da023742818bbaa163984fd7449aed5f96c"
Jul 21 02:21:41.439455 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:41.439435    2575 generic.go:356] "Generic (PLEG): container finished" podID="55cda727-c2ae-4273-9e3f-abf1bbc5676c" containerID="1efd7c8e9a84350e4a865916b35537a21e2339d468b1ef4df95560d4fcd4395c" exitCode=0
Jul 21 02:21:41.439543 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:41.439503    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q" event={"ID":"55cda727-c2ae-4273-9e3f-abf1bbc5676c","Type":"ContainerDied","Data":"1efd7c8e9a84350e4a865916b35537a21e2339d468b1ef4df95560d4fcd4395c"}
Jul 21 02:21:41.447658 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:41.447635    2575 scope.go:117] "RemoveContainer" containerID="91807af3ff1623a71f7385c31c348da023742818bbaa163984fd7449aed5f96c"
Jul 21 02:21:41.447919 ip-10-0-129-219 kubenswrapper[2575]: E0721 02:21:41.447900    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91807af3ff1623a71f7385c31c348da023742818bbaa163984fd7449aed5f96c\": container with ID starting with 91807af3ff1623a71f7385c31c348da023742818bbaa163984fd7449aed5f96c not found: ID does not exist" containerID="91807af3ff1623a71f7385c31c348da023742818bbaa163984fd7449aed5f96c"
Jul 21 02:21:41.447987 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:41.447934    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91807af3ff1623a71f7385c31c348da023742818bbaa163984fd7449aed5f96c"} err="failed to get container status \"91807af3ff1623a71f7385c31c348da023742818bbaa163984fd7449aed5f96c\": rpc error: code = NotFound desc = could not find container \"91807af3ff1623a71f7385c31c348da023742818bbaa163984fd7449aed5f96c\": container with ID starting with 91807af3ff1623a71f7385c31c348da023742818bbaa163984fd7449aed5f96c not found: ID does not exist"
Jul 21 02:21:41.452946 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:41.452921    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn"]
Jul 21 02:21:41.456554 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:41.456531    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-748a4d/rate-test-model-748a4d-kserve-5d4869fc5f-cs8nn"]
Jul 21 02:21:41.578319 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:41.578259    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q"
Jul 21 02:21:41.616020 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:41.615992    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55cda727-c2ae-4273-9e3f-abf1bbc5676c-home\") pod \"55cda727-c2ae-4273-9e3f-abf1bbc5676c\" (UID: \"55cda727-c2ae-4273-9e3f-abf1bbc5676c\") "
Jul 21 02:21:41.616203 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:41.616030    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55cda727-c2ae-4273-9e3f-abf1bbc5676c-model-cache\") pod \"55cda727-c2ae-4273-9e3f-abf1bbc5676c\" (UID: \"55cda727-c2ae-4273-9e3f-abf1bbc5676c\") "
Jul 21 02:21:41.616203 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:41.616092    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87gxj\" (UniqueName: \"kubernetes.io/projected/55cda727-c2ae-4273-9e3f-abf1bbc5676c-kube-api-access-87gxj\") pod \"55cda727-c2ae-4273-9e3f-abf1bbc5676c\" (UID: \"55cda727-c2ae-4273-9e3f-abf1bbc5676c\") "
Jul 21 02:21:41.616203 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:41.616153    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55cda727-c2ae-4273-9e3f-abf1bbc5676c-tmp-dir\") pod \"55cda727-c2ae-4273-9e3f-abf1bbc5676c\" (UID: \"55cda727-c2ae-4273-9e3f-abf1bbc5676c\") "
Jul 21 02:21:41.616203 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:41.616193    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55cda727-c2ae-4273-9e3f-abf1bbc5676c-dshm\") pod \"55cda727-c2ae-4273-9e3f-abf1bbc5676c\" (UID: \"55cda727-c2ae-4273-9e3f-abf1bbc5676c\") "
Jul 21 02:21:41.616433 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:41.616225    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55cda727-c2ae-4273-9e3f-abf1bbc5676c-tls-certs\") pod \"55cda727-c2ae-4273-9e3f-abf1bbc5676c\" (UID: \"55cda727-c2ae-4273-9e3f-abf1bbc5676c\") "
Jul 21 02:21:41.616433 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:41.616274    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55cda727-c2ae-4273-9e3f-abf1bbc5676c-home" (OuterVolumeSpecName: "home") pod "55cda727-c2ae-4273-9e3f-abf1bbc5676c" (UID: "55cda727-c2ae-4273-9e3f-abf1bbc5676c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:21:41.616433 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:41.616368    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55cda727-c2ae-4273-9e3f-abf1bbc5676c-model-cache" (OuterVolumeSpecName: "model-cache") pod "55cda727-c2ae-4273-9e3f-abf1bbc5676c" (UID: "55cda727-c2ae-4273-9e3f-abf1bbc5676c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:21:41.616601 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:41.616514    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55cda727-c2ae-4273-9e3f-abf1bbc5676c-home\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:21:41.616601 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:41.616534    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55cda727-c2ae-4273-9e3f-abf1bbc5676c-model-cache\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:21:41.616601 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:41.616554    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55cda727-c2ae-4273-9e3f-abf1bbc5676c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "55cda727-c2ae-4273-9e3f-abf1bbc5676c" (UID: "55cda727-c2ae-4273-9e3f-abf1bbc5676c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:21:41.618653 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:41.618626    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55cda727-c2ae-4273-9e3f-abf1bbc5676c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "55cda727-c2ae-4273-9e3f-abf1bbc5676c" (UID: "55cda727-c2ae-4273-9e3f-abf1bbc5676c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 02:21:41.618779 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:41.618658    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55cda727-c2ae-4273-9e3f-abf1bbc5676c-kube-api-access-87gxj" (OuterVolumeSpecName: "kube-api-access-87gxj") pod "55cda727-c2ae-4273-9e3f-abf1bbc5676c" (UID: "55cda727-c2ae-4273-9e3f-abf1bbc5676c"). InnerVolumeSpecName "kube-api-access-87gxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 02:21:41.618879 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:41.618849    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55cda727-c2ae-4273-9e3f-abf1bbc5676c-dshm" (OuterVolumeSpecName: "dshm") pod "55cda727-c2ae-4273-9e3f-abf1bbc5676c" (UID: "55cda727-c2ae-4273-9e3f-abf1bbc5676c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 02:21:41.718002 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:41.717964    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55cda727-c2ae-4273-9e3f-abf1bbc5676c-tls-certs\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:21:41.718002 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:41.717994    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87gxj\" (UniqueName: \"kubernetes.io/projected/55cda727-c2ae-4273-9e3f-abf1bbc5676c-kube-api-access-87gxj\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:21:41.718192 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:41.718025    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55cda727-c2ae-4273-9e3f-abf1bbc5676c-tmp-dir\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:21:41.718192 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:41.718036    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55cda727-c2ae-4273-9e3f-abf1bbc5676c-dshm\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:21:42.445796 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:42.445763    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q" event={"ID":"55cda727-c2ae-4273-9e3f-abf1bbc5676c","Type":"ContainerDied","Data":"d5bc899b387ad81f793fbcf8d4b0c3bb7cb6a46e7b29bd47f2833d855d7065f6"}
Jul 21 02:21:42.445963 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:42.445810    2575 scope.go:117] "RemoveContainer" containerID="1efd7c8e9a84350e4a865916b35537a21e2339d468b1ef4df95560d4fcd4395c"
Jul 21 02:21:42.445963 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:42.445776    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q"
Jul 21 02:21:42.466866 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:42.466840    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q"]
Jul 21 02:21:42.473353 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:42.473332    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c6538a/rate-test-model-c6538a-kserve-6cc784dcff-bf78q"]
Jul 21 02:21:43.025513 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:43.025479    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fea9d62-d275-4ccb-80cd-0c4d31d8efef" path="/var/lib/kubelet/pods/0fea9d62-d275-4ccb-80cd-0c4d31d8efef/volumes"
Jul 21 02:21:43.025930 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:43.025914    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55cda727-c2ae-4273-9e3f-abf1bbc5676c" path="/var/lib/kubelet/pods/55cda727-c2ae-4273-9e3f-abf1bbc5676c/volumes"
Jul 21 02:21:45.475760 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:45.475729    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-748a4d-lss68"]
Jul 21 02:21:45.476140 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:45.476061    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fea9d62-d275-4ccb-80cd-0c4d31d8efef" containerName="main"
Jul 21 02:21:45.476140 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:45.476072    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fea9d62-d275-4ccb-80cd-0c4d31d8efef" containerName="main"
Jul 21 02:21:45.476140 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:45.476085    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55cda727-c2ae-4273-9e3f-abf1bbc5676c" containerName="main"
Jul 21 02:21:45.476140 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:45.476090    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="55cda727-c2ae-4273-9e3f-abf1bbc5676c" containerName="main"
Jul 21 02:21:45.476274 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:45.476146    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fea9d62-d275-4ccb-80cd-0c4d31d8efef" containerName="main"
Jul 21 02:21:45.476274 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:45.476154    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="55cda727-c2ae-4273-9e3f-abf1bbc5676c" containerName="main"
Jul 21 02:21:45.480388 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:45.480372    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-748a4d-lss68"
Jul 21 02:21:45.482930 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:45.482905    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x582z\""
Jul 21 02:21:45.483076 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:45.482950    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 21 02:21:45.483832 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:45.483773    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-748a4d-lss68"]
Jul 21 02:21:45.548364 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:45.548333    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71f0639c-46fa-4c62-815e-c58a849e863b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-748a4d-lss68\" (UID: \"71f0639c-46fa-4c62-815e-c58a849e863b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-748a4d-lss68"
Jul 21 02:21:45.649279 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:45.649241    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71f0639c-46fa-4c62-815e-c58a849e863b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-748a4d-lss68\" (UID: \"71f0639c-46fa-4c62-815e-c58a849e863b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-748a4d-lss68"
Jul 21 02:21:45.649892 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:45.649875    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71f0639c-46fa-4c62-815e-c58a849e863b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-748a4d-lss68\" (UID: \"71f0639c-46fa-4c62-815e-c58a849e863b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-748a4d-lss68"
Jul 21 02:21:45.792419 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:45.792388    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-748a4d-lss68"
Jul 21 02:21:45.927931 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:45.927897    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-748a4d-lss68"]
Jul 21 02:21:45.931554 ip-10-0-129-219 kubenswrapper[2575]: W0721 02:21:45.931530    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f0639c_46fa_4c62_815e_c58a849e863b.slice/crio-15baa391973eacfdedd623b894f17877f5b3539c0129d6ab6ba5bbe91ee8d270 WatchSource:0}: Error finding container 15baa391973eacfdedd623b894f17877f5b3539c0129d6ab6ba5bbe91ee8d270: Status 404 returned error can't find the container with id 15baa391973eacfdedd623b894f17877f5b3539c0129d6ab6ba5bbe91ee8d270
Jul 21 02:21:46.005481 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:46.005450    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:21:46.104844 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:46.104754    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:21:46.471921 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:46.471825    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-748a4d-lss68" event={"ID":"71f0639c-46fa-4c62-815e-c58a849e863b","Type":"ContainerStarted","Data":"9bddab6b1587c566808b5d80cfeb5bdb094a537f325c4a78b5111e681257e063"}
Jul 21 02:21:46.471921 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:46.471886    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-748a4d-lss68" event={"ID":"71f0639c-46fa-4c62-815e-c58a849e863b","Type":"ContainerStarted","Data":"15baa391973eacfdedd623b894f17877f5b3539c0129d6ab6ba5bbe91ee8d270"}
Jul 21 02:21:46.487685 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:46.487636    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-748a4d-lss68" podStartSLOduration=1.487604095 podStartE2EDuration="1.487604095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:21:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:21:46.485191604 +0000 UTC m=+2726.082831955" watchObservedRunningTime="2026-07-21 02:21:46.487604095 +0000 UTC m=+2726.085244479"
Jul 21 02:21:47.477516 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:47.477478    2575 generic.go:356] "Generic (PLEG): container finished" podID="71f0639c-46fa-4c62-815e-c58a849e863b" containerID="9bddab6b1587c566808b5d80cfeb5bdb094a537f325c4a78b5111e681257e063" exitCode=0
Jul 21 02:21:47.477725 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:47.477563    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-748a4d-lss68" event={"ID":"71f0639c-46fa-4c62-815e-c58a849e863b","Type":"ContainerDied","Data":"9bddab6b1587c566808b5d80cfeb5bdb094a537f325c4a78b5111e681257e063"}
Jul 21 02:21:48.608009 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:48.607984    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-748a4d-lss68"
Jul 21 02:21:48.675276 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:48.675245    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71f0639c-46fa-4c62-815e-c58a849e863b-maas-api-service-ca\") pod \"71f0639c-46fa-4c62-815e-c58a849e863b\" (UID: \"71f0639c-46fa-4c62-815e-c58a849e863b\") "
Jul 21 02:21:48.675601 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:48.675578    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71f0639c-46fa-4c62-815e-c58a849e863b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "71f0639c-46fa-4c62-815e-c58a849e863b" (UID: "71f0639c-46fa-4c62-815e-c58a849e863b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 02:21:48.776120 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:48.776092    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71f0639c-46fa-4c62-815e-c58a849e863b-maas-api-service-ca\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:21:49.487842 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:49.487755    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-748a4d-lss68" event={"ID":"71f0639c-46fa-4c62-815e-c58a849e863b","Type":"ContainerDied","Data":"15baa391973eacfdedd623b894f17877f5b3539c0129d6ab6ba5bbe91ee8d270"}
Jul 21 02:21:49.487842 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:49.487777    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-748a4d-lss68"
Jul 21 02:21:49.487842 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:49.487803    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15baa391973eacfdedd623b894f17877f5b3539c0129d6ab6ba5bbe91ee8d270"
Jul 21 02:21:56.025318 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:21:56.025288    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:22:00.549123 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:00.549088    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:22:00.607776 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:00.607747    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-748a4d-lss68"]
Jul 21 02:22:00.615516 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:00.615493    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-748a4d-lss68"]
Jul 21 02:22:00.954446 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:00.954324    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:22:01.025752 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:01.025725    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71f0639c-46fa-4c62-815e-c58a849e863b" path="/var/lib/kubelet/pods/71f0639c-46fa-4c62-815e-c58a849e863b/volumes"
Jul 21 02:22:07.198132 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:07.198101    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:22:07.296826 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:07.296788    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:22:07.494208 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:07.494122    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:22:07.717014 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:07.716983    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:22:07.836022 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:07.835989    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6538a-7zb4q"]
Jul 21 02:22:07.836354 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:07.836342    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71f0639c-46fa-4c62-815e-c58a849e863b" containerName="revoke-keys"
Jul 21 02:22:07.836398 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:07.836356    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="71f0639c-46fa-4c62-815e-c58a849e863b" containerName="revoke-keys"
Jul 21 02:22:07.836433 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:07.836425    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="71f0639c-46fa-4c62-815e-c58a849e863b" containerName="revoke-keys"
Jul 21 02:22:07.839463 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:07.839447    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6538a-7zb4q"
Jul 21 02:22:07.841819 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:07.841793    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x582z\""
Jul 21 02:22:07.841948 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:07.841882    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 21 02:22:07.847104 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:07.847075    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6538a-7zb4q"]
Jul 21 02:22:07.948777 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:07.948744    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02f475c0-eac5-4542-8795-151a3b9c4472-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-c6538a-7zb4q\" (UID: \"02f475c0-eac5-4542-8795-151a3b9c4472\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6538a-7zb4q"
Jul 21 02:22:08.049135 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:08.049101    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02f475c0-eac5-4542-8795-151a3b9c4472-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-c6538a-7zb4q\" (UID: \"02f475c0-eac5-4542-8795-151a3b9c4472\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6538a-7zb4q"
Jul 21 02:22:08.049799 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:08.049780    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02f475c0-eac5-4542-8795-151a3b9c4472-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-c6538a-7zb4q\" (UID: \"02f475c0-eac5-4542-8795-151a3b9c4472\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6538a-7zb4q"
Jul 21 02:22:08.150812 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:08.150717    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6538a-7zb4q"
Jul 21 02:22:08.279635 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:08.279580    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6538a-7zb4q"]
Jul 21 02:22:08.281892 ip-10-0-129-219 kubenswrapper[2575]: W0721 02:22:08.281867    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f475c0_eac5_4542_8795_151a3b9c4472.slice/crio-9331eaff72252c79dfe082cdfd705c6630405910e64658df0e7622d2d51afdee WatchSource:0}: Error finding container 9331eaff72252c79dfe082cdfd705c6630405910e64658df0e7622d2d51afdee: Status 404 returned error can't find the container with id 9331eaff72252c79dfe082cdfd705c6630405910e64658df0e7622d2d51afdee
Jul 21 02:22:08.421552 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:08.421453    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:22:08.523060 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:08.523022    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:22:08.578360 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:08.578325    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6538a-7zb4q" event={"ID":"02f475c0-eac5-4542-8795-151a3b9c4472","Type":"ContainerStarted","Data":"9331eaff72252c79dfe082cdfd705c6630405910e64658df0e7622d2d51afdee"}
Jul 21 02:22:09.584698 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:09.584663    2575 generic.go:356] "Generic (PLEG): container finished" podID="02f475c0-eac5-4542-8795-151a3b9c4472" containerID="860a97851f8dc64f899a1aa335cec222a5919f7a04a130259c58d718890a0e32" exitCode=0
Jul 21 02:22:09.585113 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:09.584729    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6538a-7zb4q" event={"ID":"02f475c0-eac5-4542-8795-151a3b9c4472","Type":"ContainerDied","Data":"860a97851f8dc64f899a1aa335cec222a5919f7a04a130259c58d718890a0e32"}
Jul 21 02:22:10.722102 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:10.722076    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6538a-7zb4q"
Jul 21 02:22:10.874067 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:10.873967    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02f475c0-eac5-4542-8795-151a3b9c4472-maas-api-service-ca\") pod \"02f475c0-eac5-4542-8795-151a3b9c4472\" (UID: \"02f475c0-eac5-4542-8795-151a3b9c4472\") "
Jul 21 02:22:10.874340 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:10.874315    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02f475c0-eac5-4542-8795-151a3b9c4472-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "02f475c0-eac5-4542-8795-151a3b9c4472" (UID: "02f475c0-eac5-4542-8795-151a3b9c4472"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 02:22:10.975573 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:10.975536    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02f475c0-eac5-4542-8795-151a3b9c4472-maas-api-service-ca\") on node \"ip-10-0-129-219.ec2.internal\" DevicePath \"\""
Jul 21 02:22:11.596254 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:11.596224    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6538a-7zb4q"
Jul 21 02:22:11.596435 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:11.596262    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6538a-7zb4q" event={"ID":"02f475c0-eac5-4542-8795-151a3b9c4472","Type":"ContainerDied","Data":"9331eaff72252c79dfe082cdfd705c6630405910e64658df0e7622d2d51afdee"}
Jul 21 02:22:11.596435 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:11.596311    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9331eaff72252c79dfe082cdfd705c6630405910e64658df0e7622d2d51afdee"
Jul 21 02:22:18.460102 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:18.460064    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:22:22.913089 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:22.913045    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6538a-7zb4q"]
Jul 21 02:22:22.915332 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:22.915308    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6538a-7zb4q"]
Jul 21 02:22:22.916362 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:22.916329    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:22:23.025194 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:23.025152    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02f475c0-eac5-4542-8795-151a3b9c4472" path="/var/lib/kubelet/pods/02f475c0-eac5-4542-8795-151a3b9c4472/volumes"
Jul 21 02:22:23.332929 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:23.332897    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:22:29.314646 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:29.314590    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:22:29.414446 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:29.414407    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:22:29.590469 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:29.590384    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:22:29.831794 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:29.831757    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:22:30.301202 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:30.301152    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-795ddbf4cf-d5bzl_baf68c78-4944-4caa-8584-9cba4c511ea9/authorino/0.log"
Jul 21 02:22:36.762923 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:36.762892    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj_770d775f-9702-42e6-872b-fb85843c999f/copy-manifests/0.log"
Jul 21 02:22:36.770969 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:36.770942    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj_770d775f-9702-42e6-872b-fb85843c999f/manager/0.log"
Jul 21 02:22:37.381101 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:37.381065    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29743335-szfh2_77f62600-525d-4655-9e47-d3ef36c9fcba/cleanup/0.log"
Jul 21 02:22:38.856536 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:38.856509    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-795ddbf4cf-d5bzl_baf68c78-4944-4caa-8584-9cba4c511ea9/authorino/0.log"
Jul 21 02:22:39.720282 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:39.720247    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-x6grd_d1f95ad8-8bd7-4956-b56c-5404a04152b6/manager/0.log"
Jul 21 02:22:40.339976 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:40.339949    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cc474c686-5sshb_17a7e457-ecc9-483f-8534-35108f24e501/kube-auth-proxy/0.log"
Jul 21 02:22:41.505160 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:41.505129    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-mhlj6_b5cbb5c8-4f1b-4bf1-b323-d9b3f783df1d/main/0.log"
Jul 21 02:22:41.752513 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:41.752488    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjktpv_8db08254-ea0e-4d01-9951-60cbefb18d27/main/0.log"
Jul 21 02:22:48.171811 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:48.171778    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:22:48.271822 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:48.271783    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log" limit=1024
Jul 21 02:22:51.034801 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:51.034758    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pmhdx_5ee75f94-10ca-4a6a-a2f4-077eb597a94f/global-pull-secret-syncer/0.log"
Jul 21 02:22:51.155336 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:51.155309    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-25bgn_a54c5fd7-029d-4f15-89ee-e46e339b8312/konnectivity-agent/0.log"
Jul 21 02:22:51.255229 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:51.255188    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-219.ec2.internal_8bb34cb64ea3f44a8e9569f1b50f1dce/haproxy/0.log"
Jul 21 02:22:56.690311 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:56.690278    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-795ddbf4cf-d5bzl_baf68c78-4944-4caa-8584-9cba4c511ea9/authorino/0.log"
Jul 21 02:22:57.169842 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:57.169814    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-x6grd_d1f95ad8-8bd7-4956-b56c-5404a04152b6/manager/0.log"
Jul 21 02:22:58.872344 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:58.872307    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1e261108-17de-48b2-8051-544ca6a67905/alertmanager/0.log"
Jul 21 02:22:58.896827 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:58.896800    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1e261108-17de-48b2-8051-544ca6a67905/config-reloader/0.log"
Jul 21 02:22:58.923340 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:58.923319    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1e261108-17de-48b2-8051-544ca6a67905/kube-rbac-proxy-web/0.log"
Jul 21 02:22:58.944989 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:58.944934    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1e261108-17de-48b2-8051-544ca6a67905/kube-rbac-proxy/0.log"
Jul 21 02:22:58.967056 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:58.967038    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1e261108-17de-48b2-8051-544ca6a67905/kube-rbac-proxy-metric/0.log"
Jul 21 02:22:58.989803 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:58.989778    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1e261108-17de-48b2-8051-544ca6a67905/prom-label-proxy/0.log"
Jul 21 02:22:59.012228 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:59.012211    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1e261108-17de-48b2-8051-544ca6a67905/init-config-reloader/0.log"
Jul 21 02:22:59.362369 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:59.362333    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5g8kh_734ab4c2-b74d-48ec-9b98-a0a42eaf797e/node-exporter/0.log"
Jul 21 02:22:59.401803 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:59.401775    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5g8kh_734ab4c2-b74d-48ec-9b98-a0a42eaf797e/kube-rbac-proxy/0.log"
Jul 21 02:22:59.423469 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:59.423445    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5g8kh_734ab4c2-b74d-48ec-9b98-a0a42eaf797e/init-textfile/0.log"
Jul 21 02:22:59.640254 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:59.640173    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-gtp7f_5dec2ddc-a63d-479d-8fa7-f422b0ed0579/kube-rbac-proxy-main/0.log"
Jul 21 02:22:59.664063 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:59.664040    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-gtp7f_5dec2ddc-a63d-479d-8fa7-f422b0ed0579/kube-rbac-proxy-self/0.log"
Jul 21 02:22:59.693200 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:59.693165    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-gtp7f_5dec2ddc-a63d-479d-8fa7-f422b0ed0579/openshift-state-metrics/0.log"
Jul 21 02:22:59.732714 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:59.732692    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c426235f-0cee-44f4-b4f7-5ebc837e7f22/prometheus/0.log"
Jul 21 02:22:59.756804 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:59.756784    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c426235f-0cee-44f4-b4f7-5ebc837e7f22/config-reloader/0.log"
Jul 21 02:22:59.782531 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:59.782509    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c426235f-0cee-44f4-b4f7-5ebc837e7f22/thanos-sidecar/0.log"
Jul 21 02:22:59.805762 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:59.805740    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c426235f-0cee-44f4-b4f7-5ebc837e7f22/kube-rbac-proxy-web/0.log"
Jul 21 02:22:59.829956 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:59.829936    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c426235f-0cee-44f4-b4f7-5ebc837e7f22/kube-rbac-proxy/0.log"
Jul 21 02:22:59.854762 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:59.854742    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c426235f-0cee-44f4-b4f7-5ebc837e7f22/kube-rbac-proxy-thanos/0.log"
Jul 21 02:22:59.879517 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:59.879494    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c426235f-0cee-44f4-b4f7-5ebc837e7f22/init-config-reloader/0.log"
Jul 21 02:22:59.913694 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:59.913602    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-rgls6_941d07b2-62c9-43e6-9a57-a50b65d85283/prometheus-operator/0.log"
Jul 21 02:22:59.933307 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:59.933284    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-rgls6_941d07b2-62c9-43e6-9a57-a50b65d85283/kube-rbac-proxy/0.log"
Jul 21 02:22:59.959452 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:22:59.959429    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-78q2q_ce2e1480-0f6e-4d72-8f0e-ccb153d25c33/prometheus-operator-admission-webhook/0.log"
Jul 21 02:23:01.214910 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:01.214862    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gfbpz/perf-node-gather-daemonset-6x8p8"]
Jul 21 02:23:01.215293 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:01.215200    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02f475c0-eac5-4542-8795-151a3b9c4472" containerName="revoke-keys"
Jul 21 02:23:01.215293 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:01.215210    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="02f475c0-eac5-4542-8795-151a3b9c4472" containerName="revoke-keys"
Jul 21 02:23:01.215293 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:01.215275    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="02f475c0-eac5-4542-8795-151a3b9c4472" containerName="revoke-keys"
Jul 21 02:23:01.218415 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:01.218395    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-6x8p8"
Jul 21 02:23:01.221838 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:01.221819    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gfbpz\"/\"kube-root-ca.crt\""
Jul 21 02:23:01.222595 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:01.222579    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gfbpz\"/\"default-dockercfg-blw8z\""
Jul 21 02:23:01.222697 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:01.222635    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gfbpz\"/\"openshift-service-ca.crt\""
Jul 21 02:23:01.231666 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:01.231643    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gfbpz/perf-node-gather-daemonset-6x8p8"]
Jul 21 02:23:01.309436 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:01.309403    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2402eeb4-3182-458c-9cde-d44e679bd346-sys\") pod \"perf-node-gather-daemonset-6x8p8\" (UID: \"2402eeb4-3182-458c-9cde-d44e679bd346\") " pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-6x8p8"
Jul 21 02:23:01.309646 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:01.309462    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2402eeb4-3182-458c-9cde-d44e679bd346-podres\") pod \"perf-node-gather-daemonset-6x8p8\" (UID: \"2402eeb4-3182-458c-9cde-d44e679bd346\") " pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-6x8p8"
Jul 21 02:23:01.309646 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:01.309487    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2402eeb4-3182-458c-9cde-d44e679bd346-proc\") pod \"perf-node-gather-daemonset-6x8p8\" (UID: \"2402eeb4-3182-458c-9cde-d44e679bd346\") " pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-6x8p8"
Jul 21 02:23:01.309646 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:01.309504    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdqfg\" (UniqueName: \"kubernetes.io/projected/2402eeb4-3182-458c-9cde-d44e679bd346-kube-api-access-mdqfg\") pod \"perf-node-gather-daemonset-6x8p8\" (UID: \"2402eeb4-3182-458c-9cde-d44e679bd346\") " pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-6x8p8"
Jul 21 02:23:01.309646 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:01.309535    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2402eeb4-3182-458c-9cde-d44e679bd346-lib-modules\") pod \"perf-node-gather-daemonset-6x8p8\" (UID: \"2402eeb4-3182-458c-9cde-d44e679bd346\") " pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-6x8p8"
Jul 21 02:23:01.410529 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:01.410496    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2402eeb4-3182-458c-9cde-d44e679bd346-sys\") pod \"perf-node-gather-daemonset-6x8p8\" (UID: \"2402eeb4-3182-458c-9cde-d44e679bd346\") " pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-6x8p8"
Jul 21 02:23:01.410738 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:01.410542    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2402eeb4-3182-458c-9cde-d44e679bd346-podres\") pod \"perf-node-gather-daemonset-6x8p8\" (UID: \"2402eeb4-3182-458c-9cde-d44e679bd346\") " pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-6x8p8"
Jul 21 02:23:01.410738 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:01.410561    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2402eeb4-3182-458c-9cde-d44e679bd346-proc\") pod \"perf-node-gather-daemonset-6x8p8\" (UID: \"2402eeb4-3182-458c-9cde-d44e679bd346\") " pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-6x8p8"
Jul 21 02:23:01.410738 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:01.410578    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdqfg\" (UniqueName: \"kubernetes.io/projected/2402eeb4-3182-458c-9cde-d44e679bd346-kube-api-access-mdqfg\") pod \"perf-node-gather-daemonset-6x8p8\" (UID: \"2402eeb4-3182-458c-9cde-d44e679bd346\") " pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-6x8p8"
Jul 21 02:23:01.410738 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:01.410633    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2402eeb4-3182-458c-9cde-d44e679bd346-lib-modules\") pod \"perf-node-gather-daemonset-6x8p8\" (UID: \"2402eeb4-3182-458c-9cde-d44e679bd346\") " pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-6x8p8"
Jul 21 02:23:01.410738 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:01.410650    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2402eeb4-3182-458c-9cde-d44e679bd346-sys\") pod \"perf-node-gather-daemonset-6x8p8\" (UID: \"2402eeb4-3182-458c-9cde-d44e679bd346\") " pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-6x8p8"
Jul 21 02:23:01.410738 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:01.410708    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2402eeb4-3182-458c-9cde-d44e679bd346-podres\") pod \"perf-node-gather-daemonset-6x8p8\" (UID: \"2402eeb4-3182-458c-9cde-d44e679bd346\") " pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-6x8p8"
Jul 21 02:23:01.410738 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:01.410652    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2402eeb4-3182-458c-9cde-d44e679bd346-proc\") pod \"perf-node-gather-daemonset-6x8p8\" (UID: \"2402eeb4-3182-458c-9cde-d44e679bd346\") " pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-6x8p8"
Jul 21 02:23:01.410990 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:01.410754    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2402eeb4-3182-458c-9cde-d44e679bd346-lib-modules\") pod \"perf-node-gather-daemonset-6x8p8\" (UID: \"2402eeb4-3182-458c-9cde-d44e679bd346\") " pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-6x8p8"
Jul 21 02:23:01.419218 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:01.419189    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdqfg\" (UniqueName: \"kubernetes.io/projected/2402eeb4-3182-458c-9cde-d44e679bd346-kube-api-access-mdqfg\") pod \"perf-node-gather-daemonset-6x8p8\" (UID: \"2402eeb4-3182-458c-9cde-d44e679bd346\") " pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-6x8p8"
Jul 21 02:23:01.528700 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:01.528680    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-6x8p8"
Jul 21 02:23:01.657549 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:01.657524    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gfbpz/perf-node-gather-daemonset-6x8p8"]
Jul 21 02:23:01.659549 ip-10-0-129-219 kubenswrapper[2575]: W0721 02:23:01.659522    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2402eeb4_3182_458c_9cde_d44e679bd346.slice/crio-e079db62cddb2bbed0a178014771394ab36da5b4d934821a942b317424d2bfd5 WatchSource:0}: Error finding container e079db62cddb2bbed0a178014771394ab36da5b4d934821a942b317424d2bfd5: Status 404 returned error can't find the container with id e079db62cddb2bbed0a178014771394ab36da5b4d934821a942b317424d2bfd5
Jul 21 02:23:01.821498 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:01.821422    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-6x8p8" event={"ID":"2402eeb4-3182-458c-9cde-d44e679bd346","Type":"ContainerStarted","Data":"295d82e432d303a252c750fcafa9d683bf2137111c99119cc793c84203b01669"}
Jul 21 02:23:01.821498 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:01.821477    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-6x8p8" event={"ID":"2402eeb4-3182-458c-9cde-d44e679bd346","Type":"ContainerStarted","Data":"e079db62cddb2bbed0a178014771394ab36da5b4d934821a942b317424d2bfd5"}
Jul 21 02:23:01.821745 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:01.821545    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-6x8p8"
Jul 21 02:23:01.837662 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:01.837595    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-6x8p8" podStartSLOduration=0.837579855 podStartE2EDuration="837.579855ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 02:23:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:23:01.835662481 +0000 UTC m=+2801.433302833" watchObservedRunningTime="2026-07-21 02:23:01.837579855 +0000 UTC m=+2801.435220206"
Jul 21 02:23:02.602324 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:02.602294    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-q6t9r_0937c807-6c37-4807-8499-e71c8f8ceb02/download-server/0.log"
Jul 21 02:23:04.298469 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:04.298440    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-twkvj_06885d56-7129-4e19-9911-5720b9668058/dns/0.log"
Jul 21 02:23:04.321897 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:04.321863    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-twkvj_06885d56-7129-4e19-9911-5720b9668058/kube-rbac-proxy/0.log"
Jul 21 02:23:04.467810 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:04.467781    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fw8b6_ed12def9-d42e-4d3e-98b2-d870a1c43231/dns-node-resolver/0.log"
Jul 21 02:23:05.029603 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:05.029574    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-79db7cc8cd-f6q5z_f034e8e1-3fdf-4257-986b-34784be403cf/registry/0.log"
Jul 21 02:23:05.090462 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:05.090433    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kzvp4_7073f999-4de0-4a60-b3a2-58e9e527ffea/node-ca/0.log"
Jul 21 02:23:06.244911 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:06.244882    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cc474c686-5sshb_17a7e457-ecc9-483f-8534-35108f24e501/kube-auth-proxy/0.log"
Jul 21 02:23:07.199290 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:07.199260    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-drc4p_86db8e5d-3869-4ad0-8d3d-989e132f6038/serve-healthcheck-canary/0.log"
Jul 21 02:23:07.716286 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:07.716255    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2v6dg_7971e50e-1094-40f1-a8e4-56eb51aa83d4/kube-rbac-proxy/0.log"
Jul 21 02:23:07.742766 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:07.742744    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2v6dg_7971e50e-1094-40f1-a8e4-56eb51aa83d4/exporter/0.log"
Jul 21 02:23:07.767251 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:07.767225    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2v6dg_7971e50e-1094-40f1-a8e4-56eb51aa83d4/extractor/0.log"
Jul 21 02:23:07.835966 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:07.835941    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-6x8p8"
Jul 21 02:23:10.970166 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:10.970125    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj_770d775f-9702-42e6-872b-fb85843c999f/manager/0.log"
Jul 21 02:23:10.995767 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:10.995747    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d7c89d5b6-j7bgj_770d775f-9702-42e6-872b-fb85843c999f/copy-manifests/0.log"
Jul 21 02:23:12.137244 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:12.137215    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cc474c686-5sshb_17a7e457-ecc9-483f-8534-35108f24e501/kube-auth-proxy/0.log"
Jul 21 02:23:12.278256 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:12.278220    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-95964d979-smzzg_65d1d87b-eb13-439e-801a-d663fa1c8559/manager/0.log"
Jul 21 02:23:12.306890 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:12.306869    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-hlvs5_e2880148-447d-4138-87cb-b4b8840bb62c/openshift-lws-operator/0.log"
Jul 21 02:23:13.682702 ip-10-0-129-219 kubenswrapper[2575]: I0721 02:23:13.682673    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cc474c686-5sshb_17a7e457-ecc9-483f-8534-35108f24e501/kube-auth-proxy/0.log"